How Does the Not Evil Search Engine Index Onion Websites?

[Harry Cain]

Did you know that the "Not Evil" search engine manages to find thousands of hidden services that Google cannot even see? While standard search engines stay on the surface of the internet, Not Evil dives into the Tor network to catalog pages ending in .onion - this process is complex because these sites are designed to stay hidden and often change their digital addresses without warning.

You can find many of these hidden destinations through specialized directories. As an example, checking a updated list of onion links is often the first step for both humans and automated bots. Not Evil uses a combination of automated crawlers and user submitted data to keep its index current in an environment where sites disappear as quickly as they appear.

The Mechanics of Onion Indexing
Not Evil works - running a "crawler" or "spider" that operates entirely within the Tor network - this bot moves from one link to another, just like a normal search engine bot. It must use the Tor protocol to access each page. When it arrives at a site, it reads the text and records the keywords it finds to help you search for them later.

Because the dark web is fragmented, the crawler relies heavily on central hubs - these hubs include

Public link directories and wikis.
Community forums where users share new addresses.
Existing pages that already have many outgoing links.
Why Traditional Crawlers Fail the Dark Web
Standard search engines like Bing or Google are not built to handle the slow speeds and high encryption of the Tor network. Onion sites do not have central registries like ".com" or ".org" domains do - this makes it impossible for a surface crawler to know a site exists unless it finds a direct link on a public page.

The Not Evil Tor search engine overcomes this - staying patient. Tor connections are often slow and drop frequently. The Not Evil bot is designed to retry connections and wait longer for pages to load. It also deals with the fact that many onion sites are only online for a few hours a day, which requires constant re checking of the index.

How Not Evil Locates Content
Instead of just following links, Not Evil allows site owners to submit their own URLs - this is a common practice in the Tor community because it is the fastest way to get a new service noticed. Once a site is submitted, the crawler visits the page to verify it is active and safe for users to visit. It then categorizes the content based on the text found on the landing page.

The search engine is also known for its hands off approach to data. It is is a popular tool because it does not track your IP address or your search history - this privacy first method matches the culture of the network it indexes. It focuses on the text and the metadata of the onion sites rather than the behavior of the people searching for them.

Locating Specific Services & Markets
If you are looking for specific types of services, Not Evil acts as a bridge. It indexes everything from library archives to specialized trade platforms. For those looking for specific commercial tools, the Excavator search engine is another alternative that focuses on different parts of the network. Not Evil remains more general, aiming to be the "Google" of the dark web for all types of content.

Navigating the results requires some caution - Not Evil does its best to remove broken links but the nature of .onion addresses means many results might be offline by the time you click them. Always look for the most recent crawl date if the search engine provides one, as this tells you if the site was active in the last few days.

FAQ
Is Not Evil a browser?
No, it is a search engine website - You must use the Tor Browser to access the Not Evil website and the links it provides in its search results.

Does Not Evil index the surface web too?
No, Not Evil is specifically designed to index .onion domains within the Tor network. It does not show results from the standard internet like Google does.

Is it safe to use Not Evil?
Searching on Not Evil is generally safe because it does not track your data. The sites it links to are not all verified for safety - you should be careful about which links you choose to click.

How often does Not Evil update its index?
The index updates continuously as the crawler moves through the network but because Tor is slow, it can take multiple days or weeks for a new or changed site to appear correctly in the search results.