Your search engine’s results are just the tip of the iceberg.
As if the Internet weren’t already amazing enough, there is a whole secret underworld to it that you may not even know about.
Known as the “Invisible Web” or “Deep Web”, various estimates put unindexed sites and content at anywhere from 94% of the internet to 500 times the information available on the “Surface Web”.
That means that anywhere from 94% to 98.8% of the content on the Internet is not searchable using conventional search engines.
The information pulled by search engines such as Google or Yahoo is gathered by web crawlers (or spiders), automated programs that travel from site to site using hyperlinks to index web addresses and content – a sort of mapping of the Internet. Deep Web pages can’t be indexed by crawlers, for a number of potential reasons: they may contain no external links, (basically, no way in for web crawlers), the site may contain technical barriers to prevent indexing by web crawlers, such as password protection, or they may be script-based pages that cannot be read by crawlers (usually multi-media content, like videos).
When you use a popular search engine to find information on the Internet, your results are literally just the tip of the iceberg; pretty astounding when you consider that Google claims to have indexed well over 8 billion pages.
Because of the wealth of information available on these unindexed sites, various invisible web search engines and search techniques are being developed all the time. At Life Hacker, Wendy Boswell explains how you can refine your web searches to subject-specific databases to access sites with unindexed content.
At Life Hacker, Wendy Boswell explains how you can refine your web searches to subject-specific databases to access sites with unindexed content.
Complete Planet contains over 70,000 searchable databases.
AlltheWeb provides regular search services, as well as searching PDF, MS Word, Windows Media, Real, QuickTime and MP3 files. Finally, for the truly dedicated unearthers of the truth, I simply must mention UFOSeek: The UFO and Paranormal Search Engine which contains over 57,000 paranormal and UFO sites.
Invisible web resources and further information are widely available on the surface web by searching for terms like “invisible”, “deep” or “cloaked” web, and more of these sites are being unearthed and made widely available all the time.