What is the "Deep Web"?

Web content crawlers cannot get to all information on the Internet because some critical information resides inside databases, behind firewalls or is available only for a fee. Deep web statistics: 

• Publicly available information is 400-550 times larger than the known web (surface web)
• Consists of 7500 terabytes of information
• 200,000 deep web websites have been identified
• Content is of higher quality than the surface web
• The deep web is Growing faster than the surface web
• The deep web contains nearly 550 billion individual documents compared to the one billion of the surface Web.
• Sixty of the largest deep web sites collectively contain about 750 terabytes of information -- sufficient by themselves to exceed the size of the surface Web forty times.
• On average, deep web sites receive fifty per cent greater monthly traffic than surface sites and are more highly linked to than surface sites; however, the typical (median) deep web site is not well known to the Internet-searching public.
• The deep web is the largest growing category of new information on the Internet.
• deep web sites tend to be narrower, with deeper content, than conventional surface sites.
• Total quality content of the deep web is 1,000 to 2,000 times greater than that of the surface Web.
• deep web content is highly relevant to every information need, market, and domain.
• More than half of the deep web content resides in topic-specific databases.
• A full ninety-five per cent of the deep web is publicly accessible information -- not subject to fees or subscriptions.

To put these findings in perspective, a study at the NEC Research Institute published in Nature estimated that the search engines with the largest number of Web pages indexed (such as Google or Northern Light) each index no more than sixteen per cent of the surface Web. Since they are missing the deep web when they use such search engines, Internet searchers are therefore searching only 0.03% -- or one in 3,000 -- of the pages available to them today. Clearly, simultaneous searching of multiple surface and deep web sources is necessary when comprehensive information retrieval is needed.(according to BrightPlanet March 2000 study) 

HepConnect Features:

• Search engine independent
• Configurable user interface
• Common, consistent user interface across heterogeneous document collections
• Built-in navigation capabilities
• Built-in mark & download capability
• Field including date-range searching
• Supports access to login-restricted sites
• Supports access to sites that use cookies & other session ids

How HepConnect Works

• Translates user search requests into the syntax that is understood by the web database being searched
• Maps fields that user is searching to fields that are available in database
• Submits searches in parallel, in real time, to the various databases selected by the user
• Sends requests to web server in same format as a web browser would
• Result lists are parsed, and field values (author, title, etc) are extracted.

Advantages of HepConnect

• No crawling or replacing data required
• Centralized access to multiple databases from one search form
• User needs to learn only one search language
• Common user interface across all sources accessed
• Improved functionality, in many cases, as compared to what is provided by the source itself
• Configurable for a wide variety of web databases and information sources

For additional information on the process behind HepConnect:

Searching the Deep web:
Directed Query Engine Applications
At the Department of Energy
www.dlib.org/dlib/january01/warnick/01warnick.html

The Deep Web:
Surfacing Hidden Value
www.brightplanet.com/deepcontent/tutorials/DeepWeb/index.asp

See the presentation (13MB) Deep Web Searching: The Technology behind Heplink that we gave to the Centers for Disease Control in Atlanta in June 2003.