Ask any tech-savvy internet traveller, and they will probably tell you that they know the internet pretty well. With the help of powerful search engines such as Google, Yahoo and Bing, locating relevant and meaningful information has become a lot easier.
Google serves up more than three billion searches each day, and search-results pages are based, in part, on a ranking system called a PageRank. The higher a website’s PageRank, the higher its position in search results. It is estimated that few users look beyond the first three pages of search results – therefore, being on the first page for a specific search term is highly beneficial.
But what about information that isn’t fetched through Google’s hunt for content in publicly accessible documents? We might believe that Google has categorized and index the entire web for our perusal, but that is far from the truth. In 2010, the internet contained over 5 million terrabytes of data, and Google only managed to index 0.04% of it.
So where is the other 99.96%?
Well, it is still accessible – for the most part. It’s called the Deep Web (also known as Deepnet, Invisible Web, Undernet or Hidden Web). It is a collection of webpages that are not indexed by search engines, or content that is not part of the Surface Web.
According to Mike Bergman, founder of BrightPlanet and credited with coining the phrase, it’s like fishing in the ocean. “Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed.”
In 2000, Bergman’s BrightPlanet estimated that “public information on the deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web,” and they have actually developed a search engine specifically for the Deep Web.
What information is available within the Deep Web?
Despite what many internet users might have heard, the Deep Web is not as sinister as they have come to believe. Majority of the uncategorized webpages are actually rather boring – and consists of library catalogues, official legislative documents of governments, phone books, and other content that is dynamically prepared to respond to a query.
But there is definitely a shady side to the Deep Web. By using The Onion Router (Tor), users can download free software to surf the Deep Web, which in turn enables online anonymity and censorship resistance.
The National Security Agency (NSA) in the US described Tor as “the King of high secure, low latency Internet anonymity” with “no contenders for the throne in waiting”. Once you are on the radar of the NSA, it’s usually a bad thing…
Tor is like the Bitcoin of internet browsers, as it directs Internet traffic through a free, worldwide, volunteer network consisting of more than five thousand relays to conceal a user’s location or usage from anyone conducting network surveillance or traffic analysis – making it perfect for criminals.
But Tor doesn’t serve up your everyday webpages – Tor only accesses, mostly nefarious and hidden pages within the Deep Web with the .onion extension. According to Wikipedia, “.onion is a pseudo-top-level domain host suffix designating an anonymous hidden service reachable via the Tor network.”
.onion webpages also don’t show up in a simple search from Tor, as users will need to know where to look in order to visit them. There are a number of Tor hidden service directories, like a telephone directory which lists .onion webpages and their addresses. Out of interest, a typical webpage address accessed by Tor looks like http://kpvz7ki2v5agwt35.onion
These webpages range from anything concerning hacking, cracking, pornography to detailed personal information and black market goods. The most well-known webpage for black market purchases is Silk Road (http://silkroad6ownowfk.onion), and has been described as the “eBay for drugs”.
The site became so notorious that on 2 October 2013, the FBI shut it down, arrested Ross William Ulbricht, and charged him with allegedly facilitating murder for hire, narcotics trafficking violations, facilitating computer hacking and money laundering. According to Forbes, soon after the take-down, a second webpage sprung up, called Silk Road 2.0 which was being run by former administrators of original initial Silk Road.
FBI agents seized over 26,000 Bitcoins (worth $3.6 million at the time) from accounts on Silk Road, and seized 144,000 Bitcoins worth $28 million the FBI believed belonged to Ulbricht. In terms of sales, it is estimated that $15 million in transactions were made annually on Silk Road – all paid for in Bitcoins.
Conclusion
For the most part, the Deep Web is the bastard child of the World Wide Web, the place where nobody wants to go, or even knew existed. It’s a boring directory of musty, old, library content and uncategorized webpages that nobody could be bothered about – unless they know where to look and how to find it.
But the seedy side of the Deep Web does exist and it’s a disturbing place. Once you have to conceal your identity in order to access webpages and erase your online footprint, something is definitely wrong.
But luckily everyday users don’t have to worry that they would stumble through the gated doors by accident, as they will need specific software for that exact purpose – a wilful act.
Charlie Fripp – Consumer Tech editor
