The end of fragmentation: shifting the paradigm
Let’s take a look at what happened by the end of the 1990s, when the “portal wars” were at its most aggressive peak. It wasn’t clear who would “win” the battle. Hundreds or even thousands of competitors continued to promote more and more services, links, and tools to get access to information, and the more they managed to attract to “their” portal, the more useful it was — both to themselves (more ads) but to the users as well (more access to information).
However, there was no clear winner. There would never be: with too much fragmentation, no clear market leader was able to emerge, no matter how good the “portal” was. They could be better, but they couldn’t be uniquely and absolutely compelling for the whole world to log in there. People were constantly launching “new and improved portals”.
At some stage, some clever people just thought, “what if I simply started up a meta-portal, ie. a portal that would allow me to search for all content on all portals, and attract people that way?” Search engines were developed that way. In fact, when the first developers started their Web crawling to index data, they almost inevitably started by indexing the portal sites first (they had the most links!), and then indexed all sites linked from there.
Creating search engines is not tremendously hard. When a local search engine bragged about the “years of development” and the “huge team of academic developers”, a friend of mine wrote one in Perl over an evening and launched a press release the next morning to tell how utterly misleading that image was; anyone could hack a simple search engine with a few lines of code. The trouble, of course, is not with the technology by itself — it’s the infrastructure. That’s the major reason why currently just three search engines dominate the world (at least, the English-speaking world; local search engines haven’t yet died). Start-ups like Wikia abandoned their search engine (which is now open source), very likely because of the insane amount of infrastructure needed, which is hard to pay for to remain competitive. Google, Yahoo, and Live Search are simply too big to compete against, and they all have very successful business models to back them up financially.
But the existing search engines — specially Google’s, of course — are typical cases of disruptive technologies. At one point in time, they didn’t exist at all. At the next moment, they collapsed the whole Web 1.0 content-oriented, portal-centred model. Suddenly it was irrelevant where your content was located: search engines would find where it was, and you wouldn’t need any “portals” any more. The notion of a fragmented Web, where groups of users assembled around their favourite portal, simply ceased to exist. Instead, the world was pushed towards search engines as the way to get in touch with content — and, as Google so successfully showed, as a valid business model for selling ads. The change was quite subtle, but it nevertheless showed the power of disruptive technologies — they made us realise that the Web could be used in a totally different and absolutely unforeseen way. And, in a sense, this led to a new level of unification: content, at last, was universally accessible through a single choice (well, a few choices).
The next step, of course, was that all content-based site owners eagerly wished to jump into this new way of locating content on the Web. This led to a new trend: site owners willingly (and freely) pushed their content for search engines to index. The former business model of “you pay for links on our portal” was totally changed into the notion that “you can submit your site’s content to be indexed for free; this will make our search engine more powerful; and we (the search engine owners) will increase revenues by selling ads (or profiling data)”. Technologies like Sitemaps allowed content publishers to simply inform the search engines when their content had changed, making sure it is kept up to date and gets immediately indexed.
So, after half a decade or so of further fragmentation of the Web 1.0, finally it was getting unified again. This, however, only happened thanks to a paradigm shift, where new habits gained more acceptance (locating a search engine and using keywords to navigate to content, as opposed to going to a portal and see what links were available there).
