Creating search engines is not tremendously hard. When a local search engine bragged about the “years of development” and the “huge team of academic developers”, a friend of mine wrote one in Perl over an evening and launched a press release the next morning to tell how utterly misleading that image was; anyone could hack a simple search engine with a few lines of code. The trouble, of course, is not with the technology by itself — it’s the infrastructure. That’s the major reason why currently just three search engines dominate the world (at least, the English-speaking world; local search engines haven’t yet died). Start-ups like Wikia abandoned their search engine (which is now open source), very likely because of the insane amount of infrastructure needed, which is hard to pay for to remain competitive. Google, Yahoo, and Live Search are simply too big to compete against, and they all have very successful business models to back them up financially.
But the existing search engines — specially Google’s, of course — are typical cases of disruptive technologies. At one point in time, they didn’t exist at all. At the next moment, they collapsed the whole Web 1.0 content-oriented, portal-centred model. Suddenly it was irrelevant where your content was located: search engines would find where it was, and you wouldn’t need any “portals” any more. The notion of a fragmented Web, where groups of users assembled around their favourite portal, simply ceased to exist. Instead, the world was pushed towards search engines as the way to get in touch with content — and, as Google so successfully showed, as a valid business model for selling ads. The change was quite subtle, but it nevertheless showed the power of disruptive technologies — they made us realise that the Web could be used in a totally different and absolutely unforeseen way. And, in a sense, this lead to a new level of unification: content, at last, was universally accessible through a single choice (well, a few choices).
The next step, of course, was that all content-based site owners eagerly wished to jump into this new way of locating content on the Web. This lead to a new trend: site owners willingly (and freely) pushed their content for search engines to index. The former business model of “you pay for links on our portal” was totally changed into the notion that “you can submit your site’s content to be indexed for free; this will make our search engine more powerful; and we (the search engine owners) will increase revenues by selling ads (or profiling data)”. Technologies like Sitemaps allowed content publishers to simply inform the search engines when their content had changed, making sure it is kept up to date and gets immediately indexed.| ← Previous | | | Next → |