What it is and the problem it solves
A search engine is a tool that locates publicly accessible web content by crawling, indexing, and retrieving it. It solves the problem of information scarcity in a growing, unstructured network — but only for content that is crawlable, indexable, and not blocked.
How it works
It works in three stages: crawling pages, indexing their content into inverted lists, and searching those indexes using ranking algorithms. Archie (1990) crawled FTP file names only. JumpStation (December 1993) was the first to combine all three stages for the WWW. RankDex (1996) introduced hyperlink-based ranking. Google (1998) added paid search advertising sourced from goto.com.
What works
The triad works: crawling discovers pages, indexing maps terms to locations, and searching retrieves matches. JumpStation (December 1993) proved the triad could operate on the WWW. WebCrawler (1994) confirmed full-text search was usable at scale. RankDex (1996) showed hyperlinks could improve ranking over keyword frequency alone.
What does not
It does not index the full web. It does not understand meaning. It does not guarantee relevance. Archie indexed file names only, not contents. Modern engines still omit large swathes — private databases, password-locked sites, dynamic pages — and treat all links as equal signals unless weighted by proprietary, undisclosed rules.
What it changes
It changes how authority is assigned online. RankDex (1996) made hyperlinks a proxy for quality. Google’s adoption of paid search (1998) made visibility a function of budget as well as link structure. The engine shifted from a discovery tool to an attention marketplace.
Is it worth your time
Yes — if you rely on finding information across distributed, unstructured public text. Its core triad remains necessary infrastructure. But its commercial evolution has prioritised ad revenue over precision, and its ranking logic is opaque, not neutral.
