If you've ever searched for anything, you've probably run into stop words. Stop words are words so common they are typically ignored for search purposes. That is, if you type in a stop word as one of your search terms, the search engine will ignore that word (if it can). If you attempt to search using nothing but stop words, the search engine will throw up its hands and tell you to try again.
Oh, my, you don’t know how many times I tried to find any The The album in various web stores… If I did not know exactly the album title, I ended either with no matches or too many matches.
I remember once having an enormously frustrating time trying to use the internet to discover the name of the album by the band ‘A’ that contained their hit single, ‘Nothing’.
I’m almost positive that google didn’t ignore stopwords back in 2004 either. I remember searching for combinations stop queries ‘the the’ or ‘the who’ (a matter of personal taste in music and got nice results.
I think the differences in the stopword lists could be partly attributed to the different corpora used for frequency counts + some tuning of the number after checking the results. if oracle used their own corporate archive to decide the stopwords and mysql used the Wall Street journal archive - they’ll probably get different lists (obviously, the most frequent ones will be the same).
btw, back at 2004 google were granting keys and API for 1000 automated queries a day. use 10 email accounts and obtain 10 keys… without violating their policy.
Stop words have always been a pain, because they lack granularity (they just don’t exist). But going without any kind of search filtering is just as annoying because it brings up unwanted data, and even google still falls for this every so often.
What happens if you’re looking for video games written in PHP? A search for php video game on google returns a vast heap of results that are results for video game which happen to have a php extension somewhere in their URL. The same happens quite frequently when you’re looking for esoteric PHP-related concepts on the web. In this situation, neither stop-wording php nor avoiding any stop-words solves the problem: a more clever technique is required to eliminate or rate down some occurrences but not others.
One key point is that the list Jeff gives (522,000,000 for ‘the’ and so on) is not the frequency of the word, but the number of pages containing that word.
The word itself may appear many times within the page, meaning that the relative frequency of ‘the’ and ‘of’ compared with ‘reviews’ is much greater than indicated. A typical 500-word page will probably have a few dozen 'the’s.
I imagine Google has plenty of rules added by hand for these words, to reflect the fact that the semantics of ‘review’ are much more specific than those of ‘of’. Plus they seem to give a higher ranking to results which use the search keywords as a phrase rather than separately; so ‘the the’ and ‘to be or not to be’ are probably handled in that way. If you rearrange the words in ‘to be or not to be’ you will get different results.
The worst of all in that respect is the band Can. Searching for them has always been a real pain. At least that’s not a terrible name. But the band Ours is both troublesome to search for and an annoying name. I avoid mentioning them to people even though I quite like their music.
and got nice results.