Uncanny Valley of Search
There is a dip in usefulness between current levels of ability and true, human-like search ability. That's analogous to the dip in appeal of too-realistic 3D models of people known as the uncanny valley.
When I use Google, I use it as a deterministic tool. Here's how I find something specific on the web when I don't remember where it is. I perform a search with few keywords, knowing that if it exists on the searchable web, it will be somewhere in there. When I add more words to the search, the number of pages to look through decreases. I am sure this will happen, so I can choose the balance between specificity of search and number of pages.
Once Google gets a bit more sophisticated, this will no longer be possible. The algorithm will become non-deterministic, as adding more words will make it suddenly get a guess at what I "really mean" and suddenly bring up pages that don't contain any of the words I listed but fit the general gist. When that happens, It will be less useful to me than it was before, because I won't be able to leverage it as well by modeling what it is going to do in my own head. For example: Clippy the paperclip in Microsoft Word. When Google becomes freakish, unnatural and zombie-like, they can't say I didn't warn them.
Comments