A March Towards Understanding User's Intention

Possibly the most important element in the search - where a small search input box on a web page tries to figure out what the searchers want - is the ability of search engine technology to understand users' intent. What he/she means when a search term is entered? This almost seems like the classic AI problem, (Remember Turing's test?). Can computers understand human intentions? It is still a long journey ahead before search engines will be able to understand what users really want. Unless of course users are ready to part with their personal data and search engines are able to use that information - either from user's PC or stored online - with security and privacy guarantees provided to the users. And also provided that users are ready to trust the search engines.

Another important thing about understanding user's intention is that search engines - while building the techniques and algorithm to do so - will have to make sure they balance out what is the "perceived" notion of what the user wants and while attempt at going closer to the user's real intention. Up until now the technology world has been very good at trying to build imaginary castles out what users would or could want, but very few technologies have really tried to be "humble" to the users and understand what they really want. Search engines will have to put this perspective on top of their future agenda

It quite clear that some of the ideas that are being tossed (personalization and contextual search) around do indicate search engines have started working on these issues. Google, being the trailblazer of search engine industry, it seems has several projects going that could be crucial for the road map towards understanding user's intentions

Recently Director, Marketing at Vivisimo wrote to us about their perspective on how clustering could help search engines letting users do unstructured queries. The email interaction with Saman Haqqi suggested that "Vivisimo's main contention is that with clustering, search engines do not need to make a presumption of intent and users do not need to frame exact queries. Since results are subdivided into the main ideas contained in them, the user can easily identify the folder of interest and focus on it - thus engaging in 'selective ignorance' - ignoring results based upon knowledge rather than blindly"

But we also believe that it is one thing to organize search after having retrieved the search results and another when one has to integrate clustering at the retrieval stage itself. Given the problems faced by statistical a-contextual natural language processing technologies, clustering algorithms could really obfuscate search results, unless there is some revolution in using these technologies, current state of algorithms are just not sophisticated enough to deal with diverse set of information found on web pages.

Relevance Ranking Of Search Results, and Paid Vs. Organic Listings

It is important to note that all the facets of search we have talked about are so interlinked that many times it is difficult to segregate them. Understanding user's intention is closely connected with ranking the search results. Making the right connection between these two facets is important for search engines so that the user can make optimum sense of search results.

Since most of the search engines are engaged in making money out of the search results it will be very important for them maintain the distinction between algorithmic ranking and commercial ranking of search results. The issue of trust discussed earlier will largely depend on the ability of search engines to clearly demarcate the experience of search and its commercialization. The battle between paid search and organic search results is going to be fought along these lines.

Again collaboration, personalization and conceptualization seem to be the possible paths the search engines could take in order to achieve better ranking of search results. The biggest hurdle on the possible path to these innovations: search engines spammers, adware and spyware programmers and the craze for obsessive search engine optimization.

Possibilities in Search Interfaces and Information Display

Eventually search engines will have to find ways to innovate further from the existing list view. This does not mean the list view of search results is not useful, but it offers very limited possibilities and search engines will have to lead and nudge the users onto newer way of information usage.

On the other hand, all the attempts being made right now in alternative information displays and information visualization space appear to offer so cluttered a view of information that users might prefer to stick with the list view.

Future of Information Crawling and Indexing

As suggested earlier, perhaps the future of crawling and information indexing will largely be guided by the possible adoption of XML, RSS or Atom standards by site publisher and content management technologies. These formats could possibly embed the ideas of indexing and crawling into the site itself. This could mean that search engines don't have to do all this information indexing they do now to provide clean results. And may be this itself could provide a competitive threat to search engines? This could mean potentially anybody could build a search engine?

Future of Search: A Few Tangential Ideas

Having done some constrained analysis of possibilities in this field, may be its time to throw some slightly tangential ideas:

• How about search engine capabilities embedded in the hardware at the microchip level. Scalability of search engines with respect to information availability is going to be an issue so projects like this from Huazhong University of Science and Technology, in China could really mean a lot in this game. Or think about search embedded in storage networks built by companies like EMC.
• It seems that CCortex from Artificial Development is based on "Autonomous Cognitive Model ("ACM"), a realistic representation of the workflow of a functioning human cortex. The ACM may have immediate applications for data mining, network security, search engine technologies and natural language processing". May be some thing like this will bring a completely new dimension to search?

So whichever path search takes the future of search seems very exciting indeed!