Most of the search based on the Internet has been reference based. Google, the forerunner of all searches tends to give out results that are relevance oriented. Thats is, you type in a keyword and Google performs the search based on these keywords alone. Instead of using HTML, the delivery protocol for web pages is a desire for a new, feed-centric protocol, that is RSS. To search chronologically-ordered content, a relevance-based search like Google destroys the chronology and is inappropriate. More innovative in the search technology are sites such as PubSub, Technorati and Newsgator.

There are two ways in which a user can search the Internet for information:

Retrospective Search: This is the traditional form of search as performed by search engines such as Google, Yahoo! and other search engines. This method is called ‘searching the past’ where information is gathered by spiders, net crawlers and other forms of data entry. This information is sorted into a historical searchable selection. In such a system of search, queries are constantly changing and can be considered to be static. The results of these queries may change over a period of time but they are considered to be static because the results can be gathered only against the background of information
that is already available on the web.

Prospective Search: This new method of search was introduced by PubSub. It is termed as ‘searching the future’. Information is gathered by a variety of methods against newly updated documents. Every time a prospective search is performed the queries are stored in the database while the results are discarded. When a new document matching the query is made available, the user receives an update. In this system of search, it is the queries that are static while the results are dynamic.

With information on the Internet constantly expanding, searching for information can become a cumbersome, tiring and confusing process which takes up a lot of time as the users sift though pages and pages of information. One could argue that retrospective search is more comprehensive and heterogeneous. While this may be true, prospective search is useful in that it reduces the time taken by users to search or repeat search for a particular bit of information.

Exploring Keyword Search

Merely using keywords to locate data isn't comprehensive enough. More often than not, you receive results that you do not care for. It is advisable therefore, to use key phrases instead of keywords alone. Phrases are less generic and more descriptive. This is the kind of language that a search engine understands.

Prospective Vs Retrospective

Prospective search is not meant to be viewed as a replacement for traditional methods of search. It is useful only when it is used to complement retrospective search. While retrospective search allows users to source information that is already known and documented, prospective search allows the user to be updated whenever a new piece of information pertaining to the query is added to the wealth of the Internet database.

Prospective search throws many doors open for those in the business of content creation and management. It means that newly generated content is immediately forwarded to those who have expressed an interest in the subject matter.

Prospective search is also changing the way the Internet works by making the web more personalized. Users can receive only the information they are interested in rather than having to go through large amounts of data that they may or maynot be interested in at all. Using prospective search their query is stored and the user receives a feed or an update every time something new (pertaining to the query of course) pops up on the Internet.

Instead of going through a lot of irrelevant 'junk' information it is simpler to use prospective search offered by PubSub, Technorati, Feedster, Blogpulse and Bloglines. These services are currently offered free. In order to make this kind of search more innovative, prospective search engines could team up with library databases such as Proquest whereby a user will receive a feed once any new information is available.

Conclusion

Prospective search will make the job of information hunting and gathering less time consuming and more accurate and updated. In fact the idea that users will be able to access the latest and updated information is its greatest strength. Prospective searches will be most useful for business purposes where the availability of new information is key to the progress of an enterprise.

However, it is still important to access information that is already known and this is where retrospective search holds sway. Prospective and retrospective search are complementary to one another. One cannot be used in place of the other. The future of prospective search is largely merged with the continued usage of retrospective search. The user will most probably use a combination of retrospective and prospective search to capture data.

Key Issues

To analyze these arguments, one could either talk about the companies mentioned, or delve deeper into the abstracted issues. The latter appears more logical as it allows us to look at this discussion from variety of perspectives.

Some of the key discussion threads center - implicitly or explicitly - around the following issues: online information availability, media, information management technologies, paid search, pay-per-click advertising, information consumption, lead generation, lead qualification, lead conversion, information vs action, human-computer interaction, and eventually technology vs human intervention.

Online Information Availability and Information Consumption

The stark reality is that there seems to be a gold-rush towards making more and more information available online. As this information is made accessible by search engines, and they build business models based on making the information available, the distinction between media and technology is bound to stare them in the face. In a way, this debate is not new - from the beginning of printing presses, technology has had a profound effect on how media is created, marketed, distributed, and consumed.

Online availability of media is in a way bound to create issues on use, consumption, and exploitation of that information. You can see issues lurking behind in terms copyright (e.g. Geico v. Google trial) or free vs paid access to information, the question of information-hoarding monopoly, and so on.

When information is being made available, the act of making it available comes with certain amount of responsibility. We have seen issues around search engines trying to filter out results in terms of crime, political hate-campaigns; we have seen issues around aggregating news in countries like China, etc. Information availability will also (or has started to) define how we consume online information vs the nature of information use off-line, we already see that in information-intensive markets, these two are merging together.

Information Management Technology vs Media

The technology vs media argument stems from the fact that search advertising, has become a bigger revenue model than access to media. So far very few companies have been able to make money through other models of media distribution on the net.

So may be the distinction between a media company or a technology company is not very relevant till newer revenue models emerge to exploit the information that is being made available. It is possible that the real distinction may be the type of inter-mediary a company wants to become. Does it want to be an advertising agency, a content provider, or a combination of both? Alternately, there could also be another path where one starts with advertising and then gets into direct-selling of media.

Online information consumption now provides a value-chain starting from making the information available to creating new types of information; to exploiting that information towards business gains. Revenues right now seem to be around the aspect of making the information available, and not so much in information creation or information selling.

It also makes sense to understand what a company wants its users to do with the information being made available? Is it ok with the company if the users go away to the desired destination or it wants them to engage with that information: through paid search based model the users ramble away to their desired destination, whereas through a more media oriented model the company can hold on to the users and make them do things with the information it offers, or owns.

Besides, anybody who has worked in a technology or a media company would understand the world-view of different groups who run the company. Technologists, for instance, have a particular viewpoint and would naturally look at solving problems with technology-based automation. So whether one wants to agree or not, the differences of world-views do exist and come into play. It seems that companies that have succeeded are the ones that do not impose technology solutions onto the problems, but allow the solutions to emerge out of real needs of users.

It is interesting to watch the recruitment trends in these companies. Technology oriented companies are either hiring for technology or sales. Whereas media-oriented companies seem to be including editorial staff as well as domain experts in content category like music or broadcast - along with their technology hiring.

Technology and Human Intervention

If a company is in the business of allowing people to generate leads through search technology, human intervention will be last on its agenda, except when things get out of hand, or when some very serious globally accepted ethical issues are involved.

Despite the opinion expressed to the contrary, information management technology can serve as a de-politicizing factor in some very sensitive subjects. If one leaves aside the politics of selection (e.g.. selection of website as news-worthy or not) and the information hegemony, making information available can be done in a fairly neutral manner. Basically search technologies are about indexing information; they could be neutral to what information is indexed.

But, again, if one is in the business of content creation and content selection, then technology will have to be governed by the editorial control. Most probably there are no two extremes in this case, in order to provide a functional solution in information management one would need to strike a balance between technology and human intervention.

This entire debate is going to be interesting in the near future as major players start making the distinction as a matter of their competitive advantage, or start merging the distinction to outsmart each other.

K-Praxis is available here: http://www.k-praxis.com

About Textual Analytics Solutions

Textual Analytics Solutions Pvt Ltd (TAS) is a Bangalore, India based company which is working on innovative technology solutions in the area of text analytics and Natural Language Processing.

Text analytics, as a technology solution category, goes beyond the traditional text mining solutions and focuses on analytics of information pieces extracted from within document corpuses or large set of websites, etc.

Company's proprietary technology and methodology uses various approaches that extract, process and analyze textual information from a variety of perspectives. The company has built custom solutions on top of the core platform to enable superior decision support and business analytics capabilities for the users and business.

Evolutionary biologists indicate (ending all debates on nature-culture) that the brain has a modular learning function, where it can absorb a stimulus external to itself and process it with its own unique genetic makeup. It has thus been said that no two individuals react similarly to the same stimulus in the external environment. Moreover even in the same situation the same person would react differently at different times. On keying in these factors one gets the feeling of an anarchical and chaotic mind. It might appear that the human mind defies being pinned down to any patterns, principles or rules. This, however, may not be the case.

Patterns, Rules and Order

Though there are no easy patterns or rules to understand the functioning of the human mind, it is far from being entirely unpredictable. In a given situation, it is possible to predict (at least approximately) how an individual would think, feel or act. This kind of prediction may not correspond to the study of a behaviorist school of psychology since it is not necessarily a clinical response to a stimulus. This patterned reaction may, however, be the result of mere habit or custom. A fairly well established custom or habit is the best evidence of a self evolving system.

Language as a Self Evolving System

One of the best examples of a self evolving system is a human language. While there are several theories to explain the evolution of language, it has largely been agreed upon that it is not the work of a "plan", that it is in fact a self organizing dynamic system. The functioning of the mind shows that complex language structures are processed as not just as clusters of meaning but as clusters of relations within the language system, that is to say that the functioning of the mind is at the level of the meta-language. Thus new terms or newer sentence constructions which have not been encountered before are easily grasped. If a complex structure like a language can evolve without the intervention of human intention then similar systems could be set rolling.

The Web of Networks Outside of the Mind

Self evolving systems which replicate the functioning of the mind as it understands and processes language are already in the making. One such system is that of the web. The web has been able to create and replicate the functioning of the mind, not in a single machine but however in creating something of a global mind, or as Gazzaniga calls it, 'the social brain'. The development of the global mind depends on the evolution of distributed systems that function as the global equivalent of consciousness, memory, learning, perception, introspection, planning, creativity, and behavior.

The Web System

Based on the networks of the mind a mind writ large seems to be in the offing in the form of the internet. This system like the working of the mind requires no human intervention, at least not at an intentional intervention. The system is increasingly an open system where the input of each additional mind can enhance or affect the system. Moreover while no one individual can move to take credit for building this system, it reflects the inputs of all. The networks thus formed are neither entirely accidental nor calculatedly instrumental. It is the development of such a system, remarkably close to the mental web that is on the horizon now.

It would not require an accurate quotation of statistical data to state that the drop out rate in e learning programs is much higher than in traditional educational systems. Whether this has to do with the lack of the gentle human presence (of teachers and peers) or with the community of students who opt for e-learning programs as opposed to the ones who choose the traditional educational programs, is unclear. Since e learning is often considered a successor of the distance education model, it continues to be viewed as a secondary choice to classroom learning. E learning, however, is more self-paced and hence more convenient. Moreover it also reduces the embarrassment of accessing education at any age and from any location.

The Cheaper Option

With a growing concern regarding funding (especially of higher) education, e learning seems to provide a more viable option. Reusable Learning Objects (RLO) and Shareable Coursework Object Reference Model (SCROM), which are used extensively in e learning not only, reduce the costs of the learning package but also the efforts on the part of the instructors/designers which in turn reduces costs further.

Automated Anxieties

While there can be few doubts as to the advantages of e learning in relation to traditional methods of learning, in speaking of its desirability as a system of learning, there are several issues. In accessing the e learning models, the student would encounter certain difficulties or hindrances. Designers of these models try largely to predict these difficulties and attempt to create within the model itself, provisions by which these hindrances can be removed. The difficulty however is this: as the learning models are used more and more, newer and previously unpredicted difficulties are bound to occur. These may be in terms of incomprehension either of the technology or even of the content. How is e learning bracing itself to counter this challenge? Can the learning models be made sustainable such that there is little need for continuous human evaluation?

The Human Hand…

It would be unfair to the designers of e learning courses to expect them to create learning models which would preempt every difficulty that that the real individual accessing it would have. Furthermore it would also be a complete misjudgment of the eccentricities of the human mind to expect it to function smoothly in accordance with a machine. While the learning model can record the anomalies and difficulties, there would always be a need to evaluate and continuously upgrade the model creatively. This revision can only be done by human intervention.

The interaction between man and machine is never unmediated! Even the simplest machines would require instruction manuals and service-personnel. Similarly, e learning packages would also need the assistance of technicians who are able to comprehend the glitches in these models and revise them to suit the ability of the learners.

The use of technology for teaching and instruction has been around for several years. From learning blocks to audio-visual material, there has been a variety of non-text, non-lecture based material that is used as part of the education process. With e learning, there is a certain principled change that can be observed. The materials used for teaching are now standardized such that they can accommodate the specificity of different contexts and still retain a uniform pattern.

Learning Objects

All 'objects' that would assist the process of teaching/instruction and learning could broadly be called learning object. In this context, however learning objects refers to specific technology-driven reusable learning models.
These objects or models form the scaffolding upon which individual learning modules can be organized. Learning based on these models is standardized according to the best cognitive patterns by which an individual can be instructed. Learning objects involve an organization of the metadata of the content of the course or module. Thus while there is a certain standardization, there is no loss of information, no coercive or hegemonic syllabus which caters to only a certain group of individuals or an arbitrary selection either of the course content or the teaching practice.

Standardization of learning patterns

Learning objects standardize the learning models and are instrumental in regulating the course content to a large extent. There is however a more important issue of learning patters that differ according to different learning systems and different forms of knowledge. With e learning, all kinds of learning takes place in a similar manner. For example, in the traditional methods of schooling, multiplication tables are learnt by rote and algebra is taught according to theorems, syllogisms and on principles of logic. With e learning, however, all kinds of education is imparted in a similar mode. How much does the manner of learning inform the process of learning? Can the learning pattern be distinguished from the actual knowledge gained? This is a question that can only be answered by after a rigorous experimentation of the e-learning model.

Reduction of Disparities amongst Students

One of the most obvious advantages of the e-learning system is that it can efface the socio-economic, racial, national, language, gender and such other disparities amongst the students. Each student/learner is only treated as an individual with certain leaning patterns and differentiated capabilities. Everyone can access as much information as they want and work as per their own optimum schedules.
This however would happen only in the e-learning utopia.

Realities of a Non-Virtual World

Accessibility of the technological material may be designed to cater to a uniform audience but in the real world, that which each individual is able to access depends upon his/her situation in the non-virtual world. Steeped deep into the traditional methods of learning, it is not far-fetched (though it is indeed ironic) to see people take print-outs of material being made available (with great technological sophistication) digitally.

Standardization, like equality may be a desirable ideal to strive to. What does it mean, however, to standardize methods and content for heterogeneous individuals with disparate levels of accessibility is a question for e learning to answer as it develops and nuances its own packages.

What things would look like in e-learning utopia:

Everyone would have unlimited (or at least sufficient)       internet access

Method of learning would not be deeply linked to the content which is to be learnt

Newer forms of leaning could be learnt easily

How things stand in the real world:

Internet access is a rare and highly priced commodity

The method is also learnt along with the content; in fact there may be no distinction.

To learn the new methods of learning is in itself a task

After a chunk of data has been collected, the immediate task at hand is the sorting and organizing of this data. Usually the individual researcher who is faced with several kinds of documents is compelled to use parameters of classification, where documents are either put aside for one particular reading or the other. In the classification of the material which is collected for a particular study, there are several nuances which are overlooked, because of the size of the document corpus and the prejudices of the researcher

The size of the document corpus: While a smaller document corpus is inadequate to draw out scientific inferences, a sizable document corpus creates other kinds of problems. Even a diligent and meticulous researcher would not be able to capture all the nuances of the text and thus overlook some important evidence.

Prejudices of the researcher: Due to the inevitable pattern in the process of human understanding, a researcher would only be able to look at a given body of evidence in a particular manner. Though (s)he may not overlook any aspects, the varying facets of the same text may not always register in his analysis.
It is here that automation can again brush over the human failings. Clusters of data can be formed based on parameters which are not predetermined by the researcher but generated from within the document itself. Thus what is presented at the end of the classification is not only sorted data but also clues of the parameters which are generated through a scientific method.

Hypothesis

Very little has been said about the generation of the hypothesis. It remains till today, one of the most unscientific stages in even the most scientific experiments. For example, (going back to our favorite scientist) Newton hypothesized gravity because an apple fell on his head! Beyond creativity and chance, (which is no doubt a bonus) that is unpredictable and unscientific, how does one generate a hypothesis on the basis of a given data? The automated clustering of data that is generated, throws up several parameters or grids within which data is classified. This provides the clues for the hypothesis.

Interpretation

The way in which the hypothesis is drawn out from the given clusters of opinions, allows for an understanding based on the criteria drawn out from within the data rather than one that is based on the creativity or insight of the researcher. This eliminates most of the biases and prejudices that the researcher may inadvertently carry with him. Other than that, the automation of the classification and hence the generation of hypothesis also leaves room for an interpretation given by the researcher. The terms of the study are, however, not dictated by the automated process. The classified and sorted documents in fact present a neatly classified bundle of information which allows the researcher to interpret and present an opinion generated out of a scientific analysis.

A scientific study was often contrasted with the study of the social sciences. It was said that the norms of the social sciences were not as rigid as the laws of the sciences. The sciences (i.e. the natural science) could have laws which would hold true for all times, which would not be affected by the researchers biases and could be verified by objective tests. Over the last few decades this notion is loosing validity rapidly. Post-relativity studies, it has been asserted that even the most scientific of disciplines, physics could involve unstable laws. In fact Einstein's relativity has in effect proved the premises of Newton's theory incorrect. The laws of science then, are not universal, at least not in terms of time. A deeper look at the Newton-Einstein controversy would reveal that the two great physicists were in fact not even attempting to explain the same reality. The universality of space also is thus defunct. What then can be called 'scientific'? Or should we slide to the extreme relativist position claiming that there is no truth or that there is truth in everything? Should the attempts at finding an 'objective' analysis be given up?

The Dilemma of the Method in Academic Research

Now that the hegemony of the scientific method no longer exists, how does academic research proceed? There is a huge amount of academic energy invested in the question of methodology. In the absence of a written down thumb rule scientific method, what will come to become an acceptable method, or at least an acceptable premise to validate academic research?

What then will make a Method Scientific?

That there are methods, employed by certain disciplines which yield more acceptable, more objective and more stable results than some others is undeniable. The salience of these methods may not be in the features that we listed above (objectivity, universality and verifiability) but in features other than these.

The Fluidity of the Scientific Method

The humanities choose to call themselves the 'social sciences' since the method of their investigation is deemed 'scientific'. The exchange between the sciences and humanities is however not only one-way. The sciences, especially after the Chaos theory, Relativity and Quantum studies, have close links, methodically with the social sciences. It is no coincidence that relativity and the stream of consciousness novels were expounded at the same time. What we then call a 'scientific method' has elements of both the social sciences and the natural or pure sciences.

What about Technology?

While the method of science has been accepted by the humanities and therein also modified, technology seems to remain fairly untouched by the revolutions that the idea of methodology have undergone. The technological bend of mind continues to adopt the method of statistics and verifiable data.

The Importance of Data in Scientific Method

In the previous article we had reflected upon the uses of data and standardizing of data in academic research. The problem of data is perhaps one of the crucial features which demarcates a scientific study from a non-scientific one. The sample, field or data that is used in the proposed study is crucial. Often due to a bad sample or inadequate data, the inferences of the study are incorrect. One of the ways in which the adequacy of data can be determined is by having a through selection of all possible material. While this is not always possible, automation and technological advancements can correct this error. Where human analysts can only analyze a limited number of documents within a given period of time, a similar automated system can accomplish a much greater number in the same time.
Structured information: The other advantage of automation is to provide data which is structured on the basis on a common parameter. This reduces the biases of the researcher and presents a uniformly and coherently structured data.

Random Sampling of Data

All documents/information on a given subject cannot be exhausted. Random Sampling is a well-known method of selecting data for analysis. The assumption of this method is that if a random sample from a given body of material is analyzed it is most likely to reflect the entire population/material. Random sampling
has however, recorded far too many misgivings, one of them being that the sample is actually not random enough. When it is selected by a human researcher, the chances of a design (based on intention) creep in.

Conclusion

There is no grounded method of science. It is only in the classification and analysis of data that we can attempt to find a certain scientificity. At the early stages of research, it is evident that automation can reduce several of the difficulties of conducting a study, making it more scientific and keeping a very little margin for unscientific elements to enter. It is at the second stage of research, the level of analysis that we will again reconsider the role of automation and technology in maintaining scientificity.

With the advent of the Internet, the academic field too has been bombarded by excessive amounts of information. All information cannot and does not become available directly as knowledge (thought students will continue to lift papers off the Net and pass them off as 'researched work'). And yet there is no doubt that the information made available in the cyber world is extremely useful to the research community. There is a need to understand how information, sometimes even a surplus of information becomes useful for producing knowledge. There are ways of understanding 'information'- one from the social context and another from the technological context. Information at an everyday level refers to a bit of factual knowledge. One can call this kind of information, in order to reduce confusion, data. However, in technical terms it refers to the ordered procession of signals in a transmission network. The information in this context is not meaningful information, and has been understood as syntactic rather than semantic. Information of this kind finds it's meaning in the technological context of translation, transmission, and reproduction of signals.

Data and information

The factual material available on a particular topic would constitute the data or the datum field of the subject. The processing of this material would enable it to become information on a given subject. It is the conversion of mere data into information that becomes a matter of interest to the world of automation and artificial intelligence. Moreover it is only information that is sorted out based on certain criteria that can become useful for knowledge production.

Too much data?

Data, by its very definition, is abundant in quantity. The enormous pool of data available on a particular topic cannot be sifted easily by human effort. Automation would enable primarily a sorting out of the material. Flamenco, a search program that lets users browse a digitized collection in ways that are similar to a stroll among the shelves of a library, uses cross-linked categorization so that when people clicked on one category, they not only see the images within it but those in related categories. This allows not only information to be made available but also interesting correlations to be formed within different categories. Bruce Horn, the founder of Ingenuity Software in Mammoth Lakes, California, has created an information management system that lets people individually tailor and cross-index all kinds of files. This system collects and cross-links references to any subject that is identified by the user. These systems assist not only the management of too much information but also enable interesting correlations that could sometimes go unobserved despite human discerning.

Scattered Data Assembled together Categorized Classified Information made available. Final Information: sorted along various dimensions.

Standardizing information

As newer and newer mediums of information and information storing (archiving, scanning digitizing) are constantly evolving, there is a need to standardize the information. Unless the information is made uniform it is not possible to present it along the same horizontal search. Digitizing and automation would also assist in providing this uniformity to the information.

And still a lot more…

While automation would render redundant a lot of human labor that goes into say data collection and analysis, it in fact opens out huge possibilities for research. Pre-automation, it was humanly possible only to access and analyze a limited number of materials on basis of which knowledge would be produced. With the mechanization of this process, infinite documents could be used. The task of the academic is thus more enhanced as well as made more competitive and meaningful.

Since distance education and correspondence courses have existed long before the introduction of computers, online education (at least in its initial stages) has modeled itself on these existing methods of instruction and learning. Almost any field of instruction can be learnt outside of a real classroom and in the virtual space. For example the Jordan Park Church of Christ offers a Bible Study Correspondence or Online course . Similarly there is an online program that teaches something as skill-based as needlework. While the methods of transmitting knowledge do not change the medium of instruction enables several new things. But does this new medium incorporate the older forms of teaching? Does it rely on their age-old experience or does it usher in the new not looking back?

Simulating the Classroom?

It is not unnatural for Online Education to simulate the classroom in its initial stages at least. A fixed time to meet in a virtual space instead of a real classroom is often thought of as an interesting way of conducting online courses. Chat sessions and threaded conversations allow the student to 'converse' with their instructors or teachers. Moreover, most of the online courses have an 'orientation program' where students are physically present to be initiated into the course. Furthermore several courses also insist on a 'contact program' or a mid-term/mid-semester seminar where again the students are to be physically present. Similarly students are asked to form study groups or group up with friends so that they can assist each other like they would in a classroom.
While the designers of the online education courses do realize that there is a need to reduce the impersonal nature of online education so that the student can have a feel of the real classroom, their use of the technology is nothing more than a medium of instruction and a form in which the transfer of knowledge can take place. The traditional knowledge is packaged in a technological gift-wrap.

Systems of Knowledge and the Knowledge of Systems

The medium of instruction and the body of knowledge seem to exist fairly independent of each other in the case of online courses. While the content of the course remains largely unchanged, technology becomes a part of the process of transferring that content. In most cases the salient features of this technology go unnoticed. It is at best tapped to communicate the information faster or more easily. There are, however, ways of learning which would not distinguish the form from what is learnt. E learning is one such experiment where one not only learns the technology to learn what it can teach but also learns the technology while learning what it is communicating.

E-learning

E-learning effaces the distinction between the form of communication and the context. While most definitions of e-learning foreground the "use" of technology in learning, an important component of e learning is also the deep bond between what is being learnt and technology. Technology not only enables the communication of the information but also transforms and enhances it.

Standardizing the Best Practices

E learning encourages a more interactive and collaborative way of learning. It not only instructs but also dispenses information freely, allowing people to experiment with their own methods of learning. Moreover since the world has been brought closer by the Internet technology, these methods and practices can be standardized. People are able to see a variety of methods and judge which ones have worked and discard the ones that have failed. Thus the best of the methods can be eventually retained.

Various Kinds of Information in a Uniform Grid

Information can be made available in any form; textual, visual, numerical, graphic Information existing in one form can be translated into another form with ease Information existing in one form can be translated into another form with ease

With the impending Google IPO, and the huge amount of buzz created by the search engines in recent months, we at K-Praxis ran a series designed to give a complete round up of search engines. We now present this series formatted as a complete report. This report is aimed at understanding the phenomenon of search from the point of view of an information searcher / information consumer /knowledge worker. This report is intended to offer him/her glimpses of various facets of information search - including: search technology, search process, search engine usability, search results display, information crawling and indexing process.

The main focus of this report is to cover search engines - although the report has several references to enterprise search technologies, the central theme of the report remains online search.

Areas Covered in the Report:

Online Search, Search Technology and Search Algorithms, Search Engine Usability, Search Engine Users, Information Relevance Ranking, Search Results Clustering, Information Retrieval, Search Engines and Unstructured Data Management, Text Mining, Natural Language Processing, Enterprise Search, Paid Search, Paid Inclusion, Search Engine Displays, Search Interfaces, Paid Listings Vs Organic Listings, Contextual Advertising, Local Search, Internet Yellow Pages, Information Crawling, Information Indexing, Future of Search.

Availability of the Report View HTML Download PDF

Index

1. Understanding users intention

1.1 User intention and search
1.2 User Intention Vis-à-vis Recent Search Engine User / Usability Surveys
1.3 Simple vs. Complex Queries
1.4 Complex Queries and Search Engines

2. Relevance Ranking of Results

2.1 Information Relevance Ranking

Google
Teoma/AskJeeves:
Vivisimo
Yahoo/MSN

2.2 Information Relevance Ranking and Enterprise Search
2.3 Information Relevance Ranking: Issues

3. Search Interfaces and Information Display

3.1 Online Search Interfaces and Information Display

List view
Title, Text Snippet or Summary
Other information about the URL

3.2 Visual Search and Search Engine Interfaces
3.3 Attempts at alternative Search Interfaces
3.4 Future of Interfaces

4. Paid Listings Vs Organic Listings

4.1 Paid Inclusion Controversy
4.2 Paid Inclusion vs. Algorithmic Search Results

Argument for maintaining the purity of search results
Demarcating Paid Inclusion
Are organic listings really organic

4.3 Organic Vs Paid Listing, Shopping Sites and Internet Yellow Pages (IYPs)

5. Crawling and Indexing

5.1 Information Crawling and Indexing: Introduction
5.2 Online Information Indexes: Do Bigger Indexes Always Mean Better Indexes?
5.3 Innovations in Information Crawling and Indexing

6. The Road Ahead

6.1 The Future of Search and Search Engine Users
6.2 A March Towards Understanding User’s Intention
6.3 Relevance Ranking Of Search Results, and Paid Vs. Organic Listings
6.4 Possibilities in Search Interfaces and Information Display
6.5 Future of Information Crawling and Indexing
6.6 Future of Search: A Few Tangential Ideas

In the first article we were trying to examine the evolution of virtual communities, realizing that they cannot be likened to real world communities in an easy analog. This series has attempted to explore, not only the saliency of the online communities but also to draw out features that will elucidate what these online communities are all about. An important facet of online communities is the dissemination of information. Information, its availability and dissemination becomes the cornerstone for the survival and proliferation of a virtual community. Moreover, information itself relies on these various communities as channels for distribution. Virtual communities can largely relate to each other only through the information that is made available to them. This is to say that even when individuals are relating to each other in a virtual space, they are actually only relating to the information provided to them.

Accessing information

Accessing of the pool of information which a community possess becomes extremely convenient when the community in question is a virtual one. The inequalities of the real world are almost entirely eliminated in the online communities, where the information transfer takes almost no time (barring slow internet connections and power failures) and can be accessed by all members of the community, including ones who do not have any information to share. Lurkers benefit as much from the knowledge pool of a virtual community as does the most active member.

Complex communities, increasing information

As virtual communities evolved, they became a very loosely connected collection of different information services and communications forums. Many members stay in on only one or two different domains within the community whereas a small number of people move ideas very swiftly from one group to another. Feeding information (even and often false information) to these people ensured it was spread throughout different groups. In the hands of a core group of cross-pollinators, both public and private communication channels are employed to distribute certain kinds of ephemeral information very quickly. The distribution of this information now becomes not just intra-community but also inter-community. This ensures the growth of the community. Often the community branches off into different groups which sometimes form alliances with other groups and sometimes break away form their parent group. This amoebic operation is almost entirely based on information and subdivision of information into more specialized niches.

There is no denying that there is a minimal need for authenticity that is felt when people relate to each other. It is understandably difficult to communicate, even if its only professional and impersonal information, to an identity that is ostensibly false. Given the easy steps in which on can create an identity, there is more than just a perceived threat of fraud and inauthenticity. It is possible to create multiple identities and personas with just a few clicks on the computer. Does it then become impossible to identify, as one would in the real world, a genuine identity

Identity markers

Clues to judge identity in the virtual world are rare and often not as straightforward as ones in the real world, but indeed they are not inexistent. There are methods and programs that are extremely sensitive to replication of identity, manipulation of server name, computer generated ID, etc. It is not very difficult, given the accelerated pace at which technology is moving, to chaff out the overtly fraudulent personas. It, however, remains to be sorted how one could deal with an adoption of a persona, which is not ones own. How for example would one know if a person is a 19-year-old female student in California, as she claims or an 80-year-old sheep-grazer in the Prairies?

Real identity

While there can be no fool-proof guarantee of a persons real identity in the virtual world, it should at this point be noticed that there is none of that in the real world as well. It is as facile to claim to have access to the truth about a real person in a real world, as it is to know a persona in a virtual group. Moreover, the interactions in the virtual world may not be dependant on the authenticity of persona at all. The identity claimed by the persona might infact be a useful clue as to the way in which a person wishes others to relate to him/her.

Obliteration of Identity and Anonymity

It would not be far off the mark to claim that identity in the virtual world does not connote faithfulness to the real. It is infact an obliteration of the identity and a freedom of anonymity. Whether this allows for a more democratic society remains to be seen… it is still too early to tell!

Whether the virtual world does (and should) simulate the real is a complex question, now ready to be relegated to the list of profound philosophical and unanswerable questions. By rephrasing this conundrum and narrowing its scope, one could ask how communities in the virtual world relate to those in the real one. For this one would begin to list the features of the real world communities and of the ones in the cyber world, thus enabling a transparent comparison.

Is the real world community an analyzable unit?

In attempting to list the features of the real world communities (barring organizations and instrumental communities) [link first article] it is observed that the communities in the real world are largely described only in negation. That is to say one can only understand the community in terms of what it is not. Thus the only thing that we can say concretely about a community is that it is difficult to understand it. A list of features of a community (say a group of friends) would follow a pattern like this:

• A single feature cannot identify the members of this community.


• There is no known reason for them to form a group


• There seems to be no apparent benefit that is accrued by being part of the community (other than pop-psychological reasons like need for other individuals etc)

How does the virtual perceive the real?

In designing virtual communities there is a blueprint of the real world communities, or at least an aspect of those communities. Connectivity is one of the most primary requirements of a community. Online communities are thus concentrating on finding ways in which members of that community can keep in touch with each other. The design protocols of online communities is thus to facilitate easier ways for people to get in touch with each other.

What bonds the virtual communities?

We have not been able to identify the bonds of the real world communities, however designer communities in the cyber world would have explicit and perceivable ties by which one would be able to identify them as communities. While one of the adhesives of a virtual community would be the nature of the information/field that brings the members of that community together, the other would be the identity of those members.
Information communities

A strong glue that keeps together various online communities, unlike real world communities, is a shared interest in the information that the community generates. Thus it is not uncommon to find online communities that talk of music, religion, pets, business, wars, etc. As long as the information shared and generated by this community is interesting and alive, the community continues to remain together.

Identities in the virtual world

The nature of virtual communities is such that one can speak of identities as being fluid, fraudulent or even therapeutic. Whether this affects, or shapes the online community in any substantive way is a question that the next article will deal with

The term 'organization' is often used interchangeably with 'community'. This implies that the formation of a community is in some ways organized or structured. If one thinks of the community as an organization, it leads one immediately to look for the structure or the force that is organizing the community. Furthermore one can look for motives and intentions of a community, treating a community as though it were one large individual or an aggregate of individuals.

Communities	Organizations
Communities are not necessarily informed by a predetermined end.	Organizations are formed with a particular goal in mind
A community is largely unstructured	The structure of an organization is explicit
The members of a community do not always display a minimum commonality other than that they belong to the community	The members of an organization are part of it because of their common identity outside of the organization as well
A community cannot function as the aggregate or sum of all its member	The organization can be understood as comprising of the desires, motives and intentions of all its members.

Instrumental communities

Leaving aside the community one is born into (where there is no choice) all other communities that one is a part of as more of less chosen. Why does one choose to be part of a community? What goal is fulfilled by being a member of the community? Once that is arrived at, can that be attributed as the 'motive' or aim of that community? For example a person joins a trade union or a lobby to improve his working conditions. The purpose of the union is thus an enhancement of the working conditions of the worker. The community can thus be identified as one fulfilling this purpose. Communities of this nature are instrumental and can be called 'organizations'. They have a marked and identified end and a completion, or dissolution of that end would probably result in the end of the community.

What about other communities?

It is difficult to explicate in a similar manner most other communities that one is a part of. For example the community of friends. It may be explained as is often done that one is part of a community of friends because one likes talking to them, enjoys meeting them. It is here that a crucial question for online communities could be posed: would one stop being part of the community of friends if one were unable to talk to them or meet them, for a considerable length of time? More explicitly, are the bonds of a community identifiable?

Replicating communities

The bonds and/or intentions of a community need to be identified for it to be replicated in the cyber world. The existence of online communities, however, does not indicate that the blueprint of a community is understood. How then are we to understand online communities?

Designing communities

Most online communities are designed and constructed with a specific goal in mind. Often this goal is to replicate the real world. Since it cannot do so in entirety the online communities abstract a particular aspect of real communities and try to replicate it.

In the forthcoming series of articles on online communities K-Praxis will undertake the analysis of online communities.