K-Praxis - Topic: Unstructured Information Management - Benchmarking the automated information management software

Benchmarking is usually a very significant indicator of a growing market segment and customers of automated information management solutions are now asking for the same. They need a way of evaluating these systems so they can choose better. Now the question is who will bell the cat and how?......

The need for benchmarking raises a few more questions than enumerated in a recent article in Intelligent Enterprise Magazine (It's All Subjective).

So how should we go about benchmarking unstructured data management solutions? Will it be:

Based on the speed
Based on accuracy
Based on the number of human hours it saves

OR there are some other norms that need to be developed to test the performance of these systems?

Robustness of the taxonomy
Robustness of the classification system that is used.
Scalability in terms of numbers
Scalability in terms of integration with other systems

OR Should we add the monetizing capacity of these systems to the norms listed above?

Besides there is another question. Who will bring in these benchmarking standards? So should we turn to organizations like "National Information Standards" or "International Organization for Standardization", for establishing standards for benchmarking the automated information management software industry?