Hrm: Seems to me I've repeatedly read that the Turing Test is the best available for defining achievement of artificial intelligence.
It just occurred to me that the procedure that Shannon used to measure the entropy of English test is also a superior measure of intelligence: See how many yes/no questions the subject needs to reconstruct a given passage of natural language text.
This should be an excellent general measure of intelligence. Unlike the Turing Test, it can measure intelligence beyond the human level. It can quantitatively compare different -- even very different -- AI systems both against each other and against humans. And the test can be run entirely automatically, without need for a panel of human judges whose performance and opinion may fluctuate unpredictably based on season, weather, marital status and phase of moon.
It feels much cleaner than the standard IQ test model of making the patently absurd assumption that intelligence increases at a linear rate throughout life, so that a simple quotient can determine one's distance down the track from the starting gate. And it relates intelligence to things as fundamental and real as mass and energy, which I don't recall being done before. Where IQ is a purely floating-in-air measure relating humans to humans, the Shannon Test ties one into the relatively objective quantity of the amount of entropy in a given texual passage. E.g., we can prove things about entropy, which we really can't do about IQ.
I'd be curious to see how Shannon Test results correlate with standard IQ tests, actually. How -does- human performance vary over a lifetime? The Shannon Test can be given to illiterates or children. With a little calibration is should be quite cross-cultural, using texts in different languages describing culturally varied subjects: A few bi-lingual subjects should suffice to establish an effective mapping. It might possibly even be adaptable to measuring the intelligence of nonlinguistic subjects if a suitable replacement for natural language text can be found -- one can imagine a Shannon Test for chicken intelligence based on pecking one of two buttons in order to continue an evolving pattern of 0s and 1s. I don't recall any studies of the spread of intelligence in chickens, and certainly no effort to breed for it. :) With a little work, one might be able to apply such a test all the way from C Elegans to humans (and on up to elephants and whales :) starting with simple patterns such as alternating 0s and 1s and working on up to something like binary-coded prime numbers. Actually putting cats, dogs, chickens and minnows on the same scale might be interesting. :)
As a final observation, the fact that the standard IQ assumption of linear increase in intelligence works reasonably well is interesting in and of itself. One thing that does increase linearly over time is experience, and this could be taken as evidence that intelligence is extremely experience-driven, and thus increases in fairly direct proportion to experience. Compare this, say, to models of intelligence centered on clever search algorithms or brute CPU power or such: Based on them one might expect intelligence to stay rather constant over time, give or take an occasional nonlinear jump as a new algorithm was discovered.