Yesterday the Wolfram Alpha team announced the creation of their community:
* http://blog.wolframalpha.com/2009/05/18/announcing-the-wolframalpha-community/
They wrote:
“To that end we are officially launching the Wolfram|Alpha Community, which allows you to submit questions, ideas, and favorite inputs.
We already have a few static forms to contribute things such as facts, figures, and structured data or algorithms, methods, and models. The Community serves to supplement these types of feedback with a more free-form discussion among all Wolfram|Alpha users.”
Everyone knows by now I’m a firm believer in companies providing open, democratic and multilateral channels for users to provide feedback and to interact with the company. Given this opportunity, I suggested that the Wolfram Alpha team could look into providing answers to my test questions:
(1.) Who discovered radium?
(2.) Where is Atlantis?
(3.) How do we make gold from lead?
(4.) Can robots dream?
(5.) What is a sprite? [This is my trick question since ‘Sprite’ is a drinks brand as well as a type of fairy.]
(6.) When did Homo Erectus become Homo Sapien?
(7.) Why are we here?
(8.) How many light bulbs are there in the world?
(9.) Who is the Vitruvian Man?
(10.) Where is Schrodinger’s cat?
COBALT + ERIC SUGGEST I’M CLUELESS
In response to my suggestion, two users by the name of Cobalt and Eric wrote this:
“You misunderstand the point of Wolfram|Alpha I think. It is not a search engine like google nor is it a forum of expertise like answers.com.”
It is a tool that allows you to find and analyse data (i.e hard facts) from the web. It can only answer questions that have a definite answer or present data related to a subject. Things like the weight of 1g of gold or the average age in Australia. Questions with no definite answer such as the location of Atlantis and do robots dream will not and should not be answered as that is what google and the like already do.” — Cobalt
*********************************************************
“2. not a fact
3. not a fact
4. not a fact
6. not a fact
7. not a fact
8. not a fact
9. not a fact
10. not a fact
You don’t understand what Alpha is used for. If it’s not a fact, it can’t calculate it. Where is Atlantis? Mankind doesn’t even know if it existed, how the hell are is Alpha supposed to point it out? Can robots dream? It doesn’t create narratives, it gives you numbers in return. If you want to know why we are here, talk to a philosopher. If you want computable data, use Alpha. It couldn’t be clearer.” — Eric
MY RESPONSE: THERE’S RATIONALE IN TWAIN’S TEST
This is what I wrote in reply:
Unfortunately, Cobalt and Eric, you’re the ones who may be misinterpreting my testing approach. Stephen Wolfram, in an interview with Semantic Universe, notes that WA should be compared with the likes of Google and Yahoo! and not with HAL or Cyc, so it was reasonable for me to run WA results against Google’s and True Knowledge’s. Also let me give some context which may help.
I have a maths degree and have worked in banking, so I understand perfectly well the difference between calculable inputs to derive proofs and business models from information which is unquantifiable or simply has no quantity — such as “How is Michelle Obama related to Barack — which are questions needing answers of a qualifiable nature.
Now, Wolfram Alpha is marketed as a “computational knowledge engine” rather than a fact+figures finder/calculator so it’s supposed to be able to derive KNOWLEDGE not facts+figures alone.
Let’s tackle what logically each of my questions should have derived:
(1.) Who discovered radium —- WA gave the year but not the who (Marie Curie). Moreover, the expectation would be that the system would generate both a visual of the radium atom, some charts of radioactivity, a picture of Marie Curie and some facts+figures on the laboratory conditions of discovery.
(2.) Where is Atlantis — WA could have generated a series of maps not only of actual locations called ‘Atlantis’ (e.g. in South Africa and the US) it should also have produced geo-thermal images from archaeological expeditions that have tried to establish where Atlantis is.
(3.) How do we make gold from lead — instead of producing a “WA isn’t sure what to do with your input” the system could at least have produced some Periodic Table definitions of gold and lead, their reactivity with other chemicals and some paragraphs on historical attempts by people to try to make gold from lead.
(4.) Can robots dream — again instead of producing “WA isn’t sure what to do with your input” the system could have listed all the works of fiction by people who have actually existed (Philip Dick / Isaac Aasimov / Stanley Kubrick) who are factually connected to this phrase. After all, WA is supposed to apply NLP to derive what we mean by the inputs.
(5.) What is a sprite — WA produced a table of nutritional breakdown of Sprite the soft drink. In fact, apart from the faerie connection which is fictional entity, sprite is also a FACTUAL term used in computer graphics and the WA system failed to pick this up.
(6.) When did Homo Erectus become Homo Sapien — again WA issued a “WA isn’t sure what to do with your input” message. It’s an established FACT from anthropology and archaeology that in the evolution of Man, Homo Erectus preceded the emergence of the Homo Sapien. WA failed to produce a timeline graph of that evolution to help pinpoint whether that happened 500,000 years ago or 50,000 years ago.
(7.) Why are we here — again WA issues a “WA isn’t sure what to do with your input”. Fair enough, the system is not sophisticated enough to infer philosophical constructs yet; we are some way from truly consciously aware machines. Nevertheless, the expectation would be that some graphics of Big Bang Theory and the formation of the planets would have been produced.
(8.) How many light bulbs are there in the world — actually, this is a FACTUAL question. There are definitely numbers available of light bulb production, US expenditure on light bulbs per annum and how many light bulbs are used in each household per annum.
(9.) Who is the Vitruvian Man — unfortunately, Eric, you may not have seen sketches of Da Vinci’s masterpiece and which actually exist and are FACT-based. Instead of WA stating it “isn’t sure what to do with your input” the system should at least have generated an image of Da Vinci’s work. It could then have made the linkage of how the Vitruvian Man image has been applied in various fields of science — as clues to atomic structure as well as a reference diagram of human anatomy in medical science.
(10.) Where is Schrodinger’s cat — WA said it “isn’t sure what to do with your input”. This was the most surprising answer of all out of the questions posed. The expectation would be that the engine would at least interpret the question as one related to quantum physics and generate calculations and proofs attributable to Erwin Schrodinger. If it was even smarter it may even have done a compare/contrast with Einstein’s equations and Hawking’s postulations.
As for whether Schrodinger’s cat is a FACT or not, there are all manner of scientific phenomena that cannot be seen or established by the naked human eye (it’s somewhere else on the electromagnetic spectrum) for which generations of scientists have extrapolated proofs, corollaries and reductive provisos.
What matters in the question relating to Schrodinger’s cat is the fact that WA did not even produce an answer which said something like, “Schrodinger’s cat was a scientific experiment conceived by Erwin Schrodinger in 1935 in response to potential limitations in the Copenhagen approach and as a commentary on the ‘quantum indeterminacy or the observer’s paradox’. Schrodinger’s equation itself is applicable in wave physics, energy calculations of chemical reactions and is derived from the Hamiltonian and Poisson functions to produce:
(∂2Y/∂x2 ) + (8π2/h2)(E-V) Y = 0
where Y is Schrodinger’s wave equation.
X is the position of the particle.
E is energy in Joules per second.
V is the potential energy in Joules per second.
followed by various corollaries and supporting suppositions of the type similar to those printed in this UCLA paper:
http://www.math.ucla.edu/~tao/preprints/schrodinger.pdf
Even as the most basic answer, instead of “WA isn’t sure what to do with your input” the simple and FACTUAL answer would have been “In the Schrodinger’s cat hypothesis, the cat is placed inside a steel chamber” followed by some of those equations Erwin Schrodinger is famous for.
All of my questions are science-based and either already have definitive scientific proofs or are established hypothesis based on scientifically-derived means. This includes “how do we make gold from lead” and the evolution of Homo Erectus into Homo Sapien.
WA is marketed as a “computational knowledge engine” and on the basis of its NLP which can semantically derive what our questions mean. If I ask “Who discovered radium?” and the answer provided doesn’t even mention Marie Curie then there’s clearly room for improvement.
As I’ve written elsewhere, WA’s entry into the search/knowledge space is great for us all as information consumers, knowledge connectors and sense discoverers.
Of course it’s fantastic that a tool like WA is made available — not just for the scientific community — but for anyone who needs to crunch any form of numbers or needs a piece of knowledge to support, quantify, qualify and visually compliment their articles (whether that’s on the fluxing orbital paths of the planets or the score lines of the World Series for the last century or projected growth of the shrimp population in the Indian sub-continent).
Nevertheless, we have to identify and be realistic about its current limitations because only then can we as consumers have genuine “computational knowledge engines” which can connect facts+figures from different disciplines, make sense of the world around us (visible, invisible and maybe as yet undiscoverable) and perhaps find solutions to global common ills.
************************************************************************************************************
For me, Cobalt and Eric’s comments are interesting because both point to how users are perceiving and interpreting what these words mean:
* fact
* computational knowledge engine
* definite answers
Again, it’s relevant to semantics and the way we classify axioms via ontologies, taxonomies, folksonomies and other linguistic categoronomies (I’m coining this phrase, ha ha).
To be tic-lol (tongue-in-cheek, laugh-out-loud) I could ask, “Well, since we’re talking “facts and definite answers” how definite will Wolfram Alpha’s calculation of imaginary numbers be and has anyone actually scientifically observed them to establish them as a fact?” The reality is that imaginary numbers are philosophical constructs plucked from mathematical minds just as is most of complex algebra and even the existence or otherwise of Schrodinger’s cat (an example of quantum physicists’ paradigms), Atlantis (anthropologist-archeologists axiomatic construct of a ‘lost world”) and whether or not robots can dream!
Maybe it would have been easier if I’d asked the WA system something straightforward like, “What is the largest prime number in the universe?” or “What is the 123456789th number of π?” Ha ha. Actually not even the most powerful supercomputer has arrived at the definitive answer to the largest prime number on Earth much less the Universe! One day someone from the planet GYG is going to materialize and say that our Googol is only worth a 10 squared in their numerical scales, so the largest prime number is only 100 digits long or something!
In any case, the fact stands that the current Wolfram Alpha can’t apply its natural language processing to semantically extract that when I ask “Who discovered radium” it should give me an answer with Marie Curie in it in priority / precedence before providing me with the year of discovery.
It also can’t calculate how many light bulbs there are in the world. Clearly, some of the light bulbs in WA’s computational clusters haven’t been switched on yet to spotlight this missing information and the inputs necessary to generate an answer which would go along the lines of a simple equation like this:
L = n(A + B + C +….) + q(h1 + h2 + h3….)
where L is the total number of light bulbs in the world.
n is the number of light bulbs produced per year in each country.
A is country A.
B is country B, etc.
q is the quantity of light bulbs in each household (used and unused).
h is the number of households in each country.
By Twain deduction, Wolfram Alpha is not yet a “computational KNOWLEDGE engine” if it can’t differentiate that a sprite is either a computer graphic term, a type of tiny glowing faerie and a brand of soft drink and can’t make the neural nets connections I’ve noted above. It would probably help the WA team to communicate and market what their service can/cannot do if they called themselves a “computational DATA engine”.
In any case, people suggesting I don’t understand the nature or the application of Wolfram Alpha is good and healthy for intellectual stimulus and keeping my ego in check. It helps me sanity-check the approach of a Twain test and whether its validity stands up.
What matters most is that it will be interesting to experience how Wolfram Alpha, Google Squared et al develop and whether one day they’ll be able to answer my 10 questions — both by deriving facts as well as extrapolating the semantic and philosophical nuances of the queries.
Now THAT’s when we may see truly intelligent and proxy-consciousness agents…………