@T: ALWAYS THE TWAIN

As promised, I’ve now plugged and played with the new “computational knowledge engine” offering from Stephen Wolfram, the British-born physicist renowned for having been awarded his Ph.D. when he was just 20 and being the inventor of Mathematica, a highly regarded research tool amongst the academic scientific community.

Wolfram Alpha had a soft launch on Friday 15 May following various media “sneak previews” in late April and it’s officially live today.

As per my previous provisos (please follow links provided at end of this blog), I’ve reserved assessment on the system until today so that it’s based not on being influenced by either SemWeb hype nor journalistic ignorance, but rather objective intelligence and a mild dose of wit. Some of the articles written to-date about Wolfram Alpha have been poorly researched and re-hashes of whatever PR has been issued by the company rather than informative. This has been unhelpful for determining where and how the various search / browse / info source systems are differentiated.

This is why I decided to do my own Twain test.

I’m interested in any AI / natural language processing / Bayesian alternative / neural nets / innovative attempts to connect and make sense of the vast amount of knowledge out there (within the Internet as well as as-yet electronically unarchived sources). I’m also interested in machines which try to discern meanings, wit and nuances from our questions in an equivalent manner to how humans do naturally.

Yes, I am aware that Stephen Wolfram has provided guidance that the system is not AI. That’s clear from this Semantic Universe article:

* http://www.semanticuniverse.com/blogs-i-was-positively-impressed-wolfram-alpha.html

Now, since Wolfram Alpha is based on Mathematica I decided not to ask any straightforward numerical questions; most of us are aware by now that it can deal with statistics, indices, trigonometry, Fourier analysis, Boltzmann constants, Boyle’s gas pressures, the various constituents in organic reactions, velocity in space calculations and other scientific and quant-oriented calculations etc. fairly well.

What would be more interesting is to really test its semantic extraction, linguistic deduction and visual generation capabilities.

Below are my 10 questions accompanied by screenshots of and commentary on the results. Wolfram Alpha is being directly compared with Google’s and True Knowledge’s which are its nearest competitors in this test. Incidentally, Wolfram has apparently noted that since it’s not AI it’s unfair to compare it with HAL or Cyc but compared with Google or Yahoo. I chose True Knowledge because as was (rightfully) highlighted by some friends elsewhere, this would be an interesting case study.

TWAIN’S 10 QUESTIONS

(1.) Who discovered radium?

(2.) Where is Atlantis?

(3.) How do we make gold from lead?

(4.) Can robots dream?

(5.) What is a sprite? [This is my trick question since ‘Sprite’ is a drinks brand as well as a type of fairy.]

(6.) When did Homo Erectus become Homo Sapien?

(7.) Why are we here?

(8.) How many light bulbs are there in the world?

(9.) Who is the Vitruvian Man?

(10.) Where is Schrodinger’s cat?

These questions may seem off-the-wall, but actually they’re not.

Wolfram Alpha is being built by scientists so information on who discovered radium, the evolution of Man, the alchemy of metals, scientific expeditions and application of geothermal imaging / satellite capture of potential sites for Atlantis, the proliferation of light bulbs which are the invention of Thomas Edison and the connections to Schrodinger’s cat should be easily surfaced by the system since it is all familiar territory to scientists, machines are built in the mould and mind of their creators and before we expect Wolfram Alpha to provide us with the missing links between Marilyn Monroe and baseball (answer: Joe di Maggio) in pop culture and sports arenas — two sample areas where ‘Der Spiegel’ has already highlighted Wolfram Alpha’s current deficiencies (http://www.spiegel.de/international/zeitgeist/0,1518,624065-10,00.html) — it should at least be able to deal better with queries associated with its forté, science. Even if it’s not precise and no visual graphics can be generated, the algorithm should direct us to sources where we can delve further and derive some answers.

Unfortunately, as can be seen from the screenshots Wolfram Alpha gives several “Wolfram|Alpha isn’t sure what to do with your input” answers.

Hmmn………

By comparison, as an example, Google recognizes the deliberate spelling mistake I made on the search term “VEtruvian Man” and asks me whether I mean “VItruvian Man” and it provides multiple links to suitable sites where I might find the answers. Meanwhile, True Knowledge doesn’t catch the spelling mistake and only offers, “It sounds like the vitruvian man may be a human being, organisation or other legal person that I don’t know about yet,” with a suggestion that I teach the system about him and input my knowledge in via the wiki.

As for the question, “Where is Schrodinger’s cat?” I suppose I could have been mischievous and asked, “Where is Schrodinger’s car?” instead to determine whether any of the three systems understood that it’s still a question about quantum physics and the ‘quantum indeterminacy or the observer’s paradox’. In other words, where we are and how we observe relative to the cat (aka the reception of visual particles into our eyes vis-à-vis the cat — which is an analogy for atoms invisible to the naked human eye, btw) itself affects an outcome, so that the outcome as such does not exist unless the measurement is made. Ergo, there is no single outcome unless and, I’d say, UNTIL it is observed.

If I put the cat into the car then the computational search engines will get even more confused……….LOL.

As it is, Google does a fairly decent job of discerning that I mean “Schrodinger’s cat” and even when I use “car” it provides me with a link to a YouTube entitled ‘Schrodinger’s car and parallel universes’.

Anyway, I hope you all enjoy the screenshots of the Twain test. Yes, and everyone should be aware that no media outlets have reported any “slow script” messages from Wolfram Alpha in their tests and I found one on my second query, “Where is Atlantis?” Oops, WA……

TEST RESULT SCREENSHOTS + TWAIN CONTEXT

(1.) Who discovered radium?

This is a straightforward question and the expectation would be that a diagram of a radium atom would be generated along with a map of where it was discovered along with a picture of Marie Curie. Instead, this is what each system produced:

(2.) Where is Atlantis?

This question resulted in a “slow script” message on Wolfram Alpha before it offered a map location of Atlantis as being on the South African peninsula coast. Like True Knowledge, it didn’t make that interpretation leap to identifying Atlantis as a potential mythical construct rather than an actual geographic location.

Google does make that interpretation leap.

(3.) How do we make gold from lead?

Ideally, the generated answer should show the historical (failed) attempts by various people to turn lead into gold — including the tales from the Hermetic schools of thought on this and those during Croesus’ age.

(4.) Can robots dream?

Would Isaac Aasimov / Philip K. Dick / Stanley Kubrick be impressed by the latest machine offering which produces results like these?

Wolfram Alpha says it “isn’t sure what to do with your input”. If I’d programmed the algorithm I’d make it respond like so, “Please ask us again in the morning after we’ve had the chance to sleep on it and think about it. Thanks!”

(5.) What is a sprite?

This is my trick question since ‘Sprite’ is a drinks brand as well as a type of fairy. Interestingly, Wolfram Alpha generates what appears to be the nutritional content of a can of Sprite but fails to pick up that the query may be about a glowing elfin creature that appears at the bottom of gardens in works of fiction. Meanwhile, Google pulls in some references to the term being relevant in computer graphics as well as the faerie references.

True Knowledge goes off-base and provides us with a picture of a ferret followed by a definition of it being a drink from Coca-Cola.

(6.) When did Homo Erectus become Homo Sapien?

Again, this should have produced a straightforward answer — either in the form of a timeline chart plotting the Evolution of Man which is being pieced together by anthropologists and other scientists or in the form of a textual examination into various Jurassic, Ice, Neanderthal, Paleolithic etc. ages.

(7.) Why are we here?

The greatest Existentialist question in our search for knowledge which perplexes philosophers, physicists, Presbytarians, polemicists, party people et al alike.

(8.) How many light bulbs are there in the world?

Let’s compare WA’s answer with Google’s. Google’s first link offers some data from Wiki answers on the daily production levels of light bulbs as well as the estimated annual expenditure on them whilst WA says it isn’t sure how to use the query input — which is, effectively, what True Knowledge also says. Interestingly in the TK results, there seems to be some kind of lag and it shows answers to the previous question of ‘Why are we here?’

Perhaps the lightbulb isn’t on in the TK thought engine — LOL.

(9.) Who is the Vitruvian Man?

Here, the search / computational engines should ideally have generated an image of Da Vinci’s famous drawing within the semantic context of the query. None of the systems did. Notably, neither Wolfram Alpha nor True Knowledge auto-corrected the deliberate spelling mistake of Vitruvian Man whilst Google did spot it and auto-amend.

(10.) Where is Schrodinger’s cat?

At the very least, Wolfram Alpha should have produced some equations associated with Erwin Schrodinger’s postulations as well as their interlinkages with Einstein’s, Stephen Hawking’s and the research currently being undertaken with the Large Hadron Collider. Plus the research from the Austrian university who managed to demonstrate time-travel by sending quarks over the River Danube.

Well, that’s how my mind would work if I was trying to locate Schrodinger’s cat and its connectivity trails…..

This is what the systems gave us instead:

CONCLUSION FROM TWAIN TEST

Google isn’t going to be killed just yet with today’s launch of Wolfram Alpha. Certainly, it’s helpful to see more visual and graphical representation of computed results but, then again,……….Kosmix does that better than Wolfram Alpha, Google and True Knowledge.

Once Google Squared goes officially live we’ll probably realize and accept that Google is keeping ahead of the curve by crossing Semantic knowhow with more visual knowledge representation techniques.

Companies should avoid marketing themselves or being labelled by the media / so-called search experts as “Google killers” and paradigm shifters before they’ve actually been tested by ordinary people like me or gone live. It’s critical to manage expectations and also to be more aware of the types of random and unexpected queries which do pop up in people’s minds and that they’d like the computational and philosophical derivations to.

Since the Semantic community is aiming towards artificial agents being able to answer some of the world’s most complex questions, systems should definitely be able to either answer questions like mine or, at least, provide appropriate and meaningful links to where else I can and should seek the answers.

In any case, innovations like Wolfram Alpha and other (non) Google killers can only result in keeping Google and other tech giants on their toes and result in improved search-browse-computational-discerning-sensemaking tools for us.

Hurrah! This is gr8 for us as information consumers, knowledge connectors and sense discoverers.

Yes and any company who’d like me to road-test their systems prior to launch should contact me, :*).

*************************************

For completion, here are the two blogs I wrote last week on the today-launched Wolfram Alpha platform:

* http://www.alwaysthetwain.com/blogs/2009/05/12/wolfram-alpha-cf-true-knowledge-non-google-killers/

* http://www.alwaysthetwain.com/blogs/2009/05/10/wolfram-alpha-objective-anticipation-analysis-please/

I just read some interesting commentary from some friends which compares what they’ve seen of Wolfram Alpha with True Knowledge so I re-visited the True Knowledge site. Clearly, TK has had a redesign as these two screenshots from 2008 and 2009 will show:

The soon-to-be-live-computation WA engine is launching with the same color scheme as the old TK site as well as Primal Fusion’s choice: faded orange.

In our WA compared with TK analysis, we should be aware of and note that systems are built in the mould of their creators and their pre-orientations / pre-dispositions / accumulated pasts. This helps us to contextualize the systems, what each can do and why they’re constructed in the way they have been and are being.

The background of founders can provide us with clues on likely development and strategy of the platform.

From what I can make out from WA’s video presentation, Wolfram’s solution takes its lead from Mathematica and other natural sciences databases. In essence, it’s like taking an online calculator that can generate visuals of trigonometric function graphs (like my Casio 5100FX did when I was a teenager) crossed with elements of:

* Bloomberg + MS Excel + SAS (Statistical Analysis Software) to generate economic charts

* some biochem modeling software

That’s why ‘Der Spiegel’ (http://www.spiegel.de/international/zeitgeist/0,1518,624065,00.html) can identify WA’s limitations in information availability/accuracy on politics, popular culture, sports etc. It’s not the natural or default orientation of Wolfram’s team, who spend more time thinking about Fibonnaci and Feynman than Britney or Barack which is what Tunstall-Pedoe’s team does. William Tunstall-Pedoe, the founder of TK, is from a journalistic background so his natural information orientation would be towards what’s published in most papers (politics, popular culture, business, sport, etc.) and that’s the direction he would most likely direct his team efforts towards.

In the greater schema of the Web, my observation is that WA’s launch reflects the trend of commercializing and hybridizing previously closed niche academia sources like Mathematica for the masses. We also see this when, for example, Google takes software that was in architectural niches (e.g., Autocad) and creates free tools like Sketch-up and now Google Draw.

Personally, I’m in favor of this trend. The question will still arise for WA, “How do we make money from our platform?” but they seem to have some cost per embed of a WA-generated search / graph in their business model.

With TK, another important distinction is that its wiki capabilities allow for collective correction. We read the definitions / links provided and we can apply our naturally accumulated knowledge to orientate and refine the definitions / links provided, according to our semantic (aka linguistic) interpretations.

With WA, there may be less scope for collective correction. How many people are going to use pen and paper to check that the integrals and statistics generated by WA are accurate?

In due course, Google will probably release something which is a 3rd way of both: wiki, visual knowledge representation and semantically-linked facts+figures. In fact, it already does in some form with Google Finance:

and now its recently announced Public Data Search capabilities:

Moreover, contrary to misconceptions (or rather lack of proper investigation by some quarters of the Semantic and journalistic space) Google is interested in and has been actively building teams with semantic knowhow for several years.

I wrote a lengthy, objective and well-researched article on this topic last year. Unfortunately, I made the mistake of posting this article on a certain SemWeb platform which I’d entrusted with the safekeeping / stewardship of my and my friends’ content on its public platform. Instead of reciprocally honoring that act of trust, said SemWeb platform’s coding was so awry and poorly architected that they deleted 8 months worth of our collaborative content, including that particular post examining Google’s interests in the semantic space.

Therefore, my hard work on the issue is lost indefinitely — despite the CEO’s non-performance of his own promise to restore our content.

This is personally annoying since my article cast a contextual light on whether any of these Semantic offerings springing up are genuinely paradigm-shifters and “Google killers”. They cannot be Google killers if the basic assumptions about Google not being actively involved in utilizing Semantic knowhow and tools is either fundamentally wrong or flawed.

Alas, I cannot now reproduce that article and the links which I found about Google hiring teams from known Semantic Web-related techco’s like CYC. However, I can point to some articles from Read/Write/Web this January 2009 and from eweek.com in the same month which point towards Google having and progressively incorporating semantic search features:

http://www.readwriteweb.com/archives/google_semantic_data.php

http://googlewatch.eweek.com/content/google_search/yes_google_is_doing_semantic_search.html

http://googlewatch.eweek.com/content/google_and_semantic_web/google_ceo_hints_at_semantic_contextual_search.html

http://www.pcworld.com/businesscenter/article/161890/semantic_search_could_secure_googles_future.html

http://googleblog.blogspot.com/2005/04/just-facts-fast.html

As I note, I wrote my article last year — May 2008 — several months before R/R/W or eweek did. That SemWeb platform, in their inconsiderate and wholesale deletion of users’ content, is responsible for my article not being available for others to use as a public and democratizing information source that would put into perspective whether any SemWeb offering is a paradigm-shifter or “Google killer”.

From what I’ve road-tested in the SemWeb space to-date, none of them are.

Google remains ahead of the curve both technically as well as the way in which they service and market to users. Certainly some of the businesses and their tools could be better integrated but, nonetheless, the key components remain technically more interesting than those offered by wannabe “Google killers” to-date.

Specifically wrt Wolfram Alpha, I’m sticking with my position as stated previously: I reserve proper assessment of it until I can plug+play it myself, objectively.

This is because all kinds of people have hyped WA or tried to make me believe that “Google doesn’t do semantic search” (their words) — despite me providing analysis which contradicts their convictions and competitive intelligence insights.

If I have a positive / negative perspective on Wolfram Alpha it will be based on my own independent and objective analysis (ok also humorous), and informed with previously accumulated, distilled and connected observations of the Semantic space rather than anyone else’s spin / misinformation / ignorance.

PERSONAL NOTE

I would never trust that SemWeb platform with my content again. At least on my own blog I know my information isn’t suddenly going to be deleted because of some irrational / small-minded / inconsistent / undemocratic whim of someone else.

Most importantly, I am not giving any licensing rights to the SemWeb platform over my content (original articles, images, comments, business models etc.) and their associated copyright for the SemWebco’s use or commercial exploitation. Frankly, their actions showed themselves to be unworthy of my trust and underlined how important it is to have ownership of your content and credit assignation for it.

Their Big Brother policies and breaches of user privacy were also not very appealing as a user-member.

All-in-all, I’m glad I don’t buy into that CEO (words and actions). He’s the same guy who insisted Google isn’t into semantic search, searching with Google is like “looking for a needle in a haystack” and who hypes up supposed “Google killers”. His radar’s way off.

Clearly, mine’s more perceptive, calibrated and spot on.