Computer passing Turing Test? AI experts dismiss British claims

A four-rotor Enigma machine is shown, once used by the crews of German U-boats in World War II to send coded messages, which British World War II code-breaker mathematician Alan Turing was instrumental in breaking, and which is widely thought to have been a turning point in the war. (AP / Alex Dorgan Ross)

TORONTO -- Artificial intelligence experts are scoffing at claims that a computer has passed the famed Turing Test.

The University of Reading in England issued a press release claiming a "historic milestone" had been reached at the "Turing Test 2014" event it organized on the weekend, where a computer program designed to impersonate a teenager's language fooled a number of judges into thinking it was a real boy.

Alan Turing, a British computing pioneer, envisioned in 1950 that artificial intelligence would eventually evolve to the point that computers would fully understand human language and become capable of passing themselves off as people in on-screen conversations.

But the challenge that faced Eugene Goostman, the name of the computer program designed by a team of Russian and Ukrainian programmers, did not meet the standards of a true Turing Test, says University of Toronto Prof. Graeme Hirst.

Hirst says he was immediately skeptical of the university's proclamation given the current state of artificial intelligence.

"There's no point in doing a real Turing Test today, nothing even comes close to passing," says Hirst.

"It's so beyond the state of the art if you understand what the Turing Test really is."

The university's press release says 30 judges took part in the challenge and had online chats with both computer programs and real people. The school claims Eugene Goostman was able to fool the judges 33 per cent of the time.

Hirst is not alone in dismissing the so-called milestone. While some initial reports repeated the school's announcement unchallenged, technology experts worldwide quickly weighed in to discredit the claim. "That Computer Actually Got an F on the Turing Test," reads a headline on a BBC story.

Hirst says there's no denying that Eugene Goostman is an impressive "tour de force of a chatbot" but to claim it passed the Turing Test is wrong.

First, the baseline of passing should be 50 per cent, not the 30 per cent that the event organizers decided to go with, says Hirst.

"There's this strange idea that Turing said you only need to fool people 30 per cent of the time and since they crossed this magic 30-per-cent number they set off the fireworks -- but Turing never said that," he says.

"It's a misinterpretation."

That the computer program was designed to mimic a 13-year-old boy also dilutes the challenge of defeating the Turing Test, adds Hirst.

"You're allowed to ask all sorts of complicated questions pertaining to real intelligence and that was not on display at Reading, all we saw was clever chit chat. It was constrained to be talking to a 13-year-old about a hamster or something," Hirst says.

"Turing suggested: let's ask our computers about literature, let's ask about metaphors, let's ask about underlying feelings, let's ask about playing chess.

"I would ask in a Turing Test things like: 'I know you've read all of Jane Austen's works, tell me which did you like best and why? Which was the saddest scene in 'Pride and Prejudice' and why?"'

IBM's Watson, which defeated "Jeopardy" champions Brad Rutter and Ken Jennings in a battle of men versus machine, is the closest we've seen to a computer with enough artificial intelligence to take on a Turing Test, Hirst says.

"Watson, straight out, is the most advanced general natural language system that we have, it's really good at understanding language, it's really good even at understanding some aspects of humour and puns, and those things they do have in 'Jeopardy' questions," he says.

"But it wasn't so much intelligent as really, really, really good at fact retrieval."

Hirst refuses to predict when a computer might truly pass the Turing Test but he doesn't expect it will be any time soon.

"Watson was a breakthrough and I think nobody expected something that good to come along that early, so we could be surprised," he says.

"Usually we're far too optimistic. Just occasionally we surprise ourselves."