Ken Jennings

Message Boards

IBM's DeepQA and the Jeopardy! Challenge

The place to talk. "On topic"? "Off topic"? We make no such petty distinctions here.

Postby braggtastic » Tue Feb 15, 2011 9:33 am

That didn't stop Alex from chastising Watson when it repeated Ken's wrong response.
braggtastic
 
Posts: 796
Joined: Fri Jul 28, 2006 9:49 am

Postby Lilly » Tue Feb 15, 2011 9:35 am

My favorite moment from last night's show was when Ken buzzed in and (finally!) beat Watson to the punch, then looked surprised and said, "Oh, I don't know!! Um... the 1980's" (Of course he was right.)
Lilly
 
Posts: 358
Joined: Mon Jul 03, 2006 9:04 am
Location: Chattanooga, TN

Postby jzerocsk » Tue Feb 15, 2011 10:16 am

grodney wrote:the buzzer timing still sticks with me a little bit. I figure all 3 know the answer to a huge % of the questions. Watson *knows* when the buzzer is enabled and can react instantaneously (if he has the answer and wants to), while the other 2 have to rely on their experience and timing.


I think this "advantage" is getting a little bit more airplay than it deserves...
I alluded to this earlier in the thread, but a little more explicitly, the human player also *knows* when the buzzer is enabled, because he sees the light come on which is (hopefully) the same moment Watson gets the signal. The only difference is that Watson can react more quickly. But Watson can ONLY react to the signal. A human player gets not just the light, but also the benefit of hearing the question outloud and being able to anticipate when the buzzer will be enabled. In fact it seems a lot of the stronger Jeopardy players focus way more on anticipating when the light will come on than actually waiting for it. Waston can't do that. It can't hear the question and it can't do a damn thing about the buzzer until "the light comes on" in the form of it getting the input signal. Obviously it can often react to the signal even faster than a good player who can predict when the light will come on, but I do not think there is really any unfair advantage to this setup. Everybody "sees" the same light...they just perceive it differently.
jzerocsk
 
Posts: 766
Joined: Wed Jul 26, 2006 8:04 am

Postby Helen Turner » Tue Feb 15, 2011 10:23 am

TheConfessor wrote:
Helen Turner wrote:While IBM initially tried to keep it anonymous, NY Times readers and several bloggers “outed” Watson’s voice as that of actor/audiobook narrator Jeff Woodman.

Woodman recently confirmed this fact on a syndicated radio show. The voice is from recordings he made for IBM in 2004 for a text-to-speech program.

I don't know if that is true or not. Can you cite a credible source? This IBM site says that Andy Aaron directed the speaker to create Watson's character. It seems unlikely that he did that in 2004, since there was no plan to create the Watson character at that time.
http://www.research.ibm.com/deepqa/speech_team.shtml


It has been confirmed. Woodman was interviewed yesterday(2/14/11) by syndicated radio show host Lise Avery on her program "Anything Goes." ( The interview is archived at anythinggoesradio.com.) Woodman (and several other actors) participated in the recording of IBM's text to speech recordings, which were made in 2004, at Full House Studios in NYC, and which were supervised by Andy Aaron.

IBM began to develop Watson in 2006, and when the deal was made to have him play Jeopardy, Watson was originally going to print out his answers in on-screen text. Aaron suggested that Watson speak his answers, and offered the show's producers their choice of the 5 or 6 actors' voices they had in the text to speech data base. The producers chose Woodman's voice.
Helen Turner
 
Posts: 4
Joined: Fri Feb 04, 2011 2:12 pm

Watson's Voice

Postby Helen Turner » Tue Feb 15, 2011 10:29 am

Helen Turner
 
Posts: 4
Joined: Fri Feb 04, 2011 2:12 pm

Postby marpocky » Tue Feb 15, 2011 10:31 am

ChadHahn wrote:I'm glad that I'm DVRing the shows since after yesterday all the Watson background might get boring and I'll want to fast forward to the play.


Speak for yourself. That's the most interesting part to me.
marpocky
 
Posts: 1591
Joined: Mon Apr 14, 2008 6:39 pm
Location: Suzhou, China

Postby SpiffWilkie » Tue Feb 15, 2011 10:54 am

marpocky wrote:
ChadHahn wrote:I'm glad that I'm DVRing the shows since after yesterday all the Watson background might get boring and I'll want to fast forward to the play.


Speak for yourself. That's the most interesting part to me.


+1
A lot of the "Watson cheats!" talk could be avoided by people paying attention to the background info.
SpiffWilkie
 
Posts: 132
Joined: Wed Sep 05, 2007 9:14 pm

Postby AmericanErrorist » Tue Feb 15, 2011 12:25 pm

Ken was mentioned in McSweeney's again today. Watson makes its debut in the piece.

Has the writer of the piece read up on Watson?

Loved the skin cell bit, though.
AmericanErrorist
 
Posts: 29
Joined: Wed Nov 15, 2006 4:26 pm
Location: Tucson, AZ

Postby PamelaJaye » Tue Feb 15, 2011 2:21 pm

skullturfq wrote:Is there an easy way to legally watch episode 2 on the internet tomorrow after it airs, or should I try to remember how to program a VCR for probably the first time this millennium?


secondary problem: two of my VCRs have died, and my roommate's has as well. One of the three ate a tape in the process. SO, if you really do want to use a VCR - check to make sure it still works. I mostly stopped using mine around July of 2004. Either the tapes or the machine could be... not good.
Also, my DVD recorder intermittently chooses not to record on a disc.
The DVR blew up on 12/9 and it became an issue.
DVR is somewhat better now but is having some playback issues. (It doesn't like to playback while recording - which really puts a cramp in Prime Time. and it doesn't like to play back from drive 4, but it *sure* loves to record to that drive, no matter how much space is available elsewhere)
Pam from Boston
http://www.bakulanews.com
Scott Bakula -- Latest Appearances and News
PamelaJaye
 
Posts: 196
Joined: Fri Sep 08, 2006 5:41 pm

Postby Robert K S » Tue Feb 15, 2011 3:24 pm

Helen Turner wrote:It has been confirmed. Woodman was interviewed yesterday(2/14/11) by syndicated radio show host Lise Avery on her program "Anything Goes." ( The interview is archived at anythinggoesradio.com.) Woodman (and several other actors) participated in the recording of IBM's text to speech recordings, which were made in 2004, at Full House Studios in NYC, and which were supervised by Andy Aaron.


I don't think the interview is posted yet, but Lise assures me it will be eventually, and for prompt notification of that eventuality we should subscribe to her newsletter.
Robert K S
 
Posts: 162
Joined: Tue Jun 20, 2006 11:51 am
Location: Cleveland, Ohio

Re: some answers for the curious

Postby bwouns » Tue Feb 15, 2011 3:52 pm

christo wrote: We struggled with the decision whether to try to do something about "rebounds", but chose to spend our resources on other aspects of the Jeopardy problem. Hopefully, this one wont bite us in the end. We'll see...



I must be missing something, because it seems to me that this should be extremely easy to fix, perhaps with just a few lines of code.
bwouns
 
Posts: 1537
Joined: Wed Aug 16, 2006 4:31 am
Location: Eugene, OR

Postby Whatsahoe » Tue Feb 15, 2011 4:05 pm

If Ken or Brad win do they get real money? If the computer wins, what does it get?
Whatsahoe
 
Posts: 529
Joined: Sun Nov 02, 2008 10:58 am

Postby AmericanErrorist » Tue Feb 15, 2011 4:11 pm

Whatsahoe wrote:If Ken or Brad win do they get real money? If the computer wins, what does it get?


IBM gets the computer's money, which it will donate to charity.
AmericanErrorist
 
Posts: 29
Joined: Wed Nov 15, 2006 4:26 pm
Location: Tucson, AZ

Postby rkd » Tue Feb 15, 2011 5:06 pm

Great show so far. It's an interesting battle between Ken/Brad's massive linguistic advantage and Watson's massive buzzer-timing advantage. Seems like a fair matchup thus far, but it'll be interesting to see whether the linguistic advantage outweighs the timing advantage more heavily as the questions get harder. It'll also be interesting to analyze the stats afterward to see which questions go to which side. I expected that questions featuring specific names would be easier for Watson, but we saw at least one question on show 1 for which those names led Watson to the wrong answer. Anyway, great show, and thanks to Chris and Ed and others for the extra info.

I've got a few more questions which someone here may be able to answer. Apologies if these were answered elsewhere.

[1] Those percentages seem to represent 'confidence' in a given answer, and the one of highest value is chosen. But do those percentages reflect an entire analysis of the question, or do they reflect the moment that someone buzzes in? In other words, if a full analysis of the question would take 13 seconds but someone (Watson or other) buzzes in after 2 seconds, are we seeing the 2-second values or the 13-second values? Chris's comments about the buzzer advantage suggest a significant temporal issue for Watson; thus my question.

[2] What percentage, or percentage distribution, triggers Watson to buzz? Is it something simple like "buzz if it's >50% on one answer." Or is it more like "buzz if it's >50% on one answer iff that that answer is at least >10% greater than the second choice"?

[3] Will game strategy factor into Watson's decision as to whether or not to buzz? For instance, if a contestant is up $2800 with one $2000 question to go, the contestant should only buzz on that question if extremely confident in an answer, rather than risk losing the lead. Will Watson make similar strategic decisions? That's rather peripheral to the linguistic issues, but I'm curious as to whether that factors in.

[4] What dictates Watson's choice of category and question value?
rkd
 
Posts: 710
Joined: Fri Jun 16, 2006 5:16 pm
Location: North Hollywood, CA

Postby marpocky » Tue Feb 15, 2011 5:55 pm

The level of outrage from the Jeopardy purists over on the J! boards is borderline comical.
marpocky
 
Posts: 1591
Joined: Mon Apr 14, 2008 6:39 pm
Location: Suzhou, China

Postby econgator » Tue Feb 15, 2011 7:12 pm

marpocky wrote:The level of outrage from the Jeopardy purists over on the J! boards is borderline comical.


Disappointment much more than outrage.
econgator
 
Posts: 3632
Joined: Mon Sep 18, 2006 6:11 pm

Postby enielsen » Tue Feb 15, 2011 9:07 pm

econgator wrote:
marpocky wrote:The level of outrage from the Jeopardy purists over on the J! boards is borderline comical.


Disappointment much more than outrage.


I'm less outraged and more convinced that we will all be plugged into the matrix and used as fuel for our robotic overlords sooner rather than later.

Buzzing speed aside, I'm thoroughly impressed with Watson's linguistic analysis capabilities. He had a high degree of confidence in the correct answer on what seemed like almost every question asked. There were some missteps, but not nearly as many as I would have expected. I would love to see the code behind him.
enielsen
 
Posts: 361
Joined: Thu Jul 27, 2006 8:13 pm
Location: Austin, TX

Postby gameshowcongress » Tue Feb 15, 2011 10:54 pm

It appears that the logic for Watson took care not to have a Cliff Claven wagering response on FJ in the first game.

(interesting no chart shown on the logic for the answer - it could have made an interesting sidebar to show iterations during the playing of "Think!", a highly appropriate title given the venue for the match)
gameshowcongress
 
Posts: 698
Joined: Fri Jun 30, 2006 1:17 pm
Location: Boulder, Colorado USA

Postby TheConfessor » Wed Feb 16, 2011 1:58 am

Helen Turner wrote:
TheConfessor wrote:
Helen Turner wrote:While IBM initially tried to keep it anonymous, NY Times readers and several bloggers “outed” Watson’s voice as that of actor/audiobook narrator Jeff Woodman.

Woodman recently confirmed this fact on a syndicated radio show. The voice is from recordings he made for IBM in 2004 for a text-to-speech program.

I don't know if that is true or not. Can you cite a credible source? This IBM site says that Andy Aaron directed the speaker to create Watson's character. It seems unlikely that he did that in 2004, since there was no plan to create the Watson character at that time.
http://www.research.ibm.com/deepqa/speech_team.shtml


It has been confirmed. Woodman was interviewed yesterday(2/14/11) by syndicated radio show host Lise Avery on her program "Anything Goes." ( The interview is archived at anythinggoesradio.com.) Woodman (and several other actors) participated in the recording of IBM's text to speech recordings, which were made in 2004, at Full House Studios in NYC, and which were supervised by Andy Aaron.

IBM began to develop Watson in 2006, and when the deal was made to have him play Jeopardy, Watson was originally going to print out his answers in on-screen text. Aaron suggested that Watson speak his answers, and offered the show's producers their choice of the 5 or 6 actors' voices they had in the text to speech data base. The producers chose Woodman's voice.

Helen, you seem to have a lot of inside information about this. Do you know why IBM has refrained from giving credit to Jeff Woodman (or to anyone else)? Have you seen this video? Andy Aaron says he auditioned dozens of voice actors for the part of Watson, and the "lucky winner" spent weeks in a studio, recording thousands of phrases. He certainly implies that the work was done more recently than 2004, especially if he auditioned dozens of people and the Jeopardy producers chose the one they liked best. But maybe it's intentionally ambiguous.
http://www.youtube.com/watch?v=WIKM732oEek
TheConfessor
 
Posts: 1483
Joined: Fri Jun 16, 2006 3:11 pm
Location: Austin, TX

Postby TheConfessor » Wed Feb 16, 2011 2:03 am

gameshowcongress wrote:It appears that the logic for Watson took care not to have a Cliff Claven wagering response on FJ in the first game.

(interesting no chart shown on the logic for the answer - it could have made an interesting sidebar to show iterations during the playing of "Think!", a highly appropriate title given the venue for the match)

My recollection is that I saw a screen display at the taping that showed 12% confidence in Toronto and 11% confidence in Chicago, but I haven't been able to verify that, so my confidence is only 73% that I am right. I do know that Chicago was a very close second place.
TheConfessor
 
Posts: 1483
Joined: Fri Jun 16, 2006 3:11 pm
Location: Austin, TX

Postby gameshowcongress » Wed Feb 16, 2011 5:08 am

TheConfessor wrote:
gameshowcongress wrote:It appears that the logic for Watson took care not to have a Cliff Claven wagering response on FJ in the first game.

(interesting no chart shown on the logic for the answer - it could have made an interesting sidebar to show iterations during the playing of "Think!", a highly appropriate title given the venue for the match)

My recollection is that I saw a screen display at the taping that showed 12% confidence in Toronto and 11% confidence in Chicago, but I haven't been able to verify that, so my confidence is only 73% that I am right. I do know that Chicago was a very close second place.


It might be too proprietary, but it would be cool to have and expansion of Watson's thought process for the entire 30 seconds, perhaps with more expansion of some of the threads of consideration beyond the top 3 usually shown, or perhaps with magnification on some of the top choices with decision trees, etc.

This would be a lot like the original "Password" show, where instead of having an announcer read in a hushed voice when the answer was revealed prior to play, the Watson thought process would be displayed without audio, so that those who were watching at home would just look away during the music until they had formed their own opinon if they did not want to be influenced by the machine.

Of course that would have been additional, perhaps unnecessary programming, for the show, but I would think something like that already exists for the debug and development process, and would be a handy look under the hood if Watson takes his show to remote terminals at computer system exhibitions in the future.
gameshowcongress
 
Posts: 698
Joined: Fri Jun 30, 2006 1:17 pm
Location: Boulder, Colorado USA

Postby puck71 » Wed Feb 16, 2011 7:32 am

TheConfessor wrote:
gameshowcongress wrote:It appears that the logic for Watson took care not to have a Cliff Claven wagering response on FJ in the first game.

(interesting no chart shown on the logic for the answer - it could have made an interesting sidebar to show iterations during the playing of "Think!", a highly appropriate title given the venue for the match)

My recollection is that I saw a screen display at the taping that showed 12% confidence in Toronto and 11% confidence in Chicago, but I haven't been able to verify that, so my confidence is only 73% that I am right. I do know that Chicago was a very close second place.
Obviously Watson didn't understand the category. I was wondering if it got mixed up because it was written "US CITIES" rather than "U.S. CITIES." If it had understood the category properly, it could have defined a set of possible answers, of which Toronto would not have been a member.

Speaking of understanding the category, I wonder how Watson decided how much to wager?
puck71
 
Posts: 23
Joined: Thu Aug 23, 2007 10:01 am

Postby SpiffWilkie » Wed Feb 16, 2011 9:03 am

puck71 wrote: Obviously Watson didn't understand the category. I was wondering if it got mixed up because it was written "US CITIES" rather than "U.S. CITIES." If it had understood the category properly, it could have defined a set of possible answers, of which Toronto would not have been a member.

Speaking of understanding the category, I wonder how Watson decided how much to wager?


I really don't think that the difference in US and U.S. changed how the computer went about the answer. A brief explanation from IBM: http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html
How could the machine have been so wrong? David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine.
SpiffWilkie
 
Posts: 132
Joined: Wed Sep 05, 2007 9:14 pm

Postby rick1013 » Wed Feb 16, 2011 10:20 am

I, for one, welcome our artificially intelligent computer overlords, but I will make my own airline reservations, thanks all the same.
I chose him out of thousands. I didn't like the others, they were all too flat.
rick1013
 
Posts: 71
Joined: Fri Feb 09, 2007 10:33 am
Location: Carrollton, TX

Postby Helen Turner » Wed Feb 16, 2011 11:38 am

TheConfessor wrote:
Helen Turner wrote:
TheConfessor wrote:
Helen Turner wrote:While IBM initially tried to keep it anonymous, NY Times readers and several bloggers “outed” Watson’s voice as that of actor/audiobook narrator Jeff Woodman.

Woodman recently confirmed this fact on a syndicated radio show. The voice is from recordings he made for IBM in 2004 for a text-to-speech program.

I don't know if that is true or not. Can you cite a credible source? This IBM site says that Andy Aaron directed the speaker to create Watson's character. It seems unlikely that he did that in 2004, since there was no plan to create the Watson character at that time.
http://www.research.ibm.com/deepqa/speech_team.shtml


It has been confirmed. Woodman was interviewed yesterday(2/14/11) by syndicated radio show host Lise Avery on her program "Anything Goes." ( The interview is archived at anythinggoesradio.com.) Woodman (and several other actors) participated in the recording of IBM's text to speech recordings, which were made in 2004, at Full House Studios in NYC, and which were supervised by Andy Aaron.

IBM began to develop Watson in 2006, and when the deal was made to have him play Jeopardy, Watson was originally going to print out his answers in on-screen text. Aaron suggested that Watson speak his answers, and offered the show's producers their choice of the 5 or 6 actors' voices they had in the text to speech data base. The producers chose Woodman's voice.

Helen, you seem to have a lot of inside information about this. Do you know why IBM has refrained from giving credit to Jeff Woodman (or to anyone else)? Have you seen this video? Andy Aaron says he auditioned dozens of voice actors for the part of Watson, and the "lucky winner" spent weeks in a studio, recording thousands of phrases. He certainly implies that the work was done more recently than 2004, especially if he auditioned dozens of people and the Jeopardy producers chose the one they liked best. But maybe it's intentionally ambiguous.
http://www.youtube.com/watch?v=WIKM732oEek


What info I have stems from the fact that I'm a freelance audio technician and was working in the studio where the recordings were made in 2004. I brought this thread, and the video of Aaron's interview, to Jeff's attention, and he emailed me the following;

Hi Helen, long time! My goodness, I had no idea I'd be at the center of an online controversy, but thanks for pointing it out to me!

To answer your questions, IBM has refrained from giving me credit because they are under no contractual obligation to do so, and are understandably more invested in promoting Watson and their TTS (text-to-speech) program than they are in promoting my participation. (But they sent me a very nice check, so it's all good with me!)

Initially I was contractually obligated to keep silent about being the voice of Watson. However, when I was contacted by a reporter who had recognized my voice in an online Watson promo (having just listened to an audiobook of mine), I contacted the attorney for IBM and asked how I should proceed. I was told that IBM wished to "keep it anonymous," but by then I was receiving Google alerts on my name telling me that several bloggers had already written posts identifying Watson's voice as mine.

(Hell, even my 96 year old great-aunt left me a message saying, "Hello Sweetheart, I was just watching TV and I recognized your voice as that computer that plays 'Wheel Of Fortune!'" prompting visions of Watson in an evening gown turning letters...)

Recognizing that the information was already out there, IBM consented and sent me "talking points" for interviews. (One of their more amusing stipulations was that I reveal NONE of my knowledge of how Watson actually works, which was a no-brainer because I don't HAVE any knowledge of how Watson actually works -- I even watched the NOVA episode and I'm still clueless...)

The video is a tiny bit disingenuous, in as much as (in my understanding) no one was ever really auditioned "to be" the voice of Watson, though that's sorta the implication.

What is accurate in the video is that they did indeed audition dozens of actors for the TTS program in 2004, ultimately casting myself, Kathleen MacInerney, and Karla Hendrick, among several others, and that we did indeed each spend endless hours recording non-sequiter sentences (pulled mostly from old CNN copy) as well as sequential series of numbers, dates, etc... These recordings then formed the basis of IBM's TTS application, with which licensees could create phone menus, public address announcements, etc... (Do you have any idea how weird it is to call your bank and have your SELF ask you for your account number?)

Then about a year ago I got a call from Andy saying that IBM had developed a computer that was going to challenge some past Jeopardy champions, that my voice had been selected as the voice of the computer, and that they wanted to put me on retainer for several hours of additional recording.

The IBM technicians ultimately decided that it was less time consuming to tweak the recordings they had (there's an online video of Watson's learning curve with things like Roman numerals) than to introduce new recordings into the data base, the upshot being that I got paid for the use of my voice without having to move off the sofa -- my kinda job!

Hope this clears things up for you!

All best,

Jeff
Helen Turner
 
Posts: 4
Joined: Fri Feb 04, 2011 2:12 pm

PreviousNext

Return to Main Forum

Who is online

Users browsing this forum: Google [Bot], Yahoo [Bot] and 2 guests

cron