1.8.9.1 ECHOWORDS mode to encourage fluent reading

[JackMostow]

This issue was originally part of Revise READ UI/UX tracking #157:

Problem: get READ’s “blame the kid” strict left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.

Solution:

1. Rename RoboTutor's current ECHO as REREAD because it rereads the sentence after the kid reads it.
2. Add an ECHO mode that works more like its namesake, Project LISTEN's Reading Tutor's ECHO mode.
3. Relax the strict left-to-right policy to tolerate occasional rejected or skipped words.

Currently RoboTutor underlines the word it expects next. It turns words green if accepted and red if not.
Requiring acceptance of each word before advancing to the next word discourages fluent reading due to false rejections.

Change this behavior as follows:

1. Modify position tracking to tolerate an individual missed word but not 2 in a row.
   a. MultiMatch aligns ASR output to text to minimize total penalties for mismatches and jumps. It may or may not be the right mechanism to tolerate isolated missed words, e.g. by aligning only to i, i+1, or i+2.
   b. The policy means that the expected next word can skip from word i to i+2 but not further.
   c. Underline the expected next word to indicate the current position i.
   d. The current position should not move backwards from word i to j < i because it would be confusing.
   e. The current position can skip from i to i+2 if the ASR output ends with ... word i-1 word i+1.

2. Color words green when accepted, red when rejected, and (still) black when skipped.
   a. Or should skipped words turn red?
   b. It's ok to visibly accept and credit a previously rejected word, but not to rescind credit once granted.

3. When the reader pauses (0.5 sec) before reading word i, echo word(s) j..i-1, where word j-1 was the last echoed word, and turn each word green after echoing it. When RoboTutor reads just one word, use its isolated narration for clarity. Otherwise use the appropriate subsequence of its human narration. Note that:
   a. Reading, word, by, word will echo a word 0.5 sec after it's spoken, but wait patiently for the next word.
   b. Reading a phrase at a time will echo the phrase 0.5 sec after it's spoken.

4. Treat a tap almost like reading the current word:
   a. A tap after a hesitation will just read (and credit) the current word i.
   b. A tap with no hesitation before word i will reread the word(s) j..i. I don't expect this case to occur often.

Notes:

1. Project LISTEN's Reading Tutor changed the background color of each word to yellow while reading it, to show its position. This would be a nice feature if there's time, to make clear which word is being read.

2. Project LISTEN's Reading Tutor deferred crediting words green rather than turn them green right away. We can implement that behavior if the behavior above causes kids to wait for each word to turn green before reading the next word.

3. Replacing underlining with RoboFinger would be cute but is not worth the effort, especially if it requires representing the position of each text word on the screen.

[JackMostow]

Actually we can dispense with REREAD and just use ECHO, provided we modify ECHO to reread the entire sentence if the kid read it disfluently, in which case rereading it scaffolds comprehension.

[JackMostow]

@octavpo - can you implement ECHO by August 28 when I return?

[JackMostow]

@octavpo - Let me rephrase that request more emphatically:

XPRIZE weights reading 60%, ECHO tries to addess severe problems in RoboTutor's current ASR-driven assisted oral reading, and is important to kid-test ASAP.

So please implement ECHO in time for me to try out on August 29. If you have any questions before then, please email them to me.

I'll be offline Friday and Saturday PST, but will reply ASAP. I'll be travelling all day Tuesday, August 28.

Thanks. - Jack

[JackMostow]

Octav email of Saturday, September 01, 2018 6:29 PM:

... I'll try to put more details on the Github issue, but in short I found that in my tests most cases when a word is marked red are due to the fact that works are recognized and lost rather than because they're not recognized correctly. So I'm trying to fix that first.

Jack email of Sat 9/1/2018 9:47 PM:

“Words are recognized and lost” is very interesting. Based just on your own testing, or can you tell from the verbose logs from use in the field? How?

Please send a real example. From your brief description, I suspect a bug in the MultiMatch process that aligns ASR output against the text to read – not in MultiMatch itself, but perhaps something like omitting utterances from the speech aligned against the text if newer utterances have come in.

[JackMostow]

@octavpo - #357 said:
READ often fails to respond to oral reading miscues. Why?
a. The speech is not loud enough to pass the detection threshold, perhaps due to a noisy environment.
b. The ASR recognizes something but RoboTutor doesn't respond.

Note that a word turns red only if RoboTutor detected speech, i.e. not in case a.

[JackMostow]

@judithodili suggested that lowering the speech detection threshold would make ASR hallucinate more.
@judithodili and @amyogan - What if RoboTutor detects no speech for [3] seconds?

Options:

a. Don't supply the word automatically, but cue the kid to tap, e.g. have RoboFinger tap the audio icon
+: keeps RoboTutor from advancing without the kid doing anything
+: clearer than just displaying the audio icon
+: simple

b. Treat the word as misread, i.e. turn it red.
+: if kid attempted the word, responds visibly if not immediately
-: if kid was silent, might cause confusion or frustration due to too much time pressure

c. If RoboTutor still detects no speech, invoke the 2-strike rule, i.e. supply the word.
+: alleviate frustration caused by RoboTutor failing to detect kid's attempts to read the word
-: lets RoboTutor keep reading even if the kid says nothing

I favor option a. Do you concur?

[amyogan]

Yes, I like keeping the kids actively involved!

[JackMostow]

@octavpo - Please implement option a.