Authors：Kazuhiro Kondo (Member) and Charles T. Hemphill (Non-member)

We developed the Japanese Speech-Aware Multimedia (JAM) which controls a World Wide Web (WWW) browser using speech. This system allows the user to browse a linked page by reading the anchor text within a Web page. The user can also control the browser using speech. The system integrates new vocabulary each time a new Web page is read by extracting the anchor text, converting this text to phonetic string notation, creating a new speech recognition grammar and integrating this grammar with the system dynamically. During this anchor text-to-phone conversion process, we employ numerous exception handling techniques to accommodate counters, dates and many other phenomena.

• Speakable commands: The user can control the browser using spoken commands, e.g. “scroll,” “go back” and “add this page to my bookmark.”
• Speakable bookmarks: The user can visit their favorite pages by enrolling a keyword or phrase for the page, and later speaking the keyword.
• Speakable links: The user can read the hyperlinked anchor text to visit the linked page.
• Smart pages: There may be situations where merely reading the anchor text may be too restrictive or inconvenient. Accordingly, we have implemented smart pages, in which a pointer to a speech grammar for the page is embedded in the header. The user can speak according to this grammar to navigate to linked pages. Flexible sentences with variations are allowed since arbitrary context-free grammars are allowed for the grammar.

On IBM-compatible PCs, IBM⁵ uses its own speech recognizer, VoiceType, along with a modified version of Netscape Navigator in a system called VoiceType Connection [5]。Similar to SAM, this system extracts anchor text from a Web page, constructs a speech grammar, and allows the user to visit a linked page by speaking the corresponding anchor text. However, users can only speak the words already in the vocabulary, or train the system for new words as they appear in a speaker-dependent fashion. It also allows speech input into forms with its 22,000 word dictation system. At the time of this writing, the system was in beta, and is downloadable from their Web site.

1. Speech Recognition of Web Pages

<A HREF=”http://www.afirm.co.jp/link1.html”>Link 1</A>

<A HREF=”http://www.afirm.co..jp/link2.html”>Link 2</A>

As is shown in the example, the linked URL is shown in double quotes with the tag A HREF. The anchor text is written between two tags, <A HREF=”..”> and </A>. Thus, the system merely needs to extract the text between these tags, and convert the text to speech grammars. Neither the language nor the character code of the text is restricted, thus Japanese text can also be used.

However, there are three major character encoding methods currently in use to encode Japanese: Extended Unix Code (EUC)，Japanese Industrial Standard (JIS) code and Shift JIS code [7]. In our preliminary tests, Japanese Web pages seem to be equally divided between these codes, with 30% EUC , 28% JIS code (both new and old JIS), and 42% JIS code. There were some pages with mixed encoding. This is probably because the author of the page created parts of the page by cutting and pasting from multiple pages with different encoding methods. There are also a variety of character sets included in Japanese pages, with Zenkaku characters (double ASCII character width Kanji, Hiragana, Katakana, Alphanumeric characters and symbols), Hankaku (half-width, equivalent to one ASCII character width) Katakana characters, ASCII characters, and JIS-Roman characters. It is necessary to convert these various standards and formats into one uniform notation for internal processing.

• Pauses need to be inserted explicitly in speech grammars at appropriate positions. Generally, it is known that pauses are optionally inserted at phrase boundaries. “Bunsetsus (phrases)” are composed of clauses [8]. Thus, if optional silences are inserted between phrase boundaries, pauses will be allowed at likely positions in the sentence, although somewhat redundantly.
• The system is configured to accept the input speech after reading a predefined number of phrases from the beginning (the default is currently set to 3) so that the user can omit some parts of a long anchor text. The speech grammar needs to have optional transitions to exit states after the predefined number of phrases for this to be possible. Thus, we need to define phrase boundaries in the text to determine positions to insert this transition.
• We need appropriate units to convert text to phonetic strings. Phrases are reasonable units.

• Exceptional handling of particles “ha” (actually read as “wa”) and “he” (read as “e”).
• Numerous exceptional handling of counters and digits. To give a few examples, both discrete and natural number reading needs to be allowed for numbers (e.g., for the number “123,” both “hyaku nijuu san” and “ichi ni san” is needed). Some counters change their pronunciation according to the preceding number (e.g., “ippon nihon sanbon” for one, two and three stick-like items). Some number-related readings are historical (e.g., “tsuitachi” for the first of the month).
• Some symbols are always omitted. Other symbols are sometimes read or omitted. Words in parentheses may be totally deleted. A common example of this is when names are spelled out in Hiragana. The speech grammar needs to allow all these exceptions and variations.
• The majority of Japanese language pages include some English anchor text. During the user evaluation tests, even though all the pages were Japanese pages, 17 % of the anchor texts were written all in English, and 24 % were mixed between English and Japanese. Thus, in order to accommodate these links which total 41 %, English text needs to be processed as well, although the vocabulary may be limited.

Non-text anchors also need to be processed. These are mostly images which are used for anchors instead of text. The images serve as clickable buttons to links, and are commonly called image maps. Figure 1 shows an example. In this example, the boxes with the strings “company info,” “products/services,” etc. serve as buttons. These do not explicitly use text as anchors and so cannot be used directly to create speakable links. However, HTML provides tags to define substitute text for the images typically for users of low-speed modems. These users commonly disable auto-loading of images in their browsers. To hint at the content of the images which these users can not see, the browser substitutes text defined by an HTML tag “ALT” in place of the images. The following is HTML code for the page shown in Figure 1 which uses this tag.

<A HREF=”/corp/docs/companyinfo.html”>

<IMGSRC=”/corp/graphics/cont1w.gif” ALT=”Company Info”></A>

<A HREF=”/corp/docs/prodserv.html”>

<IMGSRC=”/corp/graphics/cont2w.gif” ALT=”Products/Services”></A>

As shown in this example, substitution text can be defined with the tag ALT=”...” when images are used as anchors. This text can be used to create speech grammars for this link, thereby enabling speech to navigate to this link. In this example, the substitution text describes the content of the image appropriately, and thus the user may use speech to visit this link. However, not all substitution text is as comprehensive. Thus the speech input which the user thought might select the link may actually not work. Further study is needed to solve this problem.

1. System Architecture
1. Speakable links, speakable bookmarks and speakable commands

A prototype system of JAM was developed on a UNIX platform. We used Netscape Navigator as the browser. Figure 2 shows the configuration of the system.

The browser downloads a Web page written in HTML from the network. This HTML code is used to render the page in the browser, and also is output to JAM for analysis. JAM first detects the character encoding of the page, and converts the page to EUC if necessary. We decided to use EUC for all internal processing since Japanese characters in EUC are easily distinguishable from ASCII text. The converted code is analyzed, and anchor text URL pairs are extracted. If the anchor points to an image file, the system looks for ALT tags and, if there is one, the text inside this tag is used as anchor text for this link. The Hankaku Katakanas and ASCII alphanumerics and symbols are converted to the corresponding Zenkaku characters. All alphabets are converted to upper case. Then, the anchor texts are segmented into “bunsetsus” using a dictionary. We applied a freely available system, “Kakasi” [9] for the dictionary look-up. “Kakasi” is software that matches the input to the dictionary entries in 1-path. It is very simple but very fast, and is well suited for real-time applications. The original objective of “Kakasi” was to look-up readings of Kanji character strings. For this purpose, “Kakasi” comes with a dictionary with 121,824 entries. We added our own entries from a dictionary we had developed before [10] to bring the total to 357,203 entries. This dictionary was also developed for the same purpose as “Kakasi.” However, the dictionary has most of its entries in “bunsetsu” units. It also exhaustively lists most conjugations and variations. The system matches the longest entry in the combined dictionary as a “bunsetsu” candidate. In the worst case, when no appropriate “bunsetsu” is found in the dictionary, word entries or single character entries will always be found, thus falling back to a redundant shorter segmentation. It is possible to use a full-scale morphological analyzer such as the “JUMAN” [11] system to segment the input into more accurate “bunsetsus,” but we have concluded that this level of accuracy is not required for our purpose and is too expensive.

The segmented anchor text is then converted to a sentence-level speech recognition grammar. An example of this grammar is shown below.

start(jam_link_command_).

jam_link_command_ ---> “日本語 リンク の 例__”.

start(“日本語 リンク の 例__”).

“日本語 リンク の 例__” ---> 日本語_, Z_1.

“日本語 リンク の 例__” ---> non_speech_, “日本語 リンク の 例__”.

Z_1 ---> リンク_, Z_2.

Z_1 ---> non_speech, Z_1.

Z_2 ---> の_, Z_3.

Z_2 ---> non_speech, Z_2.

Z_3 ---> 例_, Z_4.

Z_3 ---> “”.

Z_3 ---> non_speech, Z_3.

Z_4 ---> “”.

Z_4 ---> non_speech, Z_4.

In this example, the anchor text is segmented into four “bunsetsus” and optional pause models “non_speech,” are inserted between each “bunsetsu.” After transition through three “bunsetsus,” the grammar allows transition to the exit node, shown by empty double quote pairs (“”). In this example, speech input is accepted by this grammar if “bunsetsus” up to at least the third “bunsetsu”, or “の” are spoken. Also, all symbols are made optional in the grammar.

Speakable bookmarks, speakable commands, and smart pages, which we will explain in detail later, are all written using similar grammars. For example, grammar for the speech command which moves one page forward can be written as:

start(jam_page_forward_command).

jam_page_forward_command ---> ( 先に | 次に ) [ 進む ].

Here, (.|.) refers to selection, and [.] refers to optional. The user can program arbitrary sentences or phrases to commands. However, commands are set once during start-up, and thus cannot be modified until restarted. An example speakable bookmark grammar is as follows:

start(“テキサス インスツルメンツ”).

url(“http://www.ti.com”).

“テキサス インスツルメンツ” ---> ( テキサス インスツルメンツ | TI ).

As shown in this example, the corresponding URL is also coded in a speakable bookmark grammar. New bookmarks can be added while browsing, but by default, the page title will become the key word for the page. This key word can be reassigned arbitrarily by the user. By reloading the speakable bookmarks, new bookmarks can be used immediately.

The “bunsetsu” nodes are expanded to a sub-level grammar which define the phonetic string representing their pronunciations. “Kakasi” is used in this process again. Using the example cited before,

start(日本語_).

日本語_ ---> NIHONXGO_, Z_1_.

日本語_ ---> NIPPONXGO_, Z_1_.

日本語_ ---> non_speech_, 日本語_.

Z_1_ ---> “”.

Z_1_ ---> non_speech_, Z_1_.

Start(NIHONXGO_).

NIHONXGO_ ---> n, i, h, o, N, (ng|g), o.

This example allows two alternate pronunciations, “nihongo” and “nippongo.” The second grammar above defines the phonetic string for one of the pronunciations, “nihongo.”

Exception handling involving digits and counters which we mentioned before are also accommodated. For digit strings, grammars which allow both discrete digits and natural numbers are generated. For example,

start(1969_).

1996_ ---> (SENX_ KYUUHAKU_ KYUUJYUU_ ROKU_ | ICHI_ KYUU_ KYUU_ ROKU_).

Start(1日_).

1日_ ---> (ICHINICHI_|TSUITACHI_).

English anchor text is expanded into Hiragana. English words not in the dictionary are expanded into their spelling pronunciation, and are further expanded to Hiragana. Symbols are also expanded when appropriate to Hiragana.

start(Apple_).

Apple_ ---> APPURU_.

Start(“A T & T_”).

“A T & T_” ---> EI_ TII_ (ANXDO_|ANXPAASANXDO_) TII_.

We ran a test to estimate the performance of the text-to-phone conversion accuracy. We extracted 117 anchor texts from some major Japanese newspapers on the Web, namely Asahi [12], Yomiuri [13], and Nikkei [14]. Texts extracted from these pages were converted to phonetic notation using the modified “Kakasi” system and the conversion accuracy was measured. The vocabulary covered numerous domains, including politics, economy, science and technology, and sports. Since “Kakasi” outputs multiple candidates, we chose the closest candidate. We measured the conversion accuracy according to the following formula:

Phonetic conversion accuracy ＝


Since the dictionary packaged with the original “Kakasi” distribution has wide coverage, perhaps with the exception of proper nouns including names, the conversion errors were mostly “bunsetsu” segmentation errors and exceptions including English terms, digits, and symbols. The phonetic conversion with the original dictionary was 92.4 %. However, with the expanded dictionary and with exception handling, the accuracy increases to 97.2 %. The improvement in the correct sentence percentage was more dramatic, with 57 % and 82 % for these two cases respectively. Since the anchor text in this application is relatively long, a few errors in the phone conversion may not have significant impact on the recognition accuracy. Thus, we believe this level of accuracy is adequate for our application.

Sentence-level grammars and the pronunciation grammars defining the phonetic strings for each “bunsetsu” are output to the speech recognizer. The recognition results are interpreted and converted to browser commands. When results corresponding to speakable commands are obtained, appropriate commands are issued to the browser. For example, when “Sakini Susumu (go forward)” is recognized, the command “page forward” is issued. When results corresponding to speakable bookmarks or speakable links are recognized, the “goto URL” command for the corresponding URL is issued.

Figure 3 shows the console of the system. The console has a simple level meter to the left, and gives speech level feedback to the user. There is also a start button which additionally functions as a status indicator, showing the input status of the system; accepting input, and processing. The change in the input status is also fed back with beeps.

2. Smart Pages

With speakable commands and speakable links, users can surf most of the Web using speech. However, there are components on a Web page which are difficult to control with speech, e.g., forms and image maps. Accordingly, we propose a mechanism which we will call smart pages.

In smart pages, speech recognition grammars are used to define allowed speech input in addition to conventional links. A pointer to the speech recognition grammar is embedded in the header portion of the HTML code as follows:

<HEAD>

<LINK REL=”X-GRAMMAR” HREF=”smart_page.grm”>

</HEAD>

The <LINK> tag conforms to the HTML standard, and is used to designate the relationship of the linked resource to this page. The REL tag defines the relationship to the current document. In the above example, it designates the linked resource as a grammar file. The format of the grammar is similar to the grammar which defines speakable commands. The following example shows a smart page grammar which extracts weather information for major Japanese cities:

start(weather)

weather ---> City＿ の （現在の （天気｜気温｜湿度） ｜ 予報）

[（は｜を見せて [下さい]）].

City＿ ---> (東京｜大阪｜京都｜名古屋｜福岡｜神戸｜横浜｜札幌｜.....).

Speech recognition results using this grammar are passed on to a CGI (Common Gateway Interface) script as arguments to be used as keys for the weather database query. For example, for the recognition “forecast for Yokohama” (”横浜 の 予報”), the following URL is accessed:

http://www.tenki.co.jp/cgi-bin/tenki?横浜+の+予報

The script returns a page with the forecast for Yokohama.

As shown in this example, much more flexible speech input is possible using smart pages. Image maps can also be navigated using speech to some extent if an appropriate smart page grammar is used. The additions for smart pages do not affect conventional WWW browsers in any way.

3. Speech Recognition

We use a continuous multi-variate single Gaussian mixture HMM speech recognizer. Context-dependent phonetic models are used. The phonetic context defined using phonological features was clustered using a binary decision tree [5]. Speech segments are identified from input speech using an energy-based speech detector.

The grammar used in the system is defined as a set of regular grammars. Each start symbol of a regular grammar is connected to a terminal symbol in a higher-level regular grammar. The system allows a subset of the start symbols to start the search, and thus allows dynamic adjustment of the grammar. We can also add or delete regular grammars easily, which enables us to add or delete sentences required for each page as required.

2. Evaluation Experiments

We conducted a simple evaluation test to measure the performance of the system, and to identify problems. We conducted two types of tests. In the first, in order to measure the performance of speakable links and speakable commands, we asked the subjects to freely surf the WWW. In the second test, we tried to compare the efficiency of smart pages to regular pages. In both tests, there were 12 subjects, of which 7 were female and 5 male. Six of the subjects reported that they use the computer regularly as well as surf the Web. Two use computers a few times a week, while the remaining four never or rarely use them.

A Sun Sparc20 workstation was used in all tests. An unidirectional dynamic microphone was used. The users were asked to operate the on-off switch on the microphone for each utterance, but we also applied simple speech detection. The internal A/D conversion of the Sparc 20 was employed. All tests were conducted in a closed office. The room was relatively silent, but the air conditioning and the fan on the Sparc 20 were quite audible.

1. Speakable Links And Speakable Command Evaluations

• Sukurooru: scroll down within the same page
• Ueni sukurooru：scroll up within the same page
• Sakini susumu：Move forward one page in the history if there is one.
• Maeni modoru：Move backward one page in the history if there is one.
• Peeji sai roodo：Reload current page from the Web.
• Hoomu peeji：Move back to the home starting page used in the evaluation.

• Newspapers and publishers: Asahi News, Yomiuri News and Nikkei Electronics.
• Search engine: Yahoo Japan.
• Others: The Japanese Prime Minister’s page, The Sumo Association.

The most frequent cause of errors was speech detection errors, followed by speech recognition errors, insufficient speech input, and out of vocabulary. Insufficient speech input refers to errors caused by utterances in which the user did not speak the required three “bunsetsus.” Out of vocabulary includes misread words, utterances that were read from the middle of the anchor (which is not allowed) and attempts to read from image maps without ALT tags. The “Others” category includes pages which were not possible to parse due to HTML coding errors. There were significantly more speech detection errors with male speakers, which we believe is caused by subjects not holding the microphone in a stable manner or at appropriate distance. A close-talking head-mounted microphone may have helped here. There were also male subjects with extremely long inter-word pauses, which triggered the speech detector to end speech input. In order to deal with these, we need to improve the detection accuracy through optimization or adaption of the detection level and the introduction of other features. Insufficient entry and out of vocabulary utterances also constitute a large portion of the errors. However, we believe these errors will decrease to negligible level as the user becomes familiar with the system. The task completion without these two error categories comes to 94.1 %.

1. Comparison of Smart Pages and Regular Pages

Here, Higashi, Higashikata, Nishi, and Nishikata are the sides on which the Rikishis compete (east and west). Yokozuna and Oozeki are their ranks, and the rest are their names.

1. Conclusion

• The majority of Japanese pages include some English. Thus, practical systems need to be able to handle both English and Japanese. JAM can handle limited English words as well as Japanese.
• Unlike English, Japanese does include explicit word boundaries in the text. Thus, segmentation into appropriate units is necessary. This is required because when generating speech grammars , (1) we need to insert optional silences between these units (2) optional transition to exit node needs to be inserted in order to allow omission after three words from the top are read.

• Ambiguous anchor text: Some pages use the same anchor text for several different links. A common example is a page with “here” as the anchor text for a number of links. These links can be navigated using mouse clicks, but navigation using speech requires disambiguation. Grids can be superposed on pages, and speech can be used to specify the position of the selected link. The same mechanism can be used for image maps. Alternatively, we can disambiguate the links by altering the page.
• Non-text anchors: Most of these anchors are image maps. Some images also have “ALT” tags to define substitution text for users with the automatic image loading option turned off. The substitution text can be used to define speech input to select this link. However, not all images carry “ALT” tags. These images need to be dealt using other techniques, such as using grids as we have described in the previous paragraph. Also, some text defined with “ALT” tags may not describe the image appropriately. This text needs to be displayed, for instance, superimposed on or near the image, or in a different window.
• Handling of components other than links: Mostly buttons and form inputs. Buttons can be handled in similar ways as image maps, for instance using grids. Most form inputs should be possible to handle with smart pages.
• Pages with extremely large number of links: In the pages we encountered during our evaluation tests, the maximum number of links we found in a page was 395. Pages with this many links require noticeable amount of time to generate the speech grammars, and the recognition accuracy also suffers. The average number of links in a page was 65.9, which suggests that the majority of the pages are within reasonable range. Pages with too many links can be accommodated by limiting the construction of the speech recognition grammars to the links currently within visual range, in other words, links currently shown in the window. However, even when navigating with mouse, these long pages are often ill-organized pages which are difficult to navigate. We expect these pages to eventually decrease as the authors improve their skills.
• New HTML components: HTML is still a rapidly evolving standard. Developers of the most popular browsers are constantly introducing new proprietary components, which may eventually become standards. It is difficult to accommodate all these new components. However, not all are widely used, and some eventually become obsolete. We need to use common sense to select components which seem to have been accepted by the community and accommodate these. For instance, frames seem to have been generally accepted, and we need to be able to handle these in our system.
• HTML coding errors: A considerable number of pages still include coding errors. Some common examples are missing quotes, missing tags, misspelled tags, mixed Japanese character encoding, and missing control codes. Most of these errors seem to be introduced when the authors are using plain text editors to hand-code their pages. WYSIWIG editors and HTML-savvy editors are becoming available, which should reduce these errors. Still, we need to be able to gracefully handle these errors to some extent through robust parsing.

1. C. Hemphill, P. Thrift and J. Linn, “Speech-Aware Multimedia,” IEEE Multimedia, vol. 3, no. 1, pp. 74-78, Spring 1996.
1. Y. Yamazaki,“Men for Survey, Women for Communication: Usage Analysis of PC Communication”Nikkei Electronics, no. 665, pp. 129-138, July 1996 (in Japanese).
1. http://snow.cit.cornell.edu/noon/ListenUp.html.
1. http://www.surftalk.com.
1. http://www.software.ibm.com/is/voicetype/vtconn/vtconn.html.
1. H. Dohi, M. Ishizuka, ”A Visual Software Agent Connected to WWW/Mosaic,” Trans. IEICE, vol. J79-D-�U, no. 4, April 1996 (in Japanese).
1. Ken Lunde, “Understanding Japanese Information Processing,” pp. 59-99, O’Reilly & Associates, Sebastopol, California, 1993.
1. K. Hakoda, H. Sato, “A Pause Insertion Rule for Connected Speech,” Technical Report of the speech research group of the ASJ, S74-64, pp. 1-7, March 1975 (in Japanese).
1. H. Takahashi, ”Kakaksi: Kanji Kana Simple Inverter,” version 2.2.5, June, 1994. Available from ftp.uwtc.washinton.edu.
1. J. Picone, T. Staples, K.Kondo and N. Arai, “Kanji to Hiragana Conversion Based on a Length-Constrained N-Gram Analysis,” to be published in the IEEE Transactions on Speech and Audio Processing.
1. Y. Matsumoto, S. Kurohashi, T. Utsuro, Y. Myoogi, M. Naga, “”Japanese Morphological Analysis System JUMAN Manual,” version 2.0, July, 1994. Available from ftp://ftp.aist-nara.ac.jp/pub/nlp/tools/juman.
1. http://www.asahi.com.
1. http://www.mainichi.co.jp.
1. http://www.nikkei.co.jp.
1. Y. H. Kao, C. T. Hemphill, B. J. Wheatley and P. K. Rajasekaran, “Toward Vocabulary Independent Telephone Speech Recognition,” Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. I-117 - I-120, April 1994.
1. http://www.wnn.or.jp/wnn-t/database/rikishidata.