Wednesday, November 26, 2008

International approach to search

A new approach to search. Google the linguist.

from google blog

Our international approach to search

11/21/2008 11:46:00 AM
In previous posts in this series, you have read about the challenges of building a world-class search engine. Our goal is to make Google’s search be relevant to all people, regardless of their language or country. As my colleague Amit Singhal described, we use statistical data as the basis for making sweeping algorithmic changes. Many of these changes can be rolled out across all languages we support, but in some cases the unique characteristics of each language require some algorithmic considerations and tuning. And to make things really interesting, there are cases where the same language is different across countries. Obvious examples are "color" in the U.S. vs. "colour" in the U.K., or "camião" in Portugal vs. "caminhão" in Brazil.

My name is Daphne Dembo, and my focus is improving Google's international search. This is a tough challenge, since Google search is used in many countries and languages where our engineers have little personal knowledge. Initially, the international search improvements were done by Search Quality engineers who were passionate about their languages and countries: Lina from Sweden improved our parsing of compound words in German and Swedish; Dimitra from Greece introduced diacritical support; Ishai from Israel worked on transliteration corrections for Hebrew and Arabic; Trystan from Australia created methods for identifying local search results and ranking them together with foreign ones from the same language; Alex, a bilingual Ukrainian and Russian, introduced morphological understanding of these languages. As the importance of our international search grew, we solicited help from Googlers in all our offices. Finally, we are leveraging an international network of search specialists who help us understand search within the unique combination of their language and country.

Our first step in providing search support for a language is to train our language model on a large collection of documents in that language. This ensures that our language model is more precise and comprehensive — for example, it incorporates names, idioms, colloquial usage, and newly coined words not often found in static dictionaries. For instance, we recently started identifying Swahili, and used pages such as this one for the Parliament of Tanzania to train our system with the language's nuances. Having a trained language model helps to categorize documents during crawling and indexing of the web and to parse the user's query. Once this stage was complete, we launched Swahili search in countries such as Tanzania and Kenya, enabling local searches for the "Dar es Salaam stock exchange" [Soko la hisa dar es salaam], and "cure for Malaria" [Tiba ya malaria]. (As always, we are using square brackets to denote a search query. For example, you can search for "soccer" in Hamburg, Germany by clicking on [fußball in hamburg]).

We learn some things from our users, so as people start using our search engine, we can improve the way we rank in that language. Here are few examples:
  • Spell corrections: We recently launched spell corrections in Estonian. If your Estonian is rusty, and you don't remember how to spell "smoke detector," we can suggest a spell correction for [suitsuantur], leading to better search results.
  • Diacritical marks: Many languages have diacritical marks, which alter pronunciation. Our algorithms are built to support them, and even help users who mis-type or completely ignore them. For example, if you're a resident of Quebec, Canada and would like to know the weather forecast in Quebec City, we'll serve good results whether you type with diacritical signs [Météo à Québec] or without [meteo quebec]. Czech users can read the same excellent results for a popular kids' cartoon by searching for [krtecek] and [krteček]. On the other hand, sometimes diacriticals change the meaning of the word and we have to use them correctly. For example, in Thai, [ข้าว] is "rice," with completely different results than [ข่าว], which is "news"; or in Slovakia, results for "child" [dieťa] are different than results for "diet" [diéta].
  • Synonyms: A general case of diacritical support is the handling of synonyms in different languages. Korean searches showed that "samsung" can be viewed as a synonym of "삼성", so that when users search for [samsung], they find results which have the company's name in Korean.
  • Compounding: Some languages allow compounding, which is the formation of new words by combining together existing words. You can see a nice example in Swedish, where we return documents about a Swedish credit card for both compounded [Visakort] and non-compounded [visa kort] queries.
  • Stemming: Google has developed morphological models that can receive compound words as queries, and return pages which contain their stem, possibly as part of a different compound. For example, when searching for cars in Saudi Arabia, you can search for [سيارة] and [سيارات] because both are variants of the same stem, and both return many common results. A Polish user can search for "movie" [film], and get back results that contain other variants of the stem, such as "filmów," "filmu," "filmie," "filmy." A user from Belarus will find results for all word forms of the capital, Minsk [Мінск]: "Мінску," "Мінска," "Мінскага."
In addition to these semantic factors, Google does even more to parse documents and queries. Understanding the details of language usage in a country is important. Notation of acronyms is different across languages: In Hebrew it is double quotes before the last (left-most) character, as in "prime minister" [רה"מ]; in Thai — a dot at the end of the word, as in police station [สน. ]; while in the U.S. — dots after each character, as in [I.B.M.]. Chinese users quote works of art with a "《", as in: [《手机》剧情], and denote dates with a "日", as in: [2006年1月13日].

Beyond the linguistic elements of a language, we consider how people enter a query. For example, some languages that do not have Latin scripts require keyboards with dual alphanumeric keys. The user can switch between language input modes by typing special keystrokes. In case the user forgets to type this sequence, the queries end up being gibberish. You can see correct handling of these mistakes in Arabic ([hgsuv] corrected to [السعر]) and ([حقثسهيثىفهشم ثممثؤفهخىس ] corrected to [presidential elections]), Hebrew ([vdrk, kuyu] corrected to [הגרלת לוטו]), and Cyrillic ([rehc ljkffhf] corrected to [курс доллара]).

Another way of avoiding the inconvenience of switching keyboard modes is by typing the phonetic sounds of the query in Latin characters. Recreating the correct query in the target language isn't trivial, since there might be many possibilities. We can see several such examples in which we suggest the same query in the intended language for Russian ([biskvitnyi rulet] to [бисквитный рулет]), "movies" in Chinese ([dianying] to [电影]), and "Bank of Attica" in Greek [trapeza attikhs] returns good results for "Τράπεζα Αττικής". Users of 8 Indic languages (such as Hindi, Gujarati, Telugu) can type the phonetic sound of the query, and choose the words in Hindi script:

Ease of typing and reading is also influenced by the language used. Since every Chinese word requires several keystrokes on a standard keyboard, we provide category browsing by Images and related searches so that people don't need to type as much. Similarly, we are now launching Google Suggest, or real-time completion of queries, in many languages.

So far I described how we improve the quality of search in a language. However, there is a strong effect of the location of the user, even if it is only approximated to the country, since in many cases local content is more relevant than global information. For example, searching for Spanish Yellow Pages [Páginas Amarillas] will result in several documents of global interest and several local results in Peru, Mexico, and Spain. Similar to that, searching for [Côte d'Or] in France will return results for that region, whereas searches in Belgium will return results about the chocolate maker.

Note that the display of information should conform to the standards in that country, so we display "," as a decimal notation for Croatian users who want to know how many millimeters are in an inch [inč u milimetrima], or for Italian users who are interested in currency exchange rates [50 euro in dollari]. Similarly, temperatures in Norway [Været i Oslo] will be displayed in Celsius, while in the U.S. — in Fahrenheit [weather Boston].

If everything else fails, we provide cross-language translations based upon Google's translation technology described in this blog post. We will translate your query to English, search English documents on the web, and translate the returned results from English back into the original query language. For example, Japanese users who are interested in viewing Halloween illustrations (Halloween is a holiday which originated in Ireland) can search for [ハロウィン イラスト]. You can then request a Japanese translation of the English pages (at the bottom of the page), which will bring up the translation page in the screenshot below. Similarly, Korean users can search for the latest on Harry Potter [해리 포터], and Arabic readers can search for the opening of the Sydney Opera house [افتتاح دار الاوبرا في سيدني]. (Click on the image to see a larger version.)

All in all, Google Search is being actively developed for more than 100 languages, in 150+ countries, with dozens of improvements launched each month. So far I've covered the basics of how international search works, but this is just the surface of all the international work we do. There are many other interesting topics that impact international markets like usability, homepage and results page layout, and connectivity. An understanding of real cultural and human factors is essential to creating a search engine that resonates with the people who use it. (Click on the image to see a larger version.)

(Update: Replaced example in the 4th bullet point.)

Newer Post Older Post

We Were There: Volume 1

From The

Manga Reviews
We Were There: Volume 1
By Leroy Douresseaux
Nov 9, 2008 - 7:08:54 PM

Viz Media
Writer(s): Yuki Obata, Nancy Thistlethwaite, Tetsuichiro Miyaki
Penciller(s): Yuki Obata
Inker(s): Yuki Obata
Letterer(s): Inori Fukuda Trant
ISBN: 9781421520186
$8.99 US, $10.50 Canada, 192pp, B&W, paperback

We Were There 1 cover image courtesy of

Rated “T+” for “Older Teen”

At her new school, teen Nanami Takahashi falls for Motoharu Yano, a sly boy with carefree ways who is also the most popular boy in class (if not the entire school). In spite of her determination not to, Nanami is falling in love with the occasionally sarcastic Yano. When she discovers that Yano is grieving the death of his girlfriend who died the year before, Nanami suddenly finds herself competing with the memory of a first love.

THE LOWDOWN: A reader could judge We Were There, Vol. 1 as a bit of a downer. We Were There (originally titled, Bokura ga Ita) is a complex drama masquerading as a teen romance. Creator Yuki Obata accepts that teenagers in dramatic fiction can be as complex and maddeningly complicated as adult characters. In her narrative, life isn’t easy, and neither is falling in love. A manga that pits the attractive elements of romance with the hard edges of drama is refreshing. The fits and starts of love, however, sometimes cause We Were There to have a frequently awkward and uneven pace.

POSSIBLE AUDIENCE: Older readers of VIZ Media’s “Shojo Beat” line of manga may like the heavier romantic drama of We Were There.


© Copyright 2002-2008, Coolstreak Cartoons Inc. - All rights Reserved. All other texts, images, characters and trademarks are copyright their respective owners. Use of material in this document(including reproduction, modification, distribution, electronic transmission or republication) without prior written permission is strictly prohibited.

Rie Takada's Gaba Kawa

From The

Manga Reviews
Rie Takada's Gaba Kawa
By Leroy Douresseaux
Nov 18, 2008 - 7:38:53 PM

Viz Media
Writer(s): Rie Takada, Noritaka Minami, Lance Caselman
Penciller(s): Rie Takada
Inker(s): Rie Takada
Letterer(s): James Gaubatz
ISBN: 9781421520186
$8.99 US, $10.50 Canada, 192pp, B&W, paperback

The cover image of Gaba Kawa is courtesy of

Rated “T” for “Teen”

Rara Yamabuki is a recently arrived teenage demon, and she’s supposed to spend her time in the mortal world causing mischief and drawing humans to darkness. She’s not supposed to help humans. She’s definitely not supposed to fall in love with them, but she does. Something happens, because when Rara first sees Retsu Aku, the hottest boy at Meian Prefectural High School, she falls for him.

In Gaga Kawa, a young female demon learns that using any of her powers to help humans means that she’ll immediately lose that power. Still, Rara can’t stop loving the boy who calls her “Gaba Kawa,” even if it could mean the end of her.

THE LOWDOWN: Manga-ka mix supernatural and romance in their manga series as easily as Hollywood screenwriters, especially when it comes to high school romance. They make spooky/creepy and lovey-dovey seem like a natural fit. Gaba Kawa, a shoujo manga (comics for teen girls) from Rie Takada (Happy Hustle High), is similar to Chika Shiomi’s supernatural romance, Yurara, right down to the drawing style, although Gaba Kawa’s story never gets as dark or dangerous as Shiomi’s highly entertaining series. Gaba Kawa is standard fare; all the tropes of shoujo love stories are here, both in terms of plot/story and in terms of visual cues, but it works.

POSSIBLE AUDIENCE: Readers who can buy into a teen girl demon falling in love with a hot human boy will like Gaba Kawa.


© Copyright 2002-2008, Coolstreak Cartoons Inc. - All rights Reserved. All other texts, images, characters and trademarks are copyright their respective owners. Use of material in this document(including reproduction, modification, distribution, electronic transmission or republication) without prior written permission is strictly prohibited.


This is about searchwiki. Read and find out.

from google blog

SearchWiki: make search your own

11/20/2008 04:36:00 PM
Have you ever wanted to mark up Google search results? Maybe you're an avid hiker and the trail map site you always go to is in the 4th or 5th position and you want to move it to the top. Or perhaps it's not there at all and you'd like to add it. Or maybe you'd like to add some notes about what you found on that site and why you thought it was useful. Starting today you can do all this and tailor Google search results to best meet your needs.

Today we're launching SearchWiki, a way for you to customize search by re-ranking, deleting, adding, and commenting on search results. With just a single click you can move the results you like to the top or add a new site. You can also write notes attached to a particular site and remove results that you don't feel belong. These modifications will be shown to you every time you do the same search in the future. SearchWiki is available to signed-in Google users. We store your changes in your Google Account. If you are wondering if you are signed in, you can always check by noting if your username appears in the upper right-hand side of the page.

The changes you make only affect your own searches. But SearchWiki also is a great way to share your insights with other searchers. You can see how the community has collectively edited the search results by clicking on the "See all notes for this SearchWiki" link.

Watch our lead engineer, Amay, demonstrate a few ways to use SearchWiki in this short video:

This new feature is an example of how search is becoming increasingly dynamic, giving people tools that make search even more useful to them in their daily lives. We have been testing bits and pieces of SearchWiki for some time through live experiments, and we incorporated much of our learnings into this release. We are constantly striving to improve our users' search experience, and this is yet another step along the way.

Newer Post Older Post