Teoma is another great example of clustering search engine. They developed a different marker for establishing the rank of a page. Instead of the PageRank developed by Brin and Page, they are using a Subject Specific Popularity:

Instead of ranking results based upon the sites with the most links leading to them, Teoma analyzes the Web as it is organically organized-in naturally-occurring communities that are about or related to the same subject-to determine which sites are most relevant. Teoma is the only search technology that can locate communities on the Web within their specific subject areas, as they actually exist. And this allows us to finely tune our search process, providing more precise results.

Tags: clustering, information retrieval, machine learning

1. LS-POS, from the university of Edinburgh [UK], GNU License, English language. [pdf]
2. TreeTagger, from the university of Stuttgart [DE], for English, German, Italian, Spanish, French and old French, implemented in C, Windows interface available, Python wrapper. [pdf]
3. Log-linear Tagger, from the university of Stanford [US], GNU License, Java API, English language.

Tags: machine learning, statistics, tagging, text data mining

Mindset is another example of clustering search engine. It allows you to sort search results for your query into commercial or non-commercial (informational) results, based on whether you’re shopping or seeking information.

What do they mean by commercial and non-commercial (informational)?

Commercial implies that the primary purpose of a given page is to sell you something. Informational implies that the primary purpose of the page is to provide information related to your search.

Tags: clustering, information retrieval, machine learning

During our meeting in Oulu, Jari Laru presented his work on Mobile Learning Scaffolding:

Scaffolding collaborative inquiry learning in nature context. First phase of that design experiment was conducted in late autumn 2004 in Pudasjärvi National Park, Finland. In first phase of that design experiment group of school children [n=22] participated in study where they used tailored software in autumn 2004. This tool enabled students create, receive and modify flyers with their mobile phones in inquiry-based learning project. (Laru, Stegmann, Järvelä, 2005; Laru, Järvelä, Clariana, 2005a, Laru, Järvelä, Clariana 2005b)

In the trial they used the Nokia Flier application, described below and freely downloadable from this url:

The Nokia Flier application allows you to create and locally distribute short messages containing text and a picture. When you have created your own flier you can publish it to other Nokia Flier users, who are close by (about 10 m) and have activated Nokia Flier application on the phone. Nokia Flier uses Bluetooth wireless technology for communicating with other phones. For power saving reasons, Nokia Flier will be automatically turned off after 12 hours of use.

Tags: location awareness, Location Based Services, mobile learning, new technology

Grokker is a clustering search engine with a powerful visual output. Grokker adds value to Yahoo searches by providing topically organized results that give you the freedom to explore the web the way you want. Grokker is ideal when you’re searching Yahoo! across a broad field of inquiry.

The engine is hooked on three other search engines: Yahoo, ACM digital library and Amazon.

Tags: clustering, information retrieval, information visualization

Timo Arnall had a great idea: using post its on a desk and a mobile phone to trigger specific events and actions. The idea behind the address book desk is very simple: to each Post-It on the table top, correspond a rfid tag that can be programmed with a specific trigger. Now the mobile comes to play, because when close to the tag, with a specific reader can match the will of the user to activate the specific trigger indicated on the Post-It. Very simple, yet very powerful.

[full story]

Copyright notice: the present content was taken from the following URL, the copyrights are reserved by the respective author/s.

Tags: collaboration tools, interaction design, interactive furniture, location awareness, RFID, tagging, tangible interface

• GalaTex XQuery Full-Text Search (XML query text search)
• Lemur Language Modelling IR Toolkit
• Lucene Apache Jakarta project
• SMART Early IR engine from Cornell University
• Terrier Information Retrieval Platform
• Xapian Open source IR platform based on Muscat
• Zettair
• DIG Open source web crawling software
• iHOP Information retrieval system for the biomedical domain
• MG full-text retrieval system Now maintained by the Greenstone Digital Library Software Project
• Information Storage and Retrieval Using Mumps

Tags: information retrieval, open source

M. W. Berry, S. T. Dumais, and T. A. Letsche. Computational methods for intelligent information access. In Supercomputing ’95: Proceedings of the 1995 ACM/IEEE conference on Supercomputing, page 20, San Diego, California, USA, 1995. ACM Press. [pdf]

—————–

This paper presents a detailed introduction to the Latent Semantic Indexing method, detailing the mathematical foundations and a visual example of the idea behind the singular value decomposition which is at the core of the method.

The main assumption of LSI is that there is some underlying or latent tructure in word usage that is partially obscured by variability in word choice. A truncated singular value decomposition is at the core of the method used to estimate the structure in word usage acreoss documents. Retrieval is then performed using the database of singular values and vectors obtained from the truncated SVD.

Mainly SVD reveals important information about the structure of a matrix, sparsifying minor differences in terminology that are at the base of synonymy and polisemy, a plague of information retrieval.

The paper also details the computational costs required to maintain up to date the matrices at the base of the method.

Tags: information retrieval, Latent Semantic Analysis, Singular Value Decomposition

I always think that simple ideas can change the world: this one seems to me one of those. It seems to me so simple that I could not believe it was not implemented before. In short the idea is to use a karaoke subtitling on popular movies and song and make this available on mass media content. In this way, illiterate people can start associating the pronunciation of words with the written form.

Same-Language Subtitling:

My organization, PlanetRead, works in Mumbai and Pondicherry, India. We have developed a “Same-Language Subtitling” (SLS) methodology, which provides automatic reading practice to individuals who are excluded from the traditional educational system, or whose literacy needs are otherwise not being met. This is an educational program rooted in mass media that demonstrates how a specific literacy intervention can yield outstanding, measurable results, while complementing other formal and non-formal learning initiatives of the government, private sector, and civil society. We are fortunate to have just been selected as a Google Foundation grantee.

More than 500 million people in India have access to TV and 40 percent of these viewers have low literacy skills and are poor. Through PlanetRead’s approach, over 200 million early-literates in India are getting weekly reading practice from Same Language Subtitling (SLS) using TV. The cost of SLS? Every U.S. dollar covers regular reading for 10,000 people – for a year.

I hit upon this idea in 1996 through a most ordinary personal experience. While taking a break from dissertation writing at Cornell University, I was watching a Spanish film with friends to improve my Spanish. The Spanish movie had English subtitles, and I remember commenting that I wished it came with Spanish subtitles, if only to help us grasp the Spanish dialogue better. I then thought, ‘And if they just put Hindi subtitles on Bollywood songs in Hindi, India would become literate.’ That idea became an obsession. It was so simple, intuitively obvious, and scalable in its potential to help hundreds of millions of people read — not just in India, but globally. So you can see how it works, we’ve uploaded some folk songs using SLS into Google Video.

Tags: education, google, hack, pedagogy, politics

R. Mihalcea and P. Tarau. Textrank: Bringing order into texts. In L. Dekang and W. Dekai, editors, Proceedings of EMNLP 2004, pages 404–411, Barcelona, Spain, July 2004. Association for Computational Linguistics. [pdf]

—————

This paper presents the TextRank algorithm for information retrieval and automatic keywords extraction from texts. Based on the PageRank introduced by Brin and Page (1998) the author shows how it is possible to add vertex weighting to the graph, with better results for the retrieval.

The ranking of the vertex is done with ‘vote casting’ following the principle that when a vertex link to another one, it is basically casting a vote for that other vertex. The higher the number of votes the higher the importance of the vertex. The scores associated with each vertex are determined based on the votes that are cast for it and the score of the vertices casting these votes.

The paper presents also a good literature review on keywords extraction, some of which based on baysian methods, supervised and not. The authors chose finally one method, namely Hull (2003), as comparison limit for their study.

They use the TextRank algorithm for keywords extraction, which is based on the co-occurrence relation, controlled by distance between words occurrence: two vertices are connected if their corresponding lexical units co-occur within a window of maximum N words, where N can be set anywhere from 2 to 10 words. Therefore, co-occurrence links express relations between syntactic elements, and similar to the semantic links found useful for the task of word sense disambiguation.

Additionally, as the authors observed, the vertices added to the graph can be pruned using part of speech filtering, to restrict the algorithm only to certain syntactinc cathegories, like nouns and verbs. The authors experimented with different syntactic filters, with best results observed for nouns and adjectives only. 

The authors registered the highest F-measure for their TextRank method. Additionally, results showed that the larger the window the lower the precision, probablye explained by the fact that a relation between words that are further apart in sot strong enough to define a connection in the text graph. Results are aligned with those of Hull (2003) that linguistic information helps the process of keywords extraction.

Tags: information retrieval, statistics, text data mining