Task oriented tools for information retrieval
Date
2017
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Information Retrieval (IR) is one of the most evolving research fields and has drawn extensive attention in recent years. Because of its empirical nature, the advance of the IR field is closely related to the development of various toolkits. While the traditional IR toolkit mainly provides a platform to evaluate the effectiveness of retrieval models, there are many emerging challenges that needs to be addressed using non-existing toolkits, e.g. teaching and research tools that scales at real world application, unified reproducibility evaluation system and also new applications such as contextual suggestion. In this thesis, we build various task-orientated IR toolkits in order to better address the new challenges. ☐ First, the education oriented Virtual IR Lab (VIRLab) and Lucene-based Anserini is introduced in order to provide easy access to IR toolkits for both students and researchers. These toolkits can greatly reduce the instructor's work for teaching the IR courses especially the teaching of ranking models. VIRLab provides a web-based interface which enables the students to implement ranking models with just a few lines of API calls. It also includes many facilities such as automatic evaluation, search engine creation and the leader board for ranking models. Anserini is a command line interface (CLI) based toolkit built on top of Lucene. The advantage of Anserini lies in its capability of dealing with web-scale datasets and the utilities (e.g. multi-threaded indexing and streamlined TREC evaluation) that are essential to IR researchers. ☐ Next, we propose a privacy preserving evaluation (PPE) framework in order to provide a general solution for the reproducibility study. In the framework, users would have different permission levels of accessing to the data -- from choosing the ranking models from a list to leverage APIs to directly manipulate the index. We build a second level PPE system for a commercial dataset where users can leverage APIs to implement their ranking models and the performance is automatically returned. ☐ We then introduce another instantiation of PPE framework -- the Reproducible IR system evaluation (RISE) -- in order to provide a unified evaluation system for the reproducibility study of IR ranking models. By using RISE, it is trivial to implement the ranking models and compare its performance with all existing models. We believe this could greatly reduce the redundant work of IR researchers on the unnecessary re-implementation of other ranking models. More importantly, the unified result RISE generates is the key to validate the utility of the proposed models. ☐ Furthermore, we provide tools for analyzing the performance of existing ranking models for keyword queries. The best performing ranking functions such as BM25 and Dirichlet language model have been proposed for many years. Although there are some recently proposed ranking functions, their performances cannot easily surpass the old ones. Thus, it is interesting to investigate the reason behind this and also explore the performance upper bound if there is any. In this thesis, we first apply the gain/cost analysis in order to estimate the practical performance upper bound of single-term queries. We then identify the best subqueries for multiple-terms keyword queries by introducing several post-retrieval term relationship features. We argue that because the original queries do not follow the intuitions of the newly proposed the features, they do not achieve the better performances. ☐ Last, we design an integrated contextual suggestion toolkit for contextual suggestion. The novelty of contextual suggestion mainly lies in its "zero query" property, meaning user does not need to submit query in order to get desired recommendations. Our toolkit consists of two components: a mobile application that can automatically detect the user's "context" (e.g. location and datetime); and the other component is essentially a recommendation system that can proactively suggests interesting venues based on user's current context and user's preference history. For the recommendation part, we investigate category-based and opinion-based user profile modeling approaches. Both methods work well on TREC and Yelp collections. Detailed analysis shows the advantage of opinion-based user profile modeling as it potentially answers "why does the user like a place".
Description
Keywords
Communication and the arts, Applied sciences, Information retrieval, Task oriented tools