Conference and Labs of the Evaluation Forum

Last updated
Conference and Labs of the Evaluation Forum
CLEF Logo.png
Unlocking Information Access
AbbreviationCLEF
Discipline information retrieval
Publication details
Publisher Springer and CEUR Workshop Proceedings
History2000;23 years ago (2000); formerly known as Cross-Language Evaluation Forum from 2000 to 2009
Frequencyannual
Website www.clef-initiative.eu

The Conference and Labs of the Evaluation Forum (formerly Cross-Language Evaluation Forum ), or CLEF, is an organization promoting research in multilingual information access (currently focusing on European languages). Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards. [1] The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the US, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives. [2]

Contents

CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book [3] on the lessons learned in 20 years of evaluation activities.

Structure of CLEF

Before 2010, [4] CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs.

Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas., [5] [6]

In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr [7] and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images. [8] In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.

List of CLEF workshops and conferences

YearLocationNoteProceedingsEditors
2000 Lisbon Constituting workshopCross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000)Carol Ann Peters
2001 Darmstadt Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross—Language Evaluation Forum (CLEF 2001) Revised PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2002 Rome Advances in Cross-Language Information Retrieval: Third Workshop of the Cross—Language Evaluation Forum (CLEF 2002) Revised PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2003 Trondheim Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross—Language Evaluation Forum (CLEF 2003) Revised Selected PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2004 Bath Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross—Language Evaluation Forum (CLEF 2004) Revised Selected PapersCarol Ann Peters and Paul Clough and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini
2005 Vienna Accessing Multilingual Information Repositories: Sixth Workshop of the Cross—Language Evaluation Forum (CLEF 2005). Revised Selected PapersCarol Ann Peters and Fred C Gey and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini and Henning Müller and Maarten de Rijke
2006 Alicante Evaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross—Language Evaluation Forum (CLEF 2006). Revised Selected PapersCarol Ann Peters and Clough, P. and Fred C Gey and Jussi Karlgren and Bernardo Magnini and Doug Oard and Maarten de Rijke and M Stempfhuber
2007 Budapest Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross—Language Evaluation Forum (CLEF 2007). Revised Selected PapersCarol Ann Peters and Jijkoun, V. and Mandl, T. and Henning Müller and Oard, D. W. and Peñas, A. and Petras, V. and Santos, D.
2008 Århus Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross—Language Evaluation Forum (CLEF 2008). Revised Selected PapersCarol Ann Peters and Deselaers, T. and Nicola Ferro and Julio Gonzalo and Gareth J F Jones and Kurimo, M. and Mandl, T. and Peñas, A.
2009 Corfu Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments—Tenth Workshop of the Cross—Language Evaluation Forum (CLEF 2009). Revised Selected PapersCarol Ann Peters and Di Nunzio, G. M. and Kurimo, M. and Mandl, T. and Mostefa, D. and Peñas, A. and Roda, G.
2010 Padua First conferenceMultilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010 Maristella Agosti and Nicola Ferro and Carol Ann Peters and Maarten de Rijke and Alan Smeaton
2011 Amsterdam Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011)Forner, P. and Julio Gonzalo and Kekäläinen, J. and Lalmas, M. and Maarten de Rijke
2012 Rome Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012)Catarci, T. and Forner, P. and Hiemstra, D. and Peñas, A. and Santucci, G.
2013 Valencia Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013)Forner, P. and Henning Müller and Paredes, R. and Rosso, P. and Stein, B.
2014 Sheffield Information Access Evaluation—Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014)Kanoulas, E. and Lupu, M. and Clough, P. and Sanderson, M. and Hall, M. and Hanbury, A. and Toms, E.
2015 Toulouse Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015)Mothe, J. and Savoy, J. and Jaap Kamps and Pinel-Sauvagnat, K. and Gareth J F Jones and SanJuan, E. and Cappellato, L. and Nicola Ferro
2016 Evora Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016)Fuhr, N. and Quaresma, P. and Goncalves, T. and Larsen, B. and Balog, K. and Macdonald, C. and Cappellato, L. and Nicola Ferro
2017 Dublin Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017)Gareth J F Jones and Lawless, S. and Julio Gonzalo and Kelly, L. and Lorraine Goeuriot and Thomas Mandl and Cappellato, L. and Nicola Ferro
2018 Avignone Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018)Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian-Yun Nie, Laure Soulier, Eric SanJuan, Linda Cappellato, and Nicola Ferro
2019 Lugano Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019)Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David Losada, Gundula Heinatz, Linda Cappellato, and Nicola Ferro
2020 Thessaloniki (virtual due to COVID-19)Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020)Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, A., Linda Cappellato, and Nicola Ferro
2021 Bucharest (virtual due to COVID-19)Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021)K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro

Related Research Articles

Information retrieval (IR) in computing and information science is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. Searches can be based on full-text or other content-based indexing. Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for the metadata that describes data, and for databases of texts, images or sounds.

Cross-language information retrieval (CLIR) is a subfield of information retrieval dealing with retrieving information written in a language different from the language of the user's query. The term "cross-language information retrieval" has many synonyms, of which the following are perhaps the most frequent: cross-lingual information retrieval, translingual information retrieval, multilingual information retrieval. The term "multilingual information retrieval" refers more generally both to technology for retrieval of multilingual collections and to technology which has been moved to handle material in one language to another. The term Multilingual Information Retrieval (MLIR) involves the study of systems that accept queries for information in various languages and return objects of various languages, translated into the user's language. Cross-language information retrieval refers more specifically to the use case where users formulate their information need in one language and the system retrieves relevant documents in another. To do so, most CLIR systems use various translation techniques. CLIR techniques can be classified into different categories based on different translation resources:

An image retrieval system is a computer system used for browsing, searching and retrieving images from a large database of digital images. Most traditional and common methods of image retrieval utilize some method of adding metadata such as captioning, keywords, title or descriptions to the images so that retrieval can be performed over the annotation words. Manual image annotation is time-consuming, laborious and expensive; to address this, there has been a large amount of research done on automatic image annotation. Additionally, the increase in social web applications and the semantic web have inspired the development of several web-based image annotation tools.

Question answering (QA) is a computer science discipline within the fields of information retrieval and natural language processing (NLP) that is concerned with building systems that automatically answer questions that are posed by humans in a natural language.

In information science and information retrieval, relevance denotes how well a retrieved document or set of documents meets the information need of the user. Relevance may include concerns such as timeliness, authority or novelty of the result.

<span class="mw-page-title-main">Text Retrieval Conference</span> Meetings for information retrieval research

The Text REtrieval Conference (TREC) is an ongoing series of workshops focusing on a list of different information retrieval (IR) research areas, or tracks. It is co-sponsored by the National Institute of Standards and Technology (NIST) and the Intelligence Advanced Research Projects Activity, and began in 1992 as part of the TIPSTER Text program. Its purpose is to support and encourage research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies and to increase the speed of lab-to-product transfer of technology.

Relevance feedback is a feature of some information retrieval systems. The idea behind relevance feedback is to take the results that are initially returned from a given query, to gather user feedback, and to use information about whether or not those results are relevant to perform a new query. We can usefully distinguish between three types of feedback: explicit feedback, implicit feedback, and blind or "pseudo" feedback.

Sentiment analysis is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. With the rise of deep language models, such as RoBERTa, also more difficult data domains can be analyzed, e.g., news texts where authors typically express their opinion/sentiment less explicitly.

The TREC Genomics track was a workshop held under the auspices of NIST for the purpose of evaluating systems for information retrieval and related technologies in the genomics domain. The TREC Genomics track took place annually from 2003 to 2007, with some modifications to the task set every year; tasks included information retrieval, document classification, GeneRIF prediction, and question answering.

Query expansion (QE) is the process of reformulating a given query to improve retrieval performance in information retrieval operations, particularly in the context of query understanding. In the context of search engines, query expansion involves evaluating a user's input and expanding the search query to match additional documents. Query expansion involves techniques such as:

Plagiarism detection or content similarity detection is the process of locating instances of plagiarism or copyright infringement within a work or document. The widespread use of computers and the advent of the Internet have made it easier to plagiarize the work of others.

Geographic information retrieval (GIR) or geographical information retrieval systems are search tools for searching the Web, enterprise documents, and mobile local search that combine traditional text-based queries with location querying, such as a map or placenames. Like traditional information retrieval systems, GIR systems index text and information from structured and unstructured documents, and also augment those indices with geographic information. The development and engineering of GIR systems aims to build systems that can reliably answer queries that include a geographic dimension, such as "What wars were fought in Greece?" or "restaurants in Beirut". Semantic similarity and word-sense disambiguation are important components of GIR. To identify place names, GIR systems often rely on natural language processing or other metadata to associate text documents with locations. Such georeferencing, geotagging, and geoparsing tools often need databases of location names, known as gazetteers.

Human–computer information retrieval (HCIR) is the study and engineering of information retrieval techniques that bring human intelligence into the search process. It combines the fields of human-computer interaction (HCI) and information retrieval (IR) and creates systems that improve search by taking into account the human context, or through a multi-step search process that provides the opportunity for human feedback.

A concept search is an automated information retrieval method that is used to search electronically stored unstructured text for information that is conceptually similar to the information provided in a search query. In other words, the ideas expressed in the information retrieved in response to a concept search query are relevant to the ideas contained in the text of the query.

<span class="mw-page-title-main">LGTE</span>

Lucene Geographic and Temporal (LGTE) is an information retrieval tool developed at Technical University of Lisbon which can be used as a search engine or as evaluation system for information retrieval techniques for research purposes. The first implementation powered by LGTE was the search engine of DIGMAP, a project co-funded by the community programme eContentplus between 2006 and 2008, which was aimed to provide services available on the web over old digitized maps from a group of partners over Europe including several National Libraries.

SemEval is an ongoing series of evaluations of computational semantic analysis systems; it evolved from the Senseval word sense evaluation series. The evaluations are intended to explore the nature of meaning in language. While meaning is intuitive to humans, transferring those intuitions to computational analysis has proved elusive.

<span class="mw-page-title-main">Stemming</span> Process of reducing words to word stems

In linguistic morphology and information retrieval, stemming is the process of reducing inflected words to their word stem, base or root form—generally a written word form. The stem need not be identical to the morphological root of the word; it is usually sufficient that related words map to the same stem, even if this stem is not in itself a valid root. Algorithms for stemming have been studied in computer science since the 1960s. Many search engines treat words with the same stem as synonyms as a kind of query expansion, a process called conflation.

The Center for the Evaluation of Language and Communication Technologies (CELCT) was an organisation devoted to the evaluation of language technologies, located in Povo, Trento (Italy).

Evaluation measures for an information retrieval (IR) system assess how well an index, search engine or database returns results from a collection of resources that satisfy a user's query. They are therefore fundamental to the success of information systems and digital platforms. The success of an IR system may be judged by a range of criteria including relevance, speed, user satisfaction, usability, efficiency and reliability. However, the most important factor in determining a system's effectiveness for users is the overall relevance of results retrieved in response to a query. Evaluation measures may be categorised in various ways including offline or online, user-based or system-based and include methods such as observed user behaviour, test collections, precision and recall, and scores from prepared benchmark test sets.

<span class="mw-page-title-main">Diffeo, Inc.</span> American knowledge discovery software company

Diffeo, Inc., is a software company that developed a collaborative intelligence text mining product for defense, intelligence and financial services customers.

References

  1. ↑ Peters, Carol; Braschler, Martin; Choukri, Khalid; Gonzalo, Julio; Kluck, Michael. The Future of Evaluation for Cross-Language Information Retrieval Systems. Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001. CiteSeerX   10.1.1.109.7647 .
  2. ↑ Peters, Carol, and Martin Braschler. "European research letter: Cross‐language system evaluation: The CLEF campaigns." Journal of the Association for Information Science and Technology 52.12 (2001): 1067-1072.
  3. ↑ Ferro, Nicola; Peters, Carol (2019). Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF. volume 41 of The Information Retrieval Series. Springer International Publishing, Germany. doi:10.1007/978-3-030-22948-1.
  4. ↑ CLEF history before 2010
  5. ↑ "Special Issue on CLEF". Information Retrieval. 7 (1–2). 2004. doi:10.1023/B:INRT.0000009472.59294.f6.
  6. ↑ Fredric C. Gey, Noriko Kando, and Carol Peters "Cross-Language Information Retrieval: the way ahead" in Information Processing & Management vol. 41, no. 3, p.415-431 May 2005, doi : 10.1016/j.ipm.2004.06.006
  7. ↑ Karlgren, Jussi; Gonzalo, Julio; Clough, Paul (2006). iclef 2006 overview: Searching the flickr www photo-sharing repository. Seventh Workshop of the Cross-Language Evaluation Forum CLEF 2006..
  8. ↑ Mueller, Henning (20 May 2010). "Medical Retrieval Task". ImageCLEF - Cross-language image retrieval evaluations. Retrieved 27 May 2010.


This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.