Apache Lucene

Last updated
Lucene
Developer(s) Apache Software Foundation
Initial release1999;25 years ago (1999)
Stable release
9.10.0 / February 20, 2024;0 days ago (2024-02-20) [1]
Repository
Written in Java
Operating system Cross-platform
Type Search and index
License Apache License 2.0
Website lucene.apache.org

Apache Lucene is a free and open-source search engine software library, originally written in Java by Doug Cutting. It is supported by the Apache Software Foundation and is released under the Apache Software License. Lucene is widely used as a standard foundation for production search applications. [2] [3] [4]

Contents

Lucene has been ported to other programming languages including Object Pascal, Perl, C#, C++, Python, Ruby and PHP. [5]

History

Doug Cutting originally wrote Lucene in 1999. [6] Lucene was his fifth search engine. He had previously written two while at Xerox PARC, one at Apple, and a fourth at Excite. [7] It was initially available for download from its home at the SourceForge web site. It joined the Apache Software Foundation's Jakarta family of open-source Java products in September 2001 and became its own top-level Apache project in February 2005. The name Lucene is Doug Cutting's wife's middle name and her maternal grandmother's first name. [8]

Lucene formerly included a number of sub-projects, such as Lucene.NET, Mahout, Tika and Nutch. These three are now independent top-level projects.

In March 2010, the Apache Solr search server joined as a Lucene sub-project, merging the developer communities.

Version 4.0 was released on October 12, 2012. [9]

In March 2021, Lucene changed its logo, and Apache Solr became a top level Apache project again, independent from Lucene.

Features and common use

While suitable for any application that requires full text indexing and searching capability, Lucene is recognized for its utility in the implementation of Internet search engines and local, single-site searching. [10] [11]

Lucene includes a feature to perform a fuzzy search based on edit distance. [12]

Lucene has also been used to implement recommendation systems. [13] For example, Lucene's 'MoreLikeThis' Class can generate recommendations for similar documents. In a comparison of the term vector-based similarity approach of 'MoreLikeThis' with citation-based document similarity measures, such as co-citation and co-citation proximity analysis, Lucene's approach excelled at recommending documents with very similar structural characteristics and more narrow relatedness. [14] In contrast, citation-based document similarity measures tended to be more suitable for recommending more broadly related documents, [14] meaning citation-based approaches may be more suitable for generating serendipitous recommendations, as long as documents to be recommended contain in-text citations.

Lucene-based projects

Lucene itself is just an indexing and search library and does not contain crawling and HTML parsing functionality. However, several projects extend Lucene's capability:

See also

Related Research Articles

<span class="mw-page-title-main">Apache Nutch</span> Open source web crawler

Apache Nutch is a highly extensible and scalable open source web crawler software project.

In text retrieval, full-text search refers to techniques for searching a single computer-stored document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases.

<span class="mw-page-title-main">Alfresco Software</span> Information management software

Alfresco Software is a collection of information management software products for Microsoft Windows and Unix-like operating systems developed by Alfresco Software Inc. using Java technology. The software, branded as a Digital Business Platform is principally a proprietary & a commercially licensed open source platform, supports open standards, and provides enterprise scale. There are also open source Community Editions available licensed under LGPLv3.

<span class="mw-page-title-main">Doug Cutting</span> American information theorist

Douglass Read Cutting is a software designer, advocate, and creator of open-source search technology. He founded two technology projects, Lucene, and Nutch, with Mike Cafarella. Both projects are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop.

<span class="mw-page-title-main">Apache Solr</span> Open-source enterprise-search platform

Solr is an open-source enterprise-search platform, written in Java. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is widely used for enterprise search and analytics use cases and has an active development community and regular releases.

A document-oriented database, or document store, is a computer program and data storage system designed for storing, retrieving and managing document-oriented information, also known as semi-structured data.

Coveo is a software-as-a-service (SaaS) search engine, powered by artificial intelligence based in Quebec, Canada. The Coveo Relevance Cloud offers ecommerce, customer service, digital workforce, and website search.

Vector space model or term vector model is an algebraic model for representing text documents as vectors such that the distance between vectors represents the relevance between the documents. It is used in information filtering, information retrieval, indexing and relevancy rankings. Its first use was in the SMART Information Retrieval System.

Pentaho is business intelligence (BI) software that provides data integration, OLAP services, reporting, information dashboards, data mining and extract, transform, load (ETL) capabilities. Its headquarters are in Orlando, Florida. Pentaho was acquired by Hitachi Data Systems in 2015 and in 2017 became part of Hitachi Vantara.

<span class="mw-page-title-main">Elasticsearch</span> Search engine

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is dual-licensed under the source-available Server Side Public License and the Elastic license, while other parts fall under the proprietary (source-available) Elastic License. Official clients are available in Java, .NET (C#), PHP, Python, Ruby and many other languages. According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine.

<span class="mw-page-title-main">Sematext</span> Software services company

Sematext is a globally distributed organization that builds cloud and on-premises systems for application-performance monitoring, alerting, anomaly detecting, centralized logging, logging management, analytics, and real user monitoring. The company also provides search and Big Data consulting services and offers production support and training for Solr and Elasticsearch to clients. The company markets its core products to engineers and DevOps and its services to organizations using Elasticsearch, Solr, Lucene, Hadoop, HBase, Docker, Spark, Kafka and other platforms. Otis Gospodnetić founded Sematext. Sematext is headquartered in Brooklyn, NY, and is privately held.

Microsoft Azure Cognitive Search, formerly known as Azure Search, is a component of the Microsoft Azure Cloud Platform providing indexing and querying capabilities for data uploaded to Microsoft servers. The Search as a service framework is intended to provide developers with complex search capabilities for mobile and web development while hiding infrastructure requirements and search algorithm complexities. Azure Search is a recent addition to Microsoft's Infrastructure as a Service (IaaS) approach.

StormCrawler is an open-source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.

<span class="mw-page-title-main">Ontotext GraphDB</span> RDF-store

Ontotext GraphDB is a graph database and knowledge discovery tool compliant with RDF and SPARQL and available as a high-availability cluster. Ontotext GraphDB is used in various European research projects.

OpenSearch is a family of software consisting of a search engine, and OpenSearch Dashboards, a data visualization dashboard for that search engine. The software started in 2021 as a fork of Elasticsearch and Kibana, with development led by Amazon Web Services.

Norconex Web Crawler is a free and open-source web crawling and web scraping Software written in Java and released under an Apache License. It can export data to many repositories such as Apache Solr, Elasticsearch, Microsoft Azure Cognitive Search, Amazon CloudSearch and more.

References

  1. ↑ "Welcome to Apache Lucene". Lucene™ News section. Archived from the original on 12 February 2021. Retrieved 12 February 2020.
  2. ↑ Kamphuis, Chris; de Vries, Arjen P.; Boytsov, Leonid; Lin, Jimmy (2020), "Which BM25 do You Mean? A Large-Scale Reproducibility Study of Scoring Variants", in Jose, Joemon M.; Yilmaz, Emine; Magalhães, João; Castells, Pablo (eds.), Advances in Information Retrieval, Lecture Notes in Computer Science, vol. 12036, Cham: Springer International Publishing, pp. 28–34, doi:10.1007/978-3-030-45442-5_4, ISBN   978-3-030-45441-8, PMC   7148026
  3. ↑ Grand, Adrien; Muir, Robert; Ferenczi, Jim; Lin, Jimmy (2020), "From MAXSCORE to Block-Max Wand: The Story of How Lucene Significantly Improved Query Evaluation Performance", in Jose, Joemon M.; Yilmaz, Emine; Magalhães, João; Castells, Pablo (eds.), Advances in Information Retrieval, Lecture Notes in Computer Science, vol. 12036, Cham: Springer International Publishing, pp. 20–27, doi:10.1007/978-3-030-45442-5_3, ISBN   978-3-030-45441-8, PMC   7148045
  4. ↑ Azzopardi, Leif; Moshfeghi, Yashar; Halvey, Martin; Alkhawaldeh, Rami S.; Balog, Krisztian; Di Buccio, Emanuele; Ceccarelli, Diego; Fernández-Luna, Juan M.; Hull, Charlie; Mannix, Jake; Palchowdhury, Sauparna (2017-02-14). "Lucene4IR: Developing Information Retrieval Evaluation Resources using Lucene". ACM SIGIR Forum. 50 (2): 58–75. doi:10.1145/3053408.3053421. ISSN   0163-5840. S2CID   212416159.
  5. ↑ "LuceneImplementations". apache.org. Archived from the original on 6 October 2015. Retrieved 23 September 2015.
  6. ↑ KeywordAnalyzer "Better Search with Apache Lucene and Solr" (PDF). 19 November 2007. Archived from the original (PDF) on 31 January 2012.
  7. ↑ Cutting, Doug (2019-06-07). "I wrote a couple of search engines at Xerox PARC, then V-Twin at Apple, then re-wrote Excite's search, then Lucene. So, Lucene might be considered V-Twin 3.0? Almost 25 years later, V-Twin still lives on as Mac OS X Search Kit!". @cutting. Retrieved 2019-06-19.
  8. ↑ Barker, Deane (2016). Web Content Management. O'Reilly. p. 233. ISBN   978-1491908105.
  9. ↑ "Apache Lucene - Welcome to Apache Lucene". apache.org. Archived from the original on 4 February 2016. Retrieved 4 February 2016.
  10. ↑ McCandless, Michael; Hatcher, Erik; Gospodnetić, Otis (2010). Lucene in Action, Second Edition . Manning. p.  8. ISBN   978-1933988177.
  11. ↑ "GNU/Linux Semantic Storage System" (PDF). glscube.org. Archived from the original (PDF) on 2010-06-01.
  12. ↑ "Apache Lucene - Query Parser Syntax". lucene.apache.org. Archived from the original on 2017-05-02.
  13. ↑ J. Beel, S. Langer, and B. Gipp, “The Architecture and Datasets of Docear’s Research Paper Recommender System,” in Proceedings of the 3rd International Workshop on Mining Scientific Publications (WOSP 2014) at the ACM/IEEE Joint Conference on Digital Libraries (JCDL 2014), London, UK, 2014
  14. 1 2 M. Schwarzer, M. Schubotz, N. Meuschke, C. Breitinger, V. Markl, and B. Gipp, https://www.gipp.com/wp-content/papercite-data/pdf/schwarzer2016.pdf "Evaluating Link-based Recommendations for Wikipedia" in Proceedings of the 16th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), New York, NY, USA, 2016, pp. 191-200.
  15. ↑ Wayner, Peter. "11 cutting-edge databases worth exploring now". InfoWorld. Archived from the original on 21 September 2015. Retrieved 21 September 2015.
  16. ↑ "Elasticsearch: RESTful, Distributed Search & Analytics - Elastic". elastic.co. Archived from the original on 8 October 2015. Retrieved 23 September 2015.
  17. ↑ "The Future of Compass & Elasticsearch". the dude abides. Archived from the original on 2015-10-15. Retrieved 2015-10-14.
  18. 1 2 Natividad, Angela. "Socialtext Updates Search, Goes Kino". CMS Wire. Archived from the original on 2012-09-29. Retrieved 2011-05-31.
  19. ↑ Marvin Humphrey. "KinoSearch - Search engine library. - metacpan.org". p3rl.org. Retrieved 23 September 2015.
  20. ↑ Diment, Kieren; Trout, Matt S (2009). "Catalyst Cookbook". The Definitive Guide to Catalyst . Apress. p.  280. ISBN   978-1-4302-2365-8.
  21. ↑ Wishart, D. S.; et al. (January 2009). "HMDB: a knowledgebase for the human metabolome". Nucleic Acids Res. 37 (Database issue): D603–10. doi:10.1093/nar/gkn810. PMC   2686599 . PMID   18953024.
  22. ↑ Lim, Emilia; Pon, Allison; Djoumbou, Yannick; Knox, Craig; Shrivastava, Savita; Guo, An Chi; Neveu, Vanessa; Wishart, David S. (January 2010). "T3DB: a comprehensively annotated database of common toxins and their targets". Nucleic Acids Res. 38 (Database issue): D781–6. doi:10.1093/nar/gkp934. PMC   2808899 . PMID   19897546.

Bibliography

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.