Legal information retrieval

Last updated

Legal information retrieval is the science of information retrieval applied to legal text, including legislation, case law, and scholarly works. [1] Accurate legal information retrieval is important to provide access to the law to laymen and legal professionals. Its importance has increased because of the vast and quickly increasing amount of legal documents available through electronic means. [2] Legal information retrieval is a part of the growing field of legal informatics.

Contents

In a legal setting, it is frequently important to retrieve all information related to a specific query. However, commonly used boolean search methods (exact matches of specified terms) on full text legal documents have been shown to have an average recall rate as low as 20 percent, [3] meaning that only 1 in 5 relevant documents are actually retrieved. In that case, researchers believed that they had retrieved over 75% of relevant documents. [3] This may result in failing to retrieve important or precedential cases. In some jurisdictions this may be especially problematic, as legal professionals are ethically obligated to be reasonably informed as to relevant legal documents. [4]

Legal Information Retrieval attempts to increase the effectiveness of legal searches by increasing the number of relevant documents (providing a high recall rate) and reducing the number of irrelevant documents (a high precision rate). This is a difficult task, as the legal field is prone to jargon, [5] polysemes [6] (words that have different meanings when used in a legal context), and constant change.

Techniques used to achieve these goals generally fall into three categories: boolean retrieval, manual classification of legal text, and natural language processing of legal text.

Problems

Application of standard information retrieval techniques to legal text can be more difficult than application in other subjects. One key problem is that the law rarely has an inherent taxonomy. [7] Instead, the law is generally filled with open-ended terms, which may change over time. [7] This can be especially true in common law countries, where each decided case can subtly change the meaning of a certain word or phrase. [8]

Legal information systems must also be programmed to deal with law-specific words and phrases. Though this is less problematic in the context of words which exist solely in law, legal texts also frequently use polysemes, words may have different meanings when used in a legal or common-speech manner, potentially both within the same document. The legal meanings may be dependent on the area of law in which it is applied. For example, in the context of European Union legislation, the term "worker" has four different meanings: [9]

  1. Any worker as defined in Article 3(a) of Directive 89/391/EEC who habitually uses display screen equipment as a significant part of his normal work.
  2. Any person employed by an employer, including trainees and apprentices but excluding domestic servants;
  3. Any person carrying out an occupation on board a vessel, including trainees and apprentices, but excluding port pilots and shore personnel carrying out work on board a vessel at the quayside;
  4. Any person who, in the Member State concerned, is protected as an employee under national employment law and in accordance with national practice;

It also has the common meaning:

  1. A person who works at a specific occupation. [9]

Though the terms may be similar, correct information retrieval must differentiate between the intended use and irrelevant uses in order to return the correct results.

Even if a system overcomes the language problems inherent in law, it must still determine the relevancy of each result. In the context of judicial decisions, this requires determining the precedential value of the case. [10] Case decisions from senior or superior courts may be more relevant than those from lower courts, even where the lower court's decision contains more discussion of the relevant facts. [10] The opposite may be true, however, if the senior court has only a minor discussion of the topic (for example, if it is a secondary consideration in the case). [10] An information retrieval system must also be aware of the authority of the jurisdiction. A case from a binding authority is most likely of more value than one from a non-binding authority.

Additionally, the intentions of the user may determine which cases they find valuable. For instance, where a legal professional is attempting to argue a specific interpretation of law, he might find a minor court's decision which supports his position more valuable than a senior courts position which does not. [10] He may also value similar positions from different areas of law, different jurisdictions, or dissenting opinions. [10]

Overcoming these problems can be made more difficult because of the large number of cases available. The number of legal cases available via electronic means is constantly increasing (in 2003, US appellate courts handed down approximately 500 new cases per day [2] ), meaning that an accurate legal information retrieval system must incorporate methods of both sorting past data and managing new data. [2] [11]

Techniques

Boolean searches

Boolean searches, where a user may specify terms such as use of specific words or judgments by a specific court, are the most common type of search available via legal information retrieval systems. They are widely implemented but overcome few of the problems discussed above.

The recall and precision rates of these searches vary depending on the implementation and searches analyzed. One study found a basic boolean search's recall rate to be roughly 20%, and its precision rate to be roughly 79%. [3] Another study implemented a generic search (that is, not designed for legal uses) and found a recall rate of 56% and a precision rate of 72% among legal professionals. Both numbers increased when searches were run by non-legal professionals, to a 68% recall rate and 77% precision rate. This is likely explained because of the use of complex legal terms by the legal professionals. [12]

Manual classification

In order to overcome the limits of basic boolean searches, information systems have attempted to classify case laws and statutes into more computer friendly structures. Usually, this results in the creation of an ontology to classify the texts, based on the way a legal professional might think about them. [13] These attempt to link texts on the basis of their type, their value, and/or their topic areas. Most major legal search providers now implement some sort of classification search, such as Westlaw's “Natural Language” [14] or LexisNexis' Headnote [15] searches. Additionally, both of these services allow browsing of their classifications, via Westlaw's West Key Numbers [14] or Lexis' Headnotes. [15] Though these two search algorithms are proprietary and secret, it is known that they employ manual classification of text (though this may be computer-assisted). [13]

These systems can help overcome the majority of problems inherent in legal information retrieval systems, in that manual classification has the greatest chances of identifying landmark cases and understanding the issues that arise in the text. [16] In one study, ontological searching resulted in a precision rate of 82% and a recall rate of 97% among legal professionals. [17] The legal texts included, however, were carefully controlled to just a few areas of law in a specific jurisdiction. [18]

The major drawback to this approach is the requirement of using highly skilled legal professionals and large amounts of time to classify texts. [16] [19] As the amount of text available continues to increase, some have stated their belief that manual classification is unsustainable. [20]

Natural language processing

In order to reduce the reliance on legal professionals and the amount of time needed, efforts have been made to create a system to automatically classify legal text and queries. [2] [21] [22] Adequate translation of both would allow accurate information retrieval without the high cost of human classification. These automatic systems generally employ Natural Language Processing (NLP) techniques that are adapted to the legal domain, and also require the creation of a legal ontology. Though multiple systems have been postulated, [2] [21] [22] few have reported results. One system, “SMILE,” which attempted to automatically extract classifications from case texts, resulted in an f-measure (which is a calculation of both recall rate and precision) of under 0.3 (compared to perfect f-measure of 1.0). [23] This is probably much lower than an acceptable rate for general usage. [23] [24]

Despite the limited results, many theorists predict that the evolution of such systems will eventually replace manual classification systems. [25] [26]

Citation-Based ranking

In the mid-90s the Room 5 case law retrieval project used citation mining for summaries and ranked its search results based on citation type and count. This slightly pre-dated the Page Rank algorithm at Stanford which was also a citation-based ranking. Ranking of results was based as much on jurisdiction as on number of references. [27]

Notes

  1. ↑ Maxwell, K.T., and Schafer, B. 2009, p. 1
  2. 1 2 3 4 5 Jackson et al., p. 60
  3. 1 2 3 Blair, D.C., and Maron, M.E., 1985, p.293
  4. ↑ American Bar Association, Model Rules of Professional Conduct Rule 1.1, http://www.abanet.org/cpr/mrpc/rule_1_1.html
  5. ↑ Peters, W. et al. 2007, p. 118
  6. ↑ Peters, W. et al. 2007, p. 130
  7. 1 2 Peters, W. et al. 2007, p. 120
  8. ↑ Saravanan, M. et al. 2009, p. 101
  9. 1 2 Peters, W. et al. 2007, p. 131
  10. 1 2 3 4 5 Maxwell, K.T., and Schafer, B. 2008, p. 8
  11. ↑ Maxwell, K.T., and Schafer, B. 2007, p.1
  12. ↑ Saravanan M., et al. 2009, p. 116
  13. 1 2 Maxwell, K.T., and Schafer, B. 2008, p. 2
  14. 1 2 Westlaw Research, http://www.westlaw.com
  15. 1 2 Lexis Research, http://www.lexisnexis.com
  16. 1 2 Maxwell, K.T., and Schafer, B. 2008, p. 3
  17. ↑ Saravanan, M. et al. 2009, p. 116
  18. ↑ Saravanan, M. et al. 2009, p. 103
  19. ↑ Schweighofer, E. and Liebwald, D. 2008, p. 108
  20. ↑ Maxwell, K.T., and Schafer, B. 2008, p. 4
  21. 1 2 Ashley, K.D. and Bruninghaus, S. 2009, p. 125
  22. 1 2 Gelbart, D. and Smith, J.C. 1993, p. 142
  23. 1 2 Ashley, K.D. and Bruninghaus, S. 2009, p. 159
  24. ↑ Maxwell, K.T., and Schafer, B. 2009, p. 3
  25. ↑ Maxwell, K.T., and Schafer, B. 2009, p. 9
  26. ↑ Ashley, K.D. and Bruninghaus, S. 2009, p. 126
  27. ↑ Loui, R. P., Norman, J., Altepeter, J., Pinkard, D., Craven, D., Linsday, J., & Foltz, M. (1997, June). Progress on Room 5: A testbed for public interactive semi-formal legal argumentation. In Proceedings of the 6th international conference on Artificial intelligence and law (pp. 207-214). ACM.

Related Research Articles

Information retrieval (IR) in computing and information science is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. Searches can be based on full-text or other content-based indexing. Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for the metadata that describes data, and for databases of texts, images or sounds.

Accuracy and precision are two measures of observational error. Accuracy is how close a given set of measurements are to their true value, while precision is how close the measurements are to each other.

<span class="mw-page-title-main">Information science</span> Academic field concerned with collection and analysis of information

Information science is an academic field which is primarily concerned with analysis, collection, classification, manipulation, storage, retrieval, movement, dissemination, and protection of information. Practitioners within and outside the field study the application and the usage of knowledge in organizations in addition to the interaction between people, organizations, and any existing information systems with the aim of creating, replacing, improving, or understanding the information systems.

<span class="mw-page-title-main">Westlaw</span> Online legal research service

Westlaw is an online legal research service and proprietary database for lawyers and legal professionals available in over 60 countries. Information resources on Westlaw include more than 40,000 databases of case law, state and federal statutes, administrative codes, newspaper and magazine articles, public records, law journals, law reviews, treatises, legal forms and other information resources.

Latent semantic analysis (LSA) is a technique in natural language processing, in particular distributional semantics, of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. LSA assumes that words that are close in meaning will occur in similar pieces of text. A matrix containing word counts per document is constructed from a large piece of text and a mathematical technique called singular value decomposition (SVD) is used to reduce the number of rows while preserving the similarity structure among columns. Documents are then compared by cosine similarity between any two columns. Values close to 1 represent very similar documents while values close to 0 represent very dissimilar documents.

Document retrieval is defined as the matching of some stated user query against a set of free-text records. These records could be any type of mainly unstructured text, such as newspaper articles, real estate records or paragraphs in a manual. User queries can range from multi-sentence full descriptions of an information need to a few words.

In text retrieval, full-text search refers to techniques for searching a single computer-stored document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases.

Named-entity recognition (NER) (also known as (named)entity identification, entity chunking, and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.

Legal informatics is an area within information science.

<span class="mw-page-title-main">Outline of library and information science</span> Overview of and topical guide to library science

The following outline is provided as an overview of and topical guide to library science:

<span class="mw-page-title-main">F-score</span> Statistical measure of a tests accuracy

In statistical analysis of binary classification, the F-score or F-measure is a measure of a test's accuracy. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the number of all positive results, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified as positive. Precision is also known as positive predictive value, and recall is also known as sensitivity in diagnostic binary classification.

Query expansion (QE) is the process of reformulating a given query to improve retrieval performance in information retrieval operations, particularly in the context of query understanding. In the context of search engines, query expansion involves evaluating a user's input and expanding the search query to match additional documents. Query expansion involves techniques such as:

A focused crawler is a web crawler that collects Web pages that satisfy some specific property, by carefully prioritizing the crawl frontier and managing the hyperlink exploration process. Some predicates may be based on simple, deterministic and surface properties. For example, a crawler's mission may be to crawl pages from only the .jp domain. Other predicates may be softer or comparative, e.g., "crawl pages about baseball", or "crawl pages with large PageRank". An important page property pertains to topics, leading to 'topical crawlers'. For example, a topical crawler may be deployed to collect pages about solar power, swine flu, or even more abstract concepts like controversy while minimizing resources spent fetching pages on other topics. Crawl frontier management may not be the only device used by focused crawlers; they may use a Web directory, a Web text index, backlinks, or any other Web artifact.

<span class="mw-page-title-main">Precision and recall</span> Pattern recognition performance metrics

In pattern recognition, information retrieval, object detection and classification, precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space.

Computer-assisted legal research (CALR) or computer-based legal research is a mode of legal research that uses databases of court opinions, statutes, court documents, and secondary material. Electronic databases make large bodies of case law easily available. Databases also have additional benefits, such as Boolean searches, evaluating case authority, organizing cases by topic, and providing links to cited material. Databases are available through paid subscription or for free.

A concept search is an automated information retrieval method that is used to search electronically stored unstructured text for information that is conceptually similar to the information provided in a search query. In other words, the ideas expressed in the information retrieved in response to a concept search query are relevant to the ideas contained in the text of the query.

Ranking of query is one of the fundamental problems in information retrieval (IR), the scientific/engineering discipline behind search engines. Given a query q and a collection D of documents that match the query, the problem is to rank, that is, sort, the documents in D according to some criterion so that the "best" results appear early in the result list displayed to the user. Ranking in terms of information retrieval is an important concept in computer science and is used in many different applications such as search engine queries and recommender systems. A majority of search engines use ranking algorithms to provide users with accurate and relevant results.

Fuzzy retrieval techniques are based on the Extended Boolean model and the Fuzzy set theory. There are two classical fuzzy retrieval models: Mixed Min and Max (MMM) and the Paice model. Both models do not provide a way of evaluating query weights, however this is considered by the P-norms algorithm.

Evaluation measures for an information retrieval (IR) system assess how well an index, search engine or database returns results from a collection of resources that satisfy a user's query. They are therefore fundamental to the success of information systems and digital platforms. The success of an IR system may be judged by a range of criteria including relevance, speed, user satisfaction, usability, efficiency and reliability. However, the most important factor in determining a system's effectiveness for users is the overall relevance of results retrieved in response to a query. Evaluation measures may be categorised in various ways including offline or online, user-based or system-based and include methods such as observed user behaviour, test collections, precision and recall, and scores from prepared benchmark test sets.

Query understanding is the process of inferring the intent of a search engine user by extracting semantic meaning from the searcher’s keywords. Query understanding methods generally take place before the search engine retrieves and ranks results. It is related to natural language processing but specifically focused on the understanding of search queries. Query understanding is at the heart of technologies like Amazon Alexa, Apple's Siri. Google Assistant, IBM's Watson, and Microsoft's Cortana.

References

See also

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.