Lexical density

Lexical density is a concept in computational linguistics that measures the structure and complexity of human communication in a language. ^[1] Lexical density estimates the linguistic complexity in a written or spoken composition from the functional words (grammatical units) and content words (lexical units, lexemes). One method to calculate the lexical density is to compute the ratio of lexical items to the total number of words. Another method is to compute the ratio of lexical items to the number of higher structural items in a composition, such as the total number of clauses in the sentences. ^{[2] [3]}

The lexical density for an individual evolves with age, education, communication style, circumstances, unusual injuries or medical condition, ^[4] and his or her creativity. The inherent structure of a human language and one's first language may impact the lexical density of the individual's writing and speaking style. Further, human communication in the written form is generally more lexically dense than in the spoken form after the early childhood stage. ^{[5] [6]} The lexical density impacts the readability of a composition and the ease with which the listener or reader can comprehend a communication. ^{[7] [8]} The lexical density may also impact the memorability and retention of a sentence and the message. ^[9]

Discussion

The lexical density is the proportion of content words (lexical items) in a given discourse. It can be measured either as the ratio of lexical items to total number of words, or as the ratio of lexical items to the number of higher structural items in the sentences (for example, clauses). ^{[2] [3]} A lexical item is typically the real content and it includes nouns, verbs, adjectives and adverbs. A grammatical item typically is the functional glue and thread that weaves the content and includes pronouns, conjunctions, prepositions, determiners, and certain classes of finite verbs and adverbs. ^[5]

Lexical density is one of the methods used in discourse analysis as a descriptive parameter which varies with register and genre. There are many proposed methods for computing the lexical density of any composition or corpus. Lexical density may be determined as:

${\displaystyle L_{d}=(N_{\mathrm {lex} }/N)\times 100}$

Where:

${\displaystyle L_{d}}$ = the analysed text's lexical density

${\displaystyle N_{\mathrm {lex} }}$ = the number of lexical or grammatical tokens (nouns, adjectives, verbs, adverbs) in the analysed text

${\displaystyle N}$ = the number of all tokens (total number of words) in the analysed text

Ure lexical density

Ure proposed the following formula in 1971 to compute the lexical density of a sentence:

L_d = The number of lexical items/The total number of words * 100

Biber terms this ratio as "type-token ratio". ^[10]

Halliday lexical density

In 1985, Halliday revised the denominator of the Ure formula and proposed the following to compute the lexical density of a sentence: ^[1]

L_d = The number of lexical items/The total number of clauses * 100

In some formulations, the Halliday proposed lexical density is computed as a simple ratio, without the "100" multiplier. ^{[2] [1]}

Characteristics

Lexical density measurements may vary for the same composition depending on how a "lexical item" is defined and which items are classified as lexical or as a grammatical item. Any adopted methodology when consistently applied across various compositions provides the lexical density of those compositions. Typically, the lexical density of a written composition is higher than a spoken composition. ^{[2] [3]} According to Ure, written forms of human communication in the English language typically have lexical densities above 40%, while spoken forms tend to have lexical densities below 40%. ^[2] In a survey of historical texts by Michael Stubbs, the typical lexical density of fictional literature ranged between 40% and 54%, while non-fiction ranged between 40% and 65%. ^{[3] [11] [12]}

The relation and intimacy between the participants of a particular communication impact the lexical density, states Ure, as do the circumstances prior to the start of communication for the same speaker or writer. The higher lexical density of written forms of communication, she proposed, is primarily because written forms of human communication involve greater preparation, reflection and revisions. ^[2] Human discussions and conversations involving or anticipating feedback tend to be sparser and have lower lexical density. In contrast, state Stubbs and Biber, instructions, law enforcement orders, news read from screen prompts within the allotted time, and literature that authors expect will be available to the reader for re-reading tend to maximize lexical density. ^{[2] [13]} In surveys of lexical density of spoken and written materials across different European countries and age groups, Johansson and Strömqvist report that the lexical density of population groups were similar and depended on the morphological structure of the native language and within a country, the age groups sampled. The lexical density was highest for adults, while the variations estimated as lexical diversity, states Johansson, were higher for teenagers for the same age group (13-year-olds, 17-year-olds). ^{[14] [15]}

See also

• Content analysis
• Succinctness
• Information structure
• Peirce's type-token distinction

References

1. Michael Halliday (1985). Spoken and Written Language. Deakin University. pp. 61–64. ISBN   978-0-7300-0309-0.
2. Erik Castello (2008). Text Complexity and Reading Comprehension Tests. Peter Lang. pp. 49–51. ISBN   978-3-03911-717-8.
3. Belinda Crawford Camiciottoli (2007). The Language of Business Studies Lectures: A Corpus-assisted Analysis. John Benjamins Publishing. p. 73. ISBN   978-90-272-5400-9.
4. Paul Yoder (2006). "Predicting Lexical Density Growth Rate in Young Children With Autism Spectrum Disorders". American Journal of Speech-Language Pathology. 15 (4): 362–373.
5. Michael Halliday (1985). Spoken and Written Language. Deakin University. pp. 61–75 (Chapter 5), 76–91 (Chapter 6). ISBN   978-0-7300-0309-0.
6. Victoria Johansson (2009). Developmental aspects of text production in writing and speech. Department of Linguistics and Phonetics, Centre for Languages and Literature, Lund University. pp. 1–16. ISBN   978-91-974116-7-7.
7. V To; S Fan; DP Thomas (2013). "Lexical density and Readability: A case study of English Textbooks". The International Journal of Language, Society and Culture. 37 (7): 61–71.
8. O'Loughlin, Kieran (1995). "Lexical density in candidate output on direct and semi-direct versions of an oral proficiency test". Language Testing. SAGE Publications. 12 (2): 217–237. doi:10.1177/026553229501200205. S2CID   145638000.
9. Perfetti, Charles A. (1969). "Lexical density and phrase structure depth as variables in sentence retention". Journal of Verbal Learning and Verbal Behavior. Elsevier BV. 8 (6): 719–724. doi:10.1016/s0022-5371(69)80035-6. ISSN   0022-5371.
10. Douglas Biber (2007). Discourse on the Move: Using Corpus Analysis to Describe Discourse Structure. John Benjamins Publishing. pp. 97–98 with footnote 7. ISBN   978-90-272-2302-9.
11. Mark Warschauer; Richard Kern (2000). Network-Based Language Teaching: Concepts and Practice. Cambridge University Press. pp. 107–108. ISBN   978-0-521-66742-5.
12. Michael Stubbs (1996). Text and Corpus Analysis: Computer Assisted Studies of Language and Culture. Wiley. pp. 71–73. ISBN   978-0-631-19512-2.
13. Michael Stubbs (1986). "Lexical density: A technique and some findings". In Malcolm Coulthard (ed.). Talking about Text. University of Birmingham: English Language Research. pp. 27–42.
14. Victoria Johansson (2008). "Lexical diversity and lexical density in speech and writing: a developmental perspective". Linguistics and Phonetics Working Papers. Lund University. 53: 61–79.
15. Sven Strömqvist; Victoria Johansson; Sarah Kriz; H Ragnarsdottir; Ravid Aisenmann; Dorit Ravid (2002). "Toward a crosslinguistic comparison of lexical quanta in speech and writing". Written Language and Literacy. 5: 45–67. doi:10.1075/wll.5.1.03str.

Further reading

• Ure, J (1971). Lexical density and register differentiation. In G. Perren and J.L.M. Trim (eds), Applications of Linguistics, London: Cambridge University Press. 443–452.

External links

• Lexical density 'Textalyser'