Jq (programming language)

Last updated
jq
Jq logo.svg
Official jq logo
Paradigms Purely functional, JSON-oriented processing, tacit
Designed by Stephen Dolan
First appearedAugust 21, 2012;13 years ago (2012-08-21)
Stable release
1.8.1 [1]   OOjs UI icon edit-ltr-progressive.svg / 1 July 2025
Typing discipline dynamic
Memory management automatic
Scope lexical
Implementation language C
Platform Cross-platform [2]
OS Cross-platform [note 1]
License MIT
Website jqlang.org

jq is a widely-used command-line utility and very high-level, functional, domain-specific programming language designed for processing JSON data. jq filters its input data to produce modified output in a manner similar to AWK or sed, but operates on JSON values, rather than lines. In jq, programs consist of "filters" that can be composed in pipelines that perform a variety of operations on their inputs. [3] [4]

Contents

History

jq was created by Stephen Dolan, and released in October 2012. [5] [6] It was described as being "like sed for JSON data". [7] Support for regular expressions was added in jq version 1.5.

The original implementation of jq was in Haskell [8] before being ported to the language C.

Usage

Command-line usage

jq is typically used at the command line and can be used with other command-line utilities, such as curl. Here is an example showing how the output of a curl command can be piped to a jq filter to determine the category names associated with this Wikipedia page:

$ curl-s'https://en.wikipedia.org/w/api.php?action=parse&page=jq_(programming_language)&format=json'|jq'.parse.categories[]."*"'

The output produced by this pipeline consists of a stream of JSON strings, the first few of which are:

"Articles_with_short_description""Short_description_matches_Wikidata""Dynamically_typed_programming_languages""Functional_languages""Programming_languages""Programming_languages_created_in_2012""Query_languages""2012_software"

The curl command above uses the MediaWiki API for this page to produce a JSON response. The pipe | allows the output of curl to be accessed by jq, a standard Unix shell mechanism. [9]

The jq filter shown is an abbreviation for the jq pipeline:

.["parse"] | .["categories"] | .[] | .["*"]

This corresponds to the nested JSON structure produced by the call to curl. Notice that the jq pipeline is constructed in the same manner using the | character as the Unix-style pipeline.

Embedded usage

jq provides a C API, libjq, allowing it to be used from C. [4]

Modes of operation

jq by default acts as a "stream editor" for JSON inputs, much like the sed utility can be thought of as a "stream editor" for lines of text. However jq has several other modes of operation:

  1. it can treat its input from one or more sources as lines of text;
  2. it can gather a stream of inputs from a specified source into a JSON array;
  3. it can parse its JSON inputs using a so-called "streaming parser" that produces a stream of [path, value] arrays for all "leaf" paths.

The streaming parser is very useful when one of more of the JSON inputs is too large to fit in memory, since its memory needs are usually quite small. For example, for an arbitrarily large array of JSON objects, the peak memory need is little more than needed to handle the largest top-level object.

These modes of operation can, within certain limitations, be combined.

Syntax and semantics

Types

Every JSON value is also a value in jq, which accordingly has the data types shown in the table below. [4]

Summary of jq's supported types
TypeExamples
"number"
  • 3
  • 3.2
  • 1e6
  • nan
  • infinite
"string"
  • "Hello"
  • "😐"
"boolean"
  • true
  • false
"array"
  • [1,"2",{"mixed":"type"},[3,4]]
"object"
  • {"one":1,"two":"2","three":[3]}
"null"
  • null

null is a value, just like any other JSON scalar; it is not a pointer or a "null pointer". nan (corresponding to NaN) and infinite (see IEEE 754) are the only two jq scalars that are not also JSON values.

Forms

Special syntactic forms exist for function creation, conditionals, stream reduction, and the module system.

Filters

Here is an example of defining a named, parameterized filter for formatting an integer in any base from 2 to 36 inclusive. The implementation illustrates tacit (or point-free) programming:

def tobase($b):     def digit: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"[.:.+1];     def mod: . % $b;     def div: ((. - mod) / $b);     def digits: recurse( select(. >= $b) | div) | mod ;      select(2 <= $b and $b <= 36)     | [digits | digit] | reverse | add;

The next example demonstrates the use of generators in the classic "SEND MORE MONEY" verbal arithmetic game:

def send_more_money:     def choose(m;n;used): ([range(m;n+1)] - used)[];     def num(a;b;c;d): 1000*a + 100*b + 10*c + d;     def num(a;b;c;d;e): 10*num(a;b;c;d) + e;     first(       1 as $m       | 0 as $o       | choose(8;9;[]) as $s       | choose(2;9;[$s]) as $e       | choose(2;9;[$s,$e]) as $n       | choose(2;9;[$s,$e,$n]) as $d       | choose(2;9;[$s,$e,$n,$d]) as $r       | choose(2;9;[$s,$e,$n,$d,$r]) as $y       | select(num($s;$e;$n;$d) + num($m;$o;$r;$e) ==                num($m;$o;$n;$e;$y))       | [$s,$e,$n,$d,$m,$o,$r,$e,$m,$o,$n,$e,$y] );

jq has inspired several clones and similar programs.

gojq reimplements much of jq in Go . [10] It also supports processing YAML files.

jaq is a Rust implementation of jq developed using denotational semantics to formalize its behavior in cases where the original jq's documentation is unclear or does not match its behavior. [10]

Mike Farah's yq is a jq-like program that supports several file formats, including JSON, YAML, and XML. [11] Its syntax is not fully compatible with jq. [10]

Andrey Kislyuk's yq provides a collection of wrapper scripts that use jq to process YAML, XML, or TOML files. [12] [13]

Notes

  1. ↑ Including, but not limited to Windows, Linux, macOS, FreeBSD, and Solaris. [2]

References

Bibliography

  • Janssens, Jeroen (2021). Data Science at the Command Line. O'Reilly Media. ISBN   9781492087885.
  • Janssens, Jeroen (2014). Data Science at the Command Line: Facing the Future with Time-Tested Tools. O'Reilly Media. ISBN   9781491947807.
  • Marrs, Tom (2017). JSON at Work: Practical Data Integration for the Web. O'Reilly Media. ISBN   9781491982419.

Others

  1. ↑ https://github.com/jqlang/jq/blob/master/NEWS.md.{{cite web}}: Missing or empty |title= (help)
  2. 1 2 "Download jq". jq. Archived from the original on 10 September 2025. Retrieved 27 September 2025.
  3. ↑ Homer, Michael (19 October 2023). Branching Compositional Data Transformations in jq, Visually . Proceedings of the 2nd ACM SIGPLAN International Workshop on Programming Abstractions and Interactive Notations, Tools, and Environments. New York: Association for Computing Machinery. pp. 11–16. doi:10.1145/3623504.3623567. ISBN   9798400703997 . Retrieved 27 September 2025.
  4. 1 2 3 "jq 1.8 Manual". jq. Archived from the original on 16 September 2025. Retrieved 27 September 2025.
  5. ↑ Janssens 2014.
  6. ↑ "jqlang". GitHub: jqlang. Retrieved January 6, 2023.
  7. ↑ "like sed". Archived from the original on 2013-04-14.
  8. ↑ "Initial: jqlang/Jq@eca89ac". GitHub: jqlang.
  9. ↑ "Tutorial". jq. Retrieved January 6, 2023.
  10. 1 2 3 Färber, Michael (2023). "Denotational Semantics and a fast interpreter for jq". arXiv: 2302.10576 [cs.LO].
  11. ↑ Farah, Mike. "yq". yq. Archived from the original on 20 August 2025. Retrieved 27 September 2025.
  12. ↑ Asher, Ben (9 January 2022). "Querying JSON and XML with jq and xq". Ashby. Archived from the original on 29 April 2025. Retrieved 27 September 2025.
  13. ↑ Andrey, Kislyuk. "yq: Command-line YAML/XML/TOML processor - jq wrapper for YAML, XML, TOML documents". yq documentation. Archived from the original on 26 August 2025. Retrieved 27 September 2025.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.