![]() The official jq logo | |
Paradigms | Purely functional programming, JSON-oriented processing, tacit programming |
---|---|
Designed by | Stephen Dolan |
First appeared | August 21, 2012 |
Stable release | |
Implementation language | jq: C gojq: Go jaq: Rust jqjq: jq |
Platform | Cross-platform [note 1] |
OS | Cross-platform [note 2] |
License | MIT [note 3] |
Website | jqlang |
jq is a very high-level lexically scoped functional programming language in which every JSON value is a constant. jq supports backtracking and managing indefinitely long streams of JSON data. It is related to the Icon and Haskell programming languages. The language supports a namespace-based module system and has some support for closures. In particular, functions and functional expressions can be used as parameters of other functions.
The original implementation of jq was in Haskell [3] before being immediately ported to C.
jq was created by Stephen Dolan, and released in October 2012. [4] [5] It was described as being "like sed for JSON data". [6] Support for regular expressions was added in jq version 1.5.
A "wrapper" program for jq named yq adds support for YAML, XML and TOML. It was first released in 2017. [7]
The Go implementation, gojq, was initially released in 2019. [8] gojq notably extends jq to include support for YAML.
The Rust implementation, jaq, has as its project goals a faster and more correct implementation of jq, while preserving compatibility with jq in most cases. Explicitly excluded from the project goals as of March 2024 are certain advanced features of jq such as modules, SQL-style operators, and a streaming parser for very large JSON documents. [9]
The jq implementation, jqjq, was initially released in 2022. jqjq notably can run itself, has a REPL and supports eval.
jq is typically used at the command line and can be used with other command-line utilities, such as curl. Here is an example showing how the output of a curl
command can be piped to a jq filter to determine the category names associated with this Wikipedia page:
$ curl'https://en.wikipedia.org/w/api.php?action=parse&page=jq_(programming_language)&format=json'|jq'.parse.categories[]."*"'
The output produced by this pipeline consists of a stream of JSON strings, the first few of which are:
"Articles_with_short_description""Short_description_matches_Wikidata""Dynamically_typed_programming_languages""Functional_languages""Programming_languages""Programming_languages_created_in_2012""Query_languages""2012_software"
The curl
command above uses the MediaWiki API for this page to produce a JSON response. The pipe |
allows the output of curl
to be accessed by jq, a standard Unix shell mechanism. [10]
The jq filter shown is an abbreviation for the jq pipeline:
.["parse"] | .["categories"] | .[] | .["*"]
This corresponds to the nested JSON structure produced by the call to curl
. Notice that the jq pipeline is constructed in the same manner using the |
character as the Unix-style pipeline.
Both the C and the Go implementations provide libraries so that jq functionality can be embedded in other applications and programming environments.
For example, gojq has been integrated with SQLite so that a jq
function is available in SQL statements. [11] This function is marked as "deterministic" and can therefore be used in "CREATE INDEX" commands. [12]
jq by default acts as a "stream editor" for JSON inputs, much like the sed utility can be thought of as a "stream editor" for lines of text. However jq has several other modes of operation:
The "streaming parser" is particularly useful when one of more of the JSON inputs is too large to fit into memory, since its memory requirements are typically quite small. For example, for an arbitrarily large array of JSON objects, the peak memory requirement is not much more than required to handle the largest top-level object.
These modes of operation can, within certain limitations, be combined.
Every JSON value is itself a value in jq, which accordingly has the types shown in the table below. [13] The gojq and jaq implementations distinguish between integers and non-integer numbers. The gojq implementation supports unbounded-precision integer arithmetic, as did the original implementation of jq in Haskell.
Type | Examples |
---|---|
"number" |
|
"string" |
|
"boolean" |
|
"array" |
|
"object" |
|
"null" |
|
null
is a value, just like any other JSON scalar; it is not a pointer or a "null-pointer". nan
(corresponding to NaN) and infinite
(see IEEE 754) are the only two jq scalars that are not also JSON values.
There are special syntactic forms for function creation, conditionals, stream reduction, and the module system.
Here is an example which shows how to define a named, parameterized filter for formatting an integer in any base from 2 to 36 inclusive. The implementation illustrates tacit (or point-free) programming:
# Use gojq for infinite precision integer arithmetic def tobase($b): def digit: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"[.:.+1]; def mod: . % $b; def div: ((. - mod) / $b); def digits: recurse( select(. >= $b) | div) | mod ; select(2 <= $b and $b <= 36) | [digits | digit] | reverse | add;
The next example demonstrates the use of generators in the classic "SEND MORE MONEY" verbal arithmetic game:
def send_more_money: def choose(m;n;used): ([range(m;n+1)] - used)[]; def num(a;b;c;d): 1000*a + 100*b + 10*c + d; def num(a;b;c;d;e): 10*num(a;b;c;d) + e; first( 1 as $m | 0 as $o | choose(8;9;[]) as $s | choose(2;9;[$s]) as $e | choose(2;9;[$s,$e]) as $n | choose(2;9;[$s,$e,$n]) as $d | choose(2;9;[$s,$e,$n,$d]) as $r | choose(2;9;[$s,$e,$n,$d,$r]) as $y | select(num($s;$e;$n;$d) + num($m;$o;$r;$e) == num($m;$o;$n;$e;$y)) | [$s,$e,$n,$d,$m,$o,$r,$e,$m,$o,$n,$e,$y] );
There is a very close relationship between jq and the parsing expression grammar (PEG) formalism. [14] The relationship stems from the equivalence of the seven basic PEG operations and the jq constructs shown in the following table.
PEG operation name | PEG notation | jq operation or def |
---|---|---|
Sequence | e1e2 | e1 | e2 |
Ordered choice | e1/e2 | e1 // e2 |
Zero-or-more | e* | def star(E): (E | star(E)) // . ; |
One-or-more | e+ | def plus(E): E | (plus(E) // . ); |
Optional | e? | def optional(E): E // .; |
And-predicate | &e | def amp(E): . as $in | E | $in; |
Not-predicate | !e | def neg(E): select( [E] == [] ); |
gojq is a "pure Go" implementation. There is also a Rust implementation of a dialect of jq named jaq [9] for which a denotational semantics has been specified. [15]
Rebol is a cross-platform data exchange language and a multi-paradigm dynamic programming language designed by Carl Sassenrath for network communications and distributed computing. It introduces the concept of dialecting: small, optimized, domain-specific languages for code and data, which is also the most notable property of the language according to its designer Carl Sassenrath:
Although it can be used for programming, writing functions, and performing processes, its greatest strength is the ability to easily create domain-specific languages or dialects
sed is a Unix utility that parses and transforms text, using a simple, compact programming language. It was developed from 1973 to 1974 by Lee E. McMahon of Bell Labs, and is available today for most operating systems. sed was based on the scripting features of the interactive editor ed and the earlier qed. It was one of the earliest tools to support regular expressions, and remains in use for text processing, most notably with the substitution command. Popular alternative tools for plaintext string manipulation and "stream editing" include AWK and Perl.
OCaml is a general-purpose, high-level, multi-paradigm programming language which extends the Caml dialect of ML with object-oriented features. OCaml was created in 1996 by Xavier Leroy, Jérôme Vouillon, Damien Doligez, Didier Rémy, Ascánder Suárez, and others.
YAML(see § History and name) is a human-readable data serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. YAML targets many of the same communications applications as Extensible Markup Language (XML) but has a minimal syntax that intentionally differs from Standard Generalized Markup Language (SGML). It uses Python-style indentation to indicate nesting and does not require quotes around most string values.
Apache Groovy is a Java-syntax-compatible object-oriented programming language for the Java platform. It is both a static and dynamic language with features similar to those of Python, Ruby, and Smalltalk. It can be used as both a programming language and a scripting language for the Java Platform, is compiled to Java virtual machine (JVM) bytecode, and interoperates seamlessly with other Java code and libraries. Groovy uses a curly-bracket syntax similar to Java's. Groovy supports closures, multiline strings, and expressions embedded in strings. Much of Groovy's power lies in its AST transformations, triggered through annotations.
DIGITAL Command Language (DCL) is the standard command language adopted by many of the operating systems created by Digital Equipment Corporation. DCL had its roots in IAS, TOPS-20, and RT-11 and was implemented as a standard across most of Digital's operating systems, notably RSX-11 and RSTS/E, but took its most powerful form in VAX/VMS. DCL continues to be developed by VSI as part of OpenVMS.
In computer-based language recognition, ANTLR, or ANother Tool for Language Recognition, is a parser generator that uses a LL(*) algorithm for parsing. ANTLR is the successor to the Purdue Compiler Construction Tool Set (PCCTS), first developed in 1989, and is under active development. Its maintainer is Professor Terence Parr of the University of San Francisco.
A lightweight markup language (LML), also termed a simple or humane markup language, is a markup language with simple, unobtrusive syntax. It is designed to be easy to write using any generic text editor and easy to read in its raw form. Lightweight markup languages are used in applications where it may be necessary to read the raw document as well as the final rendered output.
Raku is a member of the Perl family of programming languages. Formerly known as Perl 6, it was renamed in October 2019. Raku introduces elements of many modern and historical languages. Compatibility with Perl was not a goal, though a compatibility mode is part of the specification. The design process for Raku began in 2000.
JSON is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays. It is a commonly used data format with diverse uses in electronic data interchange, including that of web applications with servers.
In the macOS, iOS, NeXTSTEP, and GNUstep programming frameworks, property list files are files that store serialized objects. Property list files use the filename extension .plist
, and thus are often referred to as p-list files.
Tacit programming, also called point-free style, is a programming paradigm in which function definitions do not identify the arguments on which they operate. Instead the definitions merely compose other functions, among which are combinators that manipulate the arguments. Tacit programming is of theoretical interest, because the strict use of composition results in programs that are well adapted for equational reasoning. It is also the natural style of certain programming languages, including APL and its derivatives, and concatenative languages such as Forth. The lack of argument naming gives point-free style a reputation of being unnecessarily obscure, hence the epithet "pointless style".
PowerShell is a task automation and configuration management program from Microsoft, consisting of a command-line shell and the associated scripting language. Initially a Windows component only, known as Windows PowerShell, it was made open-source and cross-platform on August 18, 2016, with the introduction of PowerShell Core. The former is built on the .NET Framework, the latter on .NET.
Nemerle is a general-purpose, high-level, statically typed programming language designed for platforms using the Common Language Infrastructure (.NET/Mono). It offers functional, object-oriented, aspect-oriented, reflective and imperative features. It has a simple C#-like syntax and a powerful metaprogramming system.
Amber Smalltalk, formerly named Jtalk, is an implementation of the programming language Smalltalk-80, that runs on the JavaScript runtime of a web browser. It is designed to enable client-side development using Smalltalk. The programming environment in Amber is named Helios.
Elixir is a functional, concurrent, high-level general-purpose programming language that runs on the BEAM virtual machine, which is also used to implement the Erlang programming language. Elixir builds on top of Erlang and shares the same abstractions for building distributed, fault-tolerant applications. Elixir also provides tooling and an extensible design. The latter is supported by compile-time metaprogramming with macros and polymorphism via protocols.
JSON streaming comprises communications protocols to delimit JSON objects built upon lower-level stream-oriented protocols, that ensures individual JSON objects are recognized, when the server and clients use the same one. This is necessary as JSON is a non-concatenative protocol.
Nim is a general-purpose, multi-paradigm, statically typed, compiled high-level systems programming language, designed and developed by a team around Andreas Rumpf. Nim is designed to be "efficient, expressive, and elegant", supporting metaprogramming, functional, message passing, procedural, and object-oriented programming styles by providing several features such as compile time code generation, algebraic data types, a foreign function interface (FFI) with C, C++, Objective-C, and JavaScript, and supporting compiling to those same languages as intermediate representations.
Ballerina is an open source general-purpose programming language designed by WSO2 for cloud-era application programmers.
JData is a light-weight data annotation and exchange open-standard designed to represent general-purpose and scientific data structures using human-readable (text-based) JSON and (binary) UBJSON formats. JData specification specifically aims at simplifying exchange of hierarchical and complex data between programming languages, such as MATLAB, Python, JavaScript etc. It defines a comprehensive list of JSON-compatible "name":value
constructs to store a wide range of data structures, including scalars, N-dimensional arrays, sparse/complex-valued arrays, maps, tables, hashes, linked lists, trees and graphs, and support optional data grouping and metadata for each data element. The generated data files are compatible with JSON/UBJSON specifications and can be readily processed by most existing parsers. JData-defined annotation keywords also permit storage of strongly-typed binary data streams in JSON, data compression, linking and referencing.