Bencode

Last updated December 28, 2024

Bencode (pronounced like Bee-encode) is the encoding used by the peer-to-peer file sharing system BitTorrent for storing and transmitting loosely structured data.^[1]

Bencoding is simple and (because numbers are encoded as text in decimal notation) is unaffected by endianness, which is important for a cross-platform application like BitTorrent. It is also fairly flexible, as long as applications ignore unexpected dictionary keys, so that new ones can be added without creating incompatibilities.

Encoding Algorithm

Bencode uses ASCII characters as delimiters and digits to encode data structures in a simple and compact format.

Integers are encoded as i<base10 integer>e.
- The integer is encoded in base 10 and may be negative (indicated by a leading hyphen-minus).
- Leading zeros are not allowed unless the integer is zero.
- Examples:
  - Zero is encoded as i0e.
  - The number 42 is encoded as i42e.
  - Negative forty-two is encoded as i-42e.

Byte Strings are encoded as <length>:<contents>.
- The length is the number of bytes in the string, encoded in base 10.
- A colon (:) separates the length and the contents.
- The contents are the exact number of bytes specified by the length.
- Examples:
  - An empty string is encoded as 0:.
  - The string "bencode" is encoded as 7:bencode.

Lists are encoded as l<elements>e.
- Begins with l and ends with e.
- Elements are bencoded values concatenated without delimiters.
- Examples:
  - An empty list is encoded as le.
  - A list containing the string "bencode" and the integer -20 is encoded as l7:bencodei-20ee.

Dictionaries are encoded as d<pairs>e.
- Begins with d and ends with e.
- Contains key-value pairs.
- Keys are byte strings and must appear in lexicographical order.
- Each key is immediately followed by its value, which can be any bencoded type.
- Examples:
  - An empty dictionary is encoded as de.
  - A dictionary with keys "wiki" → "bencode" and "meaning" → 42 is encoded as d4:wiki7:bencode7:meaningi42ee.

There are no restrictions on the types of values stored within lists and dictionaries; they may contain other lists and dictionaries, allowing for arbitrarily complex data structures.

Types of Errors in Bencode

Here is the list of the possible errors that a ill-formatted bencode may have:

Null root value.
Non-singular root item.
Invalid type encountered (character not 'i', 'l', 'd', or '0'-'9').
Missing 'e' terminator for 'i', 'l', or 'd' types.
Integer errors:
1. Contains non-digit characters.
2. Has a leading zero.
3. Is negative zero.
Byte String errors:
1. Negative length.
2. Length not followed by ':'.
3. Unexpected EOF before completing string.
Dictionary errors:
1. Key is not a string.
2. Duplicate keys.
3. Keys not sorted.
4. Missing value for a key.

Features

Bencode is a very specialized kind of binary coding with some unique properties:

For each possible (complex) value, there is only a single valid bencoding; i.e. there is a bijection between values and their encodings. This has the advantage that applications may compare bencoded values by comparing their encoded forms, eliminating the need to decode the values.
Bencoding serves similar purposes as data languages like JSON and YAML, allowing complex yet loosely structured data to be stored in a platform independent way. This allowing a linear memory storage for complex data.

Drawbacks

Many BE codegroups can be decoded manually. Since the bencoded values often contain binary data, decoding may become quite complex. Bencode is not considered a human-readable encoding format.
The specification deals with encoding characters in the ASCII set only leaving the solution to users. This brings about several solutions and less conformity.

However, this uniqueness can cause some problems:

There are very few bencode editors^[2]
Because bencoded files contain binary data, and because of some of the intricacies involved in the way binary strings are typically stored, it is often not safe to edit bencode files in text editors.

Related Research Articles

In computer programming, a string is traditionally a sequence of characters, either as a literal constant or as some kind of variable. The latter may allow its elements to be mutated and the length changed, or it may be fixed. A string is generally considered as a data type and is often implemented as an array data structure of bytes that stores a sequence of elements, typically characters, using some character encoding. String may also denote more general arrays or other sequence data types and structures.

UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit. Almost every webpage is stored in UTF-8.

A computer number format is the internal representation of numeric values in digital device hardware and software, such as in programmable computers and calculators. Numerical values are stored as groupings of bits, such as bytes and words. The encoding between numerical values and bit patterns is chosen for convenience of the operation of the computer; the encoding used by the computer's instruction set generally requires conversion for external use, such as for printing and display. Different types of processors may have different internal representations of numerical values and different conventions are used for integer and real numbers. Most calculations are carried out with number formats that fit into a processor register, but some software systems allow representation of arbitrarily large numbers using multiple words of memory.

Abstract Syntax Notation One (ASN.1) is a standard interface description language (IDL) for defining data structures that can be serialized and deserialized in a cross-platform way. It is broadly used in telecommunications and computer networking, and especially in cryptography.

Lempel–Ziv–Welch (LZW) is a universal lossless data compression algorithm created by Abraham Lempel, Jacob Ziv, and Terry Welch. It was published by Welch in 1984 as an improved implementation of the LZ78 algorithm published by Lempel and Ziv in 1978. The algorithm is simple to implement and has the potential for very high throughput in hardware implementations. It is the algorithm of the Unix file compression utility compress and is used in the GIF image format.

In computer programming, Base64 is a group of binary-to-text encoding schemes that transforms binary data into a sequence of printable characters, limited to a set of 64 unique characters. More specifically, the source binary data is taken 6 bits at a time, then this group of 6 bits is mapped to one of 64 unique characters.

The Lempel–Ziv–Markov chain algorithm (LZMA) is an algorithm used to perform lossless data compression. It has been used in the 7z format of the 7-Zip archiver since 2001. This algorithm uses a dictionary compression scheme somewhat similar to the LZ77 algorithm published by Abraham Lempel and Jacob Ziv in 1977 and features a high compression ratio and a variable compression-dictionary size, while still maintaining decompression speed similar to other commonly used compression algorithms.

External Data Representation (XDR) is a standard data serialization format, for uses such as computer network protocols. It allows data to be transferred between different kinds of computer systems. Converting from the local representation to XDR is called encoding. Converting from XDR to the local representation is called decoding. XDR is implemented as a software library of functions which is portable between different operating systems and is also independent of the transport layer.

A Java class file is a file containing Java bytecode that can be executed on the Java Virtual Machine (JVM). A Java class file is usually produced by a Java compiler from Java programming language source files containing Java classes. If a source file has more than one class, each class is compiled into a separate class file. Thus, it is called a .class file because it contains the bytecode for a single class.

Ascii85, also called Base85, is a form of binary-to-text encoding developed by Paul E. Rutter for the btoa utility. By using five ASCII characters to represent four bytes of binary data, it is more efficient than uuencode or Base64, which use four characters to represent three bytes of data.

In the macOS, iOS, NeXTSTEP, and GNUstep programming frameworks, property list files are files that store serialized objects. Property list files use the filename extension .plist, and thus are often referred to as p-list files.

KLV (Key-Length-Value) is a data encoding standard, often used to embed information in video feeds. The standard uses a type–length–value encoding scheme. Items are encoded into Key-Length-Value triplets, where key identifies the data, length specifies the data's length, and value is the data itself. It is defined in SMPTE 336M-2007, approved by the Society of Motion Picture and Television Engineers. Due to KLV's large degree of interoperability, it has also been adopted by the Motion Imagery Standards Board.

Action Message Format (AMF) is a binary format used to serialize object graphs such as ActionScript objects and XML, or send messages between an Adobe Flash client and a remote service, usually a Flash Media Server or third party alternatives. The Actionscript 3 language provides classes for encoding and decoding from the AMF format.

BSON is a computer data interchange format. The name "BSON" is based on the term JSON and stands for "Binary JSON". It is a binary form for representing simple or complex data structures including associative arrays, integer indexed arrays, and a suite of fundamental scalar types. BSON originated in 2009 at MongoDB. Several scalar data types are of specific interest to MongoDB and the format is used both as a data storage and network transfer format for the MongoDB database, but it can be used independently outside of MongoDB. Implementations are available in a variety of languages such as C, C++, C#, D, Delphi, Erlang, Go, Haskell, Java, JavaScript, Julia, Lua, OCaml, Perl, PHP, Python, Ruby, Rust, Scala, Smalltalk, and Swift.

LEB128 or Little Endian Base 128 is a variable-length code compression used to store arbitrarily large integers in a small number of bytes. LEB128 is used in the DWARF debug file format and the WebAssembly binary encoding for all integer literals.

In the BitTorrent file distribution system, a torrent file or meta-info file is a computer file that contains metadata about files and folders to be distributed, and usually also a list of the network locations of trackers, which are computers that help participants in the system find each other and form efficient distribution groups called swarms. Torrent files are normally named with the extension .torrent.

Virtual Token Descriptor for eXtensible Markup Language (VTD-XML) refers to a collection of cross-platform XML processing technologies centered on a non-extractive XML, "document-centric" parsing technique called Virtual Token Descriptor (VTD). Depending on the perspective, VTD-XML can be viewed as one of the following:

Universal Binary JSON (UBJSON) is a computer data interchange format. It is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to achieve the generality of JSON, combined with being much easier to process than JSON.

Concise Binary Object Representation (CBOR) is a binary data serialization format loosely based on JSON authored by Carsten Bormann and Paul Hoffman. Like JSON it allows the transmission of data objects that contain name–value pairs, but in a more concise manner. This increases processing and transfer speeds at the cost of human readability. It is defined in IETF RFC 8949.

JData is a light-weight data annotation and exchange open-standard designed to represent general-purpose and scientific data structures using human-readable (text-based) JSON and (binary) UBJSON formats. JData specification specifically aims at simplifying exchange of hierarchical and complex data between programming languages, such as MATLAB, Python, JavaScript etc. It defines a comprehensive list of JSON-compatible "name":value constructs to store a wide range of data structures, including scalars, N-dimensional arrays, sparse/complex-valued arrays, maps, tables, hashes, linked lists, trees and graphs, and support optional data grouping and metadata for each data element. The generated data files are compatible with JSON/UBJSON specifications and can be readily processed by most existing parsers. JData-defined annotation keywords also permit storage of strongly-typed binary data streams in JSON, data compression, linking and referencing.

References

↑ The BitTorrent Protocol Specification Archived 2019-07-26 at the Wayback Machine . BitTorrent.org. Retrieved 8 October 2018.
↑ "BEncode Editor". μTorrent Community Forums. 8 October 2007. Archived from the original on 24 October 2014. Retrieved 24 October 2014.

External links

Bencoding specification
File_Bittorrent2 - Another PHP Bencode/decode implementation
The original BitTorrent implementation in Python as standalone package
Torrent File Editor cross-platform GUI editor for BEncode files
bencode-tools - a C library for manipulating bencoded data and a XML schema like validator for bencode messages in Python
Bento - Bencode library in Elixir.
Beecoder - the file stream parser that de/encoding "B-encode" data format on Java using java.io.* stream Api.
Bencode parsing in Java
Bencode library in Scala
Bencode parsing in C
There are numerous Perl implementations on CPAN

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] The BitTorrent Protocol Specification Archived 2019-07-26 at the Wayback Machine . BitTorrent.org. Retrieved 8 October 2018.

[2] "BEncode Editor". μTorrent Community Forums. 8 October 2007. Archived from the original on 24 October 2014. Retrieved 24 October 2014.

[1]

[2]