Distcc

Last updated
distcc
Original author(s) Martin Pool
Developer(s) Fergus Henderson
Stable release
3.4 [1]   OOjs UI icon edit-ltr-progressive.svg / 11 May 2021;3 years ago (11 May 2021)
Repository
Written in C, C++, Python
Operating system Cross-platform
Type Compiler
License GNU General Public License
Website distcc.github.io
Distcc Distcc3 log and help.png
Distcc

In software development, distcc is a tool for speeding up compilation of source code by using distributed computing over a computer network. With the right configuration, distcc can dramatically reduce a project's compilation time. [2]

Contents

It is designed to work with the C programming language (and its derivatives like C++ and Objective-C) and to use GCC as its backend, though it provides varying degrees of compatibility with the Intel C++ Compiler and Sun Microsystems' Sun Studio Compiler Suite. [3] Distributed under the terms of the GNU General Public License, distcc is free software.

Design

distcc is designed to speed up compilation by taking advantage of unused processing power on other computers. A machine with distcc installed can send code to be compiled across the network to a computer which has the distccd daemon and a compatible compiler installed. [4]

distcc works as an agent for the compiler. A distcc daemon has to run on each of the participating machines. The originating machine invokes a preprocessor to handle header files, preprocessing directives (such as #ifdef) and the source files and sends the preprocessed source to other machines over the network via TCP either unencrypted or using SSH. Remote machines compile those source files without any local dependencies (such as libraries, header files or macro definitions) to object files and send them back to the originator for further compilation. [5]

distcc version 3 supports a mode (called pump mode) in which included header files are sent to the remote machines, so that preprocessing is also distributed.

distcc was an option for distributed builds in versions of Apple's Xcode development suite prior to 4.3, but has been removed.

Goma

Goma is a similar tool made by Google to replace distcc & ccache in compiling chromium.

Ccache

ccache is another tool aimed to reduce the compilation time by caching the output from the same input source files. ccache can also use distcc as its backend, providing distributed compiling if it is not already cached by using the CCACHE_PREFIX environment variable.

Icecream

icecream was created by SUSE based on distcc. Like distcc, icecream takes compile jobs from a build and distributes it among remote machines allowing a parallel build. But unlike distcc, icecream uses a central server that dynamically schedules the compile jobs to the fastest free server. [6]

See also

Related Research Articles

In computer science, a preprocessor is a program that processes its input data to produce output that is used as input in another program. The output is said to be a preprocessed form of the input data, which is often used by some subsequent programs like compilers. The amount and kind of processing done depends on the nature of the preprocessor; some preprocessors are only capable of performing relatively simple textual substitutions and macro expansions, while others have the power of full-fledged programming languages.

rsync File synchronization protocol and software

rsync is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems and is under the GPL-3.0-or-later license.

<span class="mw-page-title-main">GNU Autotools</span> GNU software packaging tools

The GNU Autotools, also known as the GNU Build System, is a suite of programming tools designed to assist in making source code packages portable to many Unix-like systems.

In software development, Make is a command-line interface (CLI) software tool that performs actions ordered by configured dependencies as defined in a configuration file called a makefile. It is commonly used for build automation to build executable code from source code. But, not limited to building, Make can perform any operation available via the operating system shell.

In software engineering, the terms frontend and backend refer to the separation of concerns between the presentation layer (frontend), and the data access layer (backend) of a piece of software, or the physical infrastructure or hardware. In the client–server model, the client is usually considered the frontend and the server is usually considered the backend, even when some presentation work is actually done on the server itself.

The GNU toolchain is a broad collection of programming tools produced by the GNU Project. These tools form a toolchain used for developing software applications and operating systems.

A compile farm is a server farm, a collection of one or more servers, which has been set up to compile computer programs remotely for various reasons. Uses of a compile farm include:

<span class="mw-page-title-main">LLVM</span> Compiler backend for multiple programming languages

LLVM is a set of compiler and toolchain technologies that can be used to develop a frontend for any programming language and a backend for any instruction set architecture. LLVM is designed around a language-independent intermediate representation (IR) that serves as a portable, high-level assembly language that can be optimized with a variety of transformations over multiple passes. The name LLVM originally stood for Low Level Virtual Machine, though the project has expanded and the name is no longer officially an initialism.

<span class="mw-page-title-main">SCons</span>

SCons is a computer software build tool that automatically analyzes source code file dependencies and operating system adaptation requirements from a software project description and generates final binary executables for installation on the target operating system platform. Its function is analogous to the traditional GNU build system based on the make utility and the autoconf tools.

HeaderDoc is a documentation generator developed and maintained by Apple Inc. Using specially commented source code files as input, HeaderDoc generates documentation for the code in HTML or XML format. Syntax for HeaderDoc comment tags is largely similar to, and as of HeaderDoc version 8, supportive of Javadoc tags. HeaderDoc 8.7 and later also provides partial support for many Doxygen tags. Apple's HeaderDoc project is free, open source software distributed under the Apple Public Source License.

The Amsterdam Compiler Kit (ACK) is a retargetable compiler suite and toolchain written by Andrew Tanenbaum and Ceriel Jacobs, since 2005 maintained by David Given. It has frontends for the following programming languages: C, Pascal, Modula-2, Occam, and BASIC.

<span class="mw-page-title-main">Waf (build system)</span>

Waf is a build automation tool designed to assist in the automatic compilation and installation of computer software. It is written in Python and maintained by Thomas Nagy.

CacheFS is a family of software technologies designed to speed up distributed file system file access for networked computers. They store copies (caches) of files on secondary memory, typically a local hard disk, so that if a file is accessed again, it can be fetched locally at much higher speeds than networks typically allow.

<span class="mw-page-title-main">Software portability</span> Ability of a program to run on different platforms with little alteration

Software portability is a design objective for source code to be easily made to run on different platforms. An aid to portability is the generalized abstraction between the application logic and system interfaces. When software with the same functionality is produced for several computing platforms, portability is the key issue for development cost reduction.

Single compilation unit (SCU) is a computer programming technique for the C and C++ languages, which reduces compilation time for programs spanning multiple files. Specifically, it allows the compiler to keep data from shared header files, definitions and templates, so that it need not recreate them for each file. It is an instance of program optimization. The technique can be applied to an entire program or to some subset of source files; when applied to an entire program, it is also known as a unity build.

<span class="mw-page-title-main">BOINC client–server technology</span> BOINC volunteer computing client–server structure

BOINC client–server technology refers to the model under which BOINC works. The BOINC framework consists of two layers which operate under the client–server architecture. Once the BOINC software is installed in a machine, the server starts sending tasks to the client. The operations are performed client-side and the results are uploaded to the server-side.

The Java Development Kit (JDK) is a distribution of Java technology by Oracle Corporation. It implements the Java Language Specification (JLS) and the Java Virtual Machine Specification (JVMS) and provides the Standard Edition (SE) of the Java Application Programming Interface (API). It is derivative of the community driven OpenJDK which Oracle stewards. It provides software for working with Java applications. Examples of included software are the Java virtual machine, a compiler, performance monitoring tools, a debugger, and other utilities that Oracle considers useful for Java programmers.

<span class="mw-page-title-main">Meson (software)</span> Build automation system

Meson is a software tool for automating the building (compiling) of software. Meson adopts a convention over configuration approach, promoting standard development tools and practice instead of providing unlimited flexibility to the user. Meson is free and open-source software written in Python, under the Apache License 2.0.

References

  1. "Release 3.4". 11 May 2021. Retrieved 21 May 2021.
  2. Laurence Bonney (2004-07-22). "Reduce compile time with distcc". IBM. Retrieved 2011-08-24.
  3. Martin Pool (2007-02-23). "distcc frequently asked questions". Distcc.googlecode.com. Archived from the original on 2009-09-07. Retrieved 2009-10-19.
  4. Daniel Robbins (2004-02-01). "Distcc & Distributed Computing". Dr.Dobb's. Retrieved 2011-08-24.
  5. V. L. Simpson (October 2004). "Speed Compiling with Distcc". linuxgazette.net. Retrieved 2011-08-25.
  6. icecream: Distributed compiler with a central scheduler to share build load, icecc, 2017-12-27, retrieved 2017-12-27