Distcc

Last updated
distcc
Original author(s) Martin Pool
Developer(s) Fergus Henderson
Stable release
3.4 [1]   OOjs UI icon edit-ltr-progressive.svg / 11 May 2021;2 years ago (11 May 2021)
Repository
Written in C, C++, Python
Operating system Cross-platform
Type Compiler
License GNU General Public License
Website distcc.github.io

In software development, distcc is a tool for speeding up compilation of source code by using distributed computing over a computer network. With the right configuration, distcc can dramatically reduce a project's compilation time. [2]

Contents

It is designed to work with the C programming language (and its derivatives like C++ and Objective-C) and to use GCC as its backend, though it provides varying degrees of compatibility with the Intel C++ Compiler and Sun Microsystems' Sun Studio Compiler Suite. [3] Distributed under the terms of the GNU General Public License, distcc is free software.

Design

distcc is designed to speed up compilation by taking advantage of unused processing power on other computers. A machine with distcc installed can send code to be compiled across the network to a computer which has the distccd daemon and a compatible compiler installed. [4]

distcc works as an agent for the compiler. A distcc daemon has to run on each of the participating machines. The originating machine invokes a preprocessor to handle header files, preprocessing directives (such as #ifdef) and the source files and sends the preprocessed source to other machines over the network via TCP either unencrypted or using SSH. Remote machines compile those source files without any local dependencies (such as libraries, header files or macro definitions) to object files and send them back to the originator for further compilation. [5]

distcc version 3 supports a mode (called pump mode) in which included header files are sent to the remote machines, so that preprocessing is also distributed.

distcc was an option for distributed builds in versions of Apple's Xcode development suite prior to 4.3, but has been removed.

Goma

Goma is a similar tool made by Google to replace distcc & ccache in compiling chromium.

Ccache

ccache is another tool aimed to reduce the compilation time by caching the output from the same input source files. ccache can also use distcc as its backend, providing distributed compiling if it is not already cached by using the CCACHE_PREFIX environment variable.

Icecream

icecream was created by SUSE based on distcc. Like distcc, icecream takes compile jobs from a build and distributes it among remote machines allowing a parallel build. But unlike distcc, icecream uses a central server that dynamically schedules the compile jobs to the fastest free server. [6]

See also

Related Research Articles

In computer science, a preprocessor is a program that processes its input data to produce output that is used as input in another program. The output is said to be a preprocessed form of the input data, which is often used by some subsequent programs like compilers. The amount and kind of processing done depends on the nature of the preprocessor; some preprocessors are only capable of performing relatively simple textual substitutions and macro expansions, while others have the power of full-fledged programming languages.

rsync File synchronization protocol and software

rsync is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems and is under the GPL-3.0-or-later license.

<span class="mw-page-title-main">GNU Autotools</span> Suite of programming tools

The GNU Autotools, also known as the GNU Build System, is a suite of programming tools designed to assist in making source code packages portable to many Unix-like systems.

In software engineering, the terms frontend and backend refer to the separation of concerns between the presentation layer (frontend), and the data access layer (backend) of a piece of software, or the physical infrastructure or hardware. In the client–server model, the client is usually considered the frontend and the server is usually considered the backend, even when some presentation work is actually done on the server itself.

Ccache is a software development tool that caches compilations so that the next time, the same compilation can be avoided and the results can be taken from the cache. This can greatly speed up recompilation time. The detection is done by hashing different kinds of information that should be unique for the compilation and then using the hash sum to identify the cached output. Ccache is licensed under the GNU General Public License.

A compile farm is a server farm, a collection of one or more servers, which has been set up to compile computer programs remotely for various reasons. Uses of a compile farm include:

<span class="mw-page-title-main">LLVM</span> Compiler backend for multiple programming languages

LLVM is a set of compiler and toolchain technologies that can be used to develop a frontend for any programming language and a backend for any instruction set architecture. LLVM is designed around a language-independent intermediate representation (IR) that serves as a portable, high-level assembly language that can be optimized with a variety of transformations over multiple passes. The name LLVM originally stood for Low Level Virtual Machine, though the project has expanded and the name is no longer officially an initialism.

<span class="mw-page-title-main">SCons</span>

SCons is a computer software build tool that automatically analyzes source code file dependencies and operating system adaptation requirements from a software project description and generates final binary executables for installation on the target operating system platform. Its function is analogous to the traditional GNU build system based on the make utility and the autoconf tools.

HeaderDoc is a documentation generator developed and maintained by Apple Inc. Using specially commented source code files as input, HeaderDoc generates documentation for the code in HTML or XML format. Syntax for HeaderDoc comment tags is largely similar to, and as of HeaderDoc version 8, supportive of Javadoc tags. HeaderDoc 8.7 and later also provides partial support for many Doxygen tags. Apple's HeaderDoc project is free, open source software distributed under the Apple Public Source License.

<span class="mw-page-title-main">Waf (build system)</span>

Waf is a build automation tool designed to assist in the automatic compilation and installation of computer software. It is written in Python and maintained by Thomas Nagy.

CacheFS is the name used for several similar software technologies designed to speed up distributed file system file access for networked computers. These technologies operate by storing (cached) copies of files on secondary memory, typically a local hard disk, so that if a file is accessed again, it can be done locally at much higher speeds than networks typically allow.

Single compilation unit (SCU) is a computer programming technique for the C and C++ languages, which reduces compilation time for programs spanning multiple files. Specifically, it allows the compiler to keep data from shared header files, definitions and templates, so that it need not recreate them for each file. It is an instance of program optimization. The technique can be applied to an entire program or to some subset of source files; when applied to an entire program, it is also known as a unity build.

<span class="mw-page-title-main">BOINC client–server technology</span> BOINC volunteer computing client–server structure

BOINC client–server technology refers to the model under which BOINC works. The BOINC framework consists of two layers which operate under the client–server architecture. Once the BOINC software is installed in a machine, the server starts sending tasks to the client. The operations are performed client-side and the results are uploaded to the server-side.

The Java Development Kit (JDK) is a distribution of Java technology by Oracle Corporation. It implements the Java Language Specification (JLS) and the Java Virtual Machine Specification (JVMS) and provides the Standard Edition (SE) of the Java Application Programming Interface (API). It is derivative of the community driven OpenJDK which Oracle stewards. It provides software for working with Java applications. Examples of included software are the Java virtual machine, a compiler, performance monitoring tools, a debugger, and other utilities that Oracle considers useful for Java programmers.

The Berkeley Network, or Berknet, was an early wide area network, developed at the University of California, Berkeley in 1978, primarily by Eric Schmidt as part of his master's thesis work. The network continuously connected about a dozen computers running BSD and provided email, file transfer, printing and remote command execution services to its users, and it connected to the two other major networks in use at the time, the ARPANET and UUCPNET.

<span class="mw-page-title-main">Meson (software)</span> Build automation system

Meson is a software tool for automating the building (compiling) of software. The overall goal for Meson is to promote programmer productivity. Meson is free and open-source software written in Python, under the Apache License 2.0.

<span class="mw-page-title-main">Bazel (software)</span> Software tool that automates software builds and tests

Bazel is a free and open-source software tool used for the automation of building and testing software. Google uses the build tool Blaze internally and released an open-sourced port of the Blaze tool as Bazel, named as an anagram of Blaze. Bazel was first released in March 2015 and was in beta status by September 2015. Version 1.0 was released in October 2019.

In software engineering, a unity build is a method used in C and C++ software development to speed up the compilation of projects by combining multiple translation units into a single one, usually achieved by using include directives to bundle multiple source files into one larger file.

References

  1. "Release 3.4". 11 May 2021. Retrieved 21 May 2021.
  2. Laurence Bonney (2004-07-22). "Reduce compile time with distcc". IBM. Retrieved 2011-08-24.
  3. Martin Pool (2007-02-23). "distcc frequently asked questions". Distcc.googlecode.com. Archived from the original on 2009-09-07. Retrieved 2009-10-19.
  4. Daniel Robbins (2004-02-01). "Distcc & Distributed Computing". Dr.Dobb's. Retrieved 2011-08-24.
  5. V. L. Simpson (October 2004). "Speed Compiling with Distcc". linuxgazette.net. Retrieved 2011-08-25.
  6. icecream: Distributed compiler with a central scheduler to share build load, icecc, 2017-12-27, retrieved 2017-12-27