Dask (software)

Last updated
Dask
Original author(s) Matthew Rocklin
Developer(s) Dask
Initial release January 8, 2015;9 years ago (2015-01-08)
Stable release
2024.2.1 / February 23, 2024;6 months ago (2024-02-23)
Repository Dask Repository
Written in Python
Operating system Linux, Microsoft Windows, macOS
Available in Python
Type Data analytics
License New BSD
Website dask.org

Dask is an open-source Python library for parallel computing. Dask [1] scales Python code from multi-core local machines to large distributed clusters in the cloud. Dask provides a familiar user interface by mirroring the APIs of other libraries in the PyData ecosystem including: Pandas, scikit-learn and NumPy. It also exposes low-level APIs that help programmers run custom algorithms in parallel.

Contents

Dask was created by Matthew Rocklin [2] in December 2014 [3] and has over 9.8k stars and 500 contributors on GitHub. [4]

Dask is used by retail, financial, governmental organizations, as well as life science and geophysical institutes. Walmart, [5] Wayfair, [6] JDA, [7] GrubHub, [8] General Motors, [9] Nvidia, [10] Harvard Medical School, [7] Capital One [11] and NASA [12] are among the organizations that use Dask.

Overview

Dask has two parts: [13]

Dask's high-level parallel collections – DataFrames, [14] Bags, [15] and Arrays [16] – operate in parallel on datasets that may not fit into memory.

Dask’s task scheduler [10] executes task graphs in parallel. It can scale to thousand-node clusters. This powers the high-level collections as well as custom, user-defined workloads using low-level collections.

Dask collections

Dask supports several user interfaces [17] called high-level and low-level collections:

High-level

Low-level

Under the hood, each of these user interfaces adopts the same parallel computing machinery.

High-level collections

Dask's high-level collections are the natural entry point for users who are interested in scaling up their pandas, NumPy or scikit-learn workload. Dask’s DataFrame, Array and Dask-ML are alternatives to Pandas DataFrame, Numpy Array and scikit-learn respectively with slight variations to the original interfaces.

Dask Array

Dask Array [16] is a high-level collection that parallelizes array-based workloads and maintains the familiar NumPy API, such as slicing, arithmetic, reductions, mathematics, etc., making it easy for Numpy users to scale up array operations.

A Dask array comprises many smaller n-dimensional Numpy arrays and uses a blocked algorithm to enable computation on larger-than-memory arrays. During an operation, Dask translates the array operation into a task graph, breaks up large Numpy arrays into multiple smaller chunks, and executes the work on each chunk in parallel. Results from each chunk are combined to produce the final output.

Dask DataFrame

Dask DataFrame [14] is a high-level collection that parallelizes DataFrame based workloads. A Dask DataFrame comprises many smaller Pandas DataFrames partitioned along the index. It maintains the familiar Pandas API, making it easy for Pandas users to scale up DataFrame workloads. During a DataFrame operation, Dask creates a task graph and triggers operations on the constituent DataFrames in a manner that reduces memory footprint and increases parallelism through sharing and deleting of intermediate results.

Dask Bag

Dask Bag [15] is an unordered collection of repeated objects, a hybrid between a set and a list. Dask Bag is used to parallelize computation of semi-structured or unstructured data, such as JSON records, text data, log files or user-defined Python objects using operations such as filter, fold, map and groupby. Dask Bags can be created from an existing Python iterable or can load data directly from text files and binary files in the Avro format.

Low-level collections

The Dask low-level interface allows for more customization. It is suitable for data that does not fall within the scope of a Dask DataFrame, Bag or Array. Dask has the following low-level collections:

Delayed

Dask delayed [20] is an interface used to parallelize generic Python code that does not fit into high level collections like Dask Array or Dask DataFrame. Python functions decorated with Dask delayed adopt a lazy evaluation strategy by deferring execution and generating a task graph with the function and its arguments. The Python function will only execute when .compute is invoked. Dask delayed can be used as a function dask.delayed or as a decorator @dask.delayed.

Futures

Dask Futures, [21] an immediate (non-lazy) alternative to Dask delayed, provides a real-time task framework that extends Python’s concurrent.futures interface, which provides a high-level interface for asynchronous execution of callables.

It is common to combine high and low-level interfaces. For example, users can run Dask array/bag/dataframe to load and pre-process data, then switch to Dask delayed for a custom algorithm that is specific to their domain, then switch back to Dask array/dataframe to clean up and store results.

Scheduling

Dask’s high and low-level collections create a directed acyclic graph of tasks, [22] which represents the relationship between computation tasks. A node in a task graph represents a Python function that performs a unit of computation and an edge represents the data dependency between the upstream and downstream task. After a task graph is generated, the task scheduler manages the workflow based on the given task graph by assigning tasks to workers in a manner that improves parallelism and respects the data dependencies.

Dask provides two families of schedulers: single-machine scheduler and distributed scheduler.

Single-machine scheduler

A single-machine scheduler is the default scheduler which provides basic features on local processes or thread pool and is meant to be used on a single machine. It is simple and cheap to use but does not scale.

Local threads
A threaded scheduler leverages Python’s concurrent.futures.ThreadPoolExecuter to execute computations. It has a low memory footprint and does not require any setup. As all the computations occur in the same process, threaded schedulers incur minimal task overhead and no cost from transfer of data between tasks. Due to Python’s Global Interpreter Lock, local threads provide parallelism only when the computation is primarily non-Python code, which is the case for Pandas DataFrame, Numpy arrays or other Python/C/C++ based projects.
Local process
A multiprocessing scheduler leverages Python’s concurrent.futures.ProcessPoolExecutor to execute computations. Tasks and its dependencies are transferred from the main process to a local process, executed, and the results are transferred back to the main process. This allows bypassing of issues with Python’s Global Interpretable Lock and provides parallelism for computation tasks with primarily Python code. However, transferring data between the main and local processes degrades performance, especially in cases when the size of data transferred is large.
Single thread
A single threaded scheduler executes computation with no parallelism. It is used for debugging purposes.

Distributed scheduler

Dask’s distributed scheduler [23] can be set up on a local machine or scale out on a cluster. Dask can work with resource managers, such as Hadoop YARN, Kubernetes, or PBS, Slurm, SGD and LSF for High Performance Computing (HPC) clusters.

Dask-ML

Dask-ML is compatible with scikit-learn’s estimator API of fit, transform and predict and is well integrated with machine learning and deep learning frameworks such XGBoost, LightGBM, PyTorch, Keras, and TensorFlow through scikit-learn compatible wrappers.

Integrations

scikit-learn integration

Selected scikit-learn estimators and utilities can be parallelized [24] through executing jobs across multiple CPU cores using the Joblib library. The number of processes are determined by the n_jobs parameters. By default, the Joblib library uses loky as its multi-processing back-end. Dask offers an alternative Joblib backend which is useful for scaling of Joblib-backed scikit-learn algorithms out to a cluster of machines for compute constrained workloads.

For memory constrained workloads, Dask offers alternatives, such as Parallel Meta-estimators [25] for parallelizing and scaling out tasks that are not parallelized within scikit-learn and Incremental Hyperparameter Optimization [26] for scaling hyper-parameter search and parallelized estimators. [27]

XGBoost & LightGBM integrations

XGBoost [28] and LightGBM [29] are popular algorithms that are based on Gradient Boosting and both are integrated with Dask for distributed learning. Dask does not power XGBoost or LightGBM, rather it facilitates setting up of the cluster, scheduler, and workers required then hands off the data to the machine learning framework to perform distributed training.

Training an XGBoost model with Dask, [30] a Dask cluster is composed of a central scheduler and multiple distributed workers, is accomplished by spinning up an XGBoost scheduler in the same process running the Dask central scheduler and XGBoost worker in the same process running the Dask workers. Dask workers then hand over the Pandas DataFrame to the local XGBoost worker for distributed training.

PyTorch integration

Skorch [31] is a scikit-learn compatible wrapper for PyTorch, which enables Dask-ML to be used together with PyTorch.

Keras and TensorFlow integrations

SciKeras [32] is an scikit-learn compatible wrapper for Keras models which enables Dask-ML to be used with Keras.

Applications

Retail

Examples of retail use include:

Life sciences

Dask is used for high resolution, 4-dimensional, cellular imagery by Harvard Medical School, Howard Hughes Medical Institute, Chan Zuckerberg Initiative, and the UC Berkeley Advanced Bioimaging Center. [7] They record the evolution and movements of a 3-dimensional cell over time, in maximum detail. This generates large amounts of data that is difficult to analyze with traditional methods. Dask helps them scale their data analysis workflows with its API that resembles NumPy, Pandas, and scikit-learn code.

Dask is also used at the Novartis Institute for Biomedical Research to scale machine learning prototypes.

Finance industry

Geophysical sciences

Dask is used in Climate Science, Energy, Hydrology, Meteorology, and Satellite Imaging by companies such as NASA, LANL, PANGEO: [37] Earth Science and the UK Meteorology Office. [38]

Oceanographers produce massive simulated datasets of the Earth’s oceans and researchers can look at large seismology datasets from sensors around the world, collect a large number of observations from satellites and weather stations, and run big simulations.

Software libraries

Dask is integrated into many libraries, such as Pangeo [39] and xarray; [19] time series software like Prophet [40] and tsfresh; [41] ETL/ML software like scikit-learn, [42] RAPIDS, [43] and XGBoost; [28] workflow management tools like Apache Airflow [44] and Prefect. [45]

History

2014–2015

Dask [1] was originally developed at Continuum Analytics, a for-profit Python consulting company that eventually became Anaconda, Inc., [46] the creator of many open-source packages and the Anaconda Python distribution. Dask grew out of the Blaze [47] project, a DARPA [48] funded project to accelerate computation in open source.

Blaze was an ambitious project that tried to redefine computation, storage, compression, and data science APIs for Python, led originally by Travis Oliphant and Peter Wang, the co-founders of Anaconda. However, Blaze’s approach of being an ecosystem-in-a-package meant that it was harder for new users to easily adopt.

Instead of rewriting a software ecosystem, Dask’s team intended to augment the existing one with the right component. With this idea in mind, on December 21, 2014 [3] Matthew Rocklin created Dask. The purpose [49] of Dask was originally to parallelize NumPy so that it could use one full workstation computer, which was common in finance shops at the time.

2015–2017

The first projects to really adopt Dask were Xarray [50] (commonly used in geo sciences) and Scikit-Image [51] (commonly used in image processing). Dask was integrated into Xarray within a few months of being created. It provided Dask with its first user community, which remains to this day.

Understanding that there is demand for a lightweight parallelism solution for Pandas DataFrames [52] and machine learning tools, such as scikit-learn, [42] Dask quickly evolved to support other projects as well.

2018

Since 2018, other teams and institutions in academia, tech companies, and large corporations such as NASA, UK Met Office, Blue Yonder and Nvidia, became interested in Dask and began integrating it into their systems.

Dask received support from a diverse set of sources: [53] the US Government (DARPA grant), the Gordon and Betty Moore Foundation, Anaconda, NSF, NASA (US research grants with collaborations like Pangeo) and Nvidia.

2020–present

In 2020, Matthew Rocklin founded Coiled Computing, Inc. [54] to provide further support for Dask development and enable companies to deploy Dask clusters in the cloud. In May 2021, the company raised $21 million in series A funding led by Bessemer Venture Partners. [55]

Related Research Articles

<span class="mw-page-title-main">Python (programming language)</span> General-purpose programming language

Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation.

<span class="mw-page-title-main">NumPy</span> Python library for numerical programming

NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays. The predecessor of NumPy, Numeric, was originally created by Jim Hugunin with contributions from several other developers. In 2005, Travis Oliphant created NumPy by incorporating features of the competing Numarray into Numeric, with extensive modifications. NumPy is open-source software and has many contributors. NumPy is a NumFOCUS fiscally sponsored project.

In computer software, a general-purpose programming language (GPL) is a programming language for building software in a wide variety of application domains. Conversely, a domain-specific programming language (DSL) is used within a specific area. For example, Python is a GPL, while SQL is a DSL for querying relational databases.

<span class="mw-page-title-main">Orange (software)</span> Open-source data analysis software

Orange is an open-source data visualization, machine learning and data mining toolkit. It features a visual programming front-end for exploratory qualitative data analysis and interactive data visualization.

<span class="mw-page-title-main">Matplotlib</span> Library for creating static, animated, and interactive visualizations in Python.

Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. It provides an object-oriented API for embedding plots into applications using general-purpose GUI toolkits like Tkinter, wxPython, Qt, or GTK. There is also a procedural "pylab" interface based on a state machine, designed to closely resemble that of MATLAB, though its use is discouraged. SciPy makes use of Matplotlib.

In computer science, clamping, or clipping is the process of limiting a value to a range between a minimum and a maximum value. Unlike wrapping, clamping merely moves the point to the nearest available value.

<span class="mw-page-title-main">IPython</span> Advanced interactive shell for Python

IPython is a command shell for interactive computing in multiple programming languages, originally developed for the Python programming language, that offers introspection, rich media, shell syntax, tab completion, and history. IPython provides the following features:

<span class="mw-page-title-main">Cython</span> Programming language

Cython is a superset of the programming language Python, which allows developers to write Python code that yields performance comparable to that of C.

scikit-learn Python library for machine learning

scikit-learn is a free and open-source machine learning library for the Python programming language. It features various classification, regression and clustering algorithms including support-vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy. Scikit-learn is a NumFOCUS fiscally sponsored project.

pandas (software) Python library for data analysis

Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. The name is derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals, as well as a play on the phrase "Python data analysis". Wes McKinney started building what would become Pandas at AQR Capital while he was a researcher there from 2007 to 2010.

David Cournapeau is a data scientist. He is the original author of the scikit-learn package, an open source machine learning library in the Python programming language.

<span class="mw-page-title-main">XGBoost</span> Gradient boosting machine learning library

XGBoost is an open-source software library which provides a regularizing gradient boosting framework for C++, Java, Python, R, Julia, Perl, and Scala. It works on Linux, Microsoft Windows, and macOS. From the project description, it aims to provide a "Scalable, Portable and Distributed Gradient Boosting Library". It runs on a single machine, as well as the distributed processing frameworks Apache Hadoop, Apache Spark, Apache Flink, and Dask.

scikit-multiflow Machine learning library for data streams in Python

scikit-mutliflow is a free and open source software machine learning library for multi-output/multi-label and stream data written in Python.

LightGBM, short for Light Gradient-Boosting Machine, is a free and open-source distributed gradient-boosting framework for machine learning, originally developed by Microsoft. It is based on decision tree algorithms and used for ranking, classification and other machine learning tasks. The development focus is on performance and scalability.

CuPy is an open source library for GPU-accelerated computing with Python programming language, providing support for multi-dimensional arrays, sparse matrices, and a variety of numerical algorithms implemented on top of them. CuPy shares the same API set as NumPy and SciPy, allowing it to be a drop-in replacement to run NumPy/SciPy code on GPU. CuPy supports Nvidia CUDA GPU platform, and AMD ROCm GPU platform starting in v9.0.

References

  1. 1 2 "Dask". dask.org. Retrieved 2022-05-12.
  2. 1 2 "Matthew Rocklin - Bio". matthewrocklin.com. Retrieved 2022-05-12.[ permanent dead link ]
  3. 1 2 "GitHub, Dask, 2014". github.com. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  4. ↑ "GitHub, Dask, 2022". github.com. Archived from the original on 2022-08-31. Retrieved 2022-05-12.
  5. ↑ Caulfield, Brian. "Walmart, NVIDIA Discuss How They're Working Together to Transform Retail". blogs.nvidia.com. Archived from the original on 2022-05-21. Retrieved 2022-05-12.
  6. ↑ Sharma Meenakshi, Gonsalves Nick. "Transforming Model Training Workflows at Wayfair". aboutwayfair.com. Retrieved 2022-05-12.
  7. 1 2 3 Eswaramoorthy, Pavithra. "Who Uses Dask?". coiled.io. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  8. ↑ Bowne-Anderson, Hugo. "Dask and TensorFlow in Production at Grubhub". coiled.io. Retrieved 2022-05-12.
  9. ↑ "Companies Currently Using Dask". discovery.hgdata.com. Archived from the original on 2023-03-24. Retrieved 2022-05-12.
  10. 1 2 3 "DASK". nvidia.com. Retrieved 2022-05-12.
  11. ↑ Eswaramoorthy, Pavithra. "Distributed Machine Learning at Capital One". coiled.io. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  12. ↑ "Using Dask at NAS". nas.nasa.gov. Archived from the original on 2022-10-21. Retrieved 2022-05-12.
  13. ↑ "Scalable computing with Dask". ULHPC Tutorials. Archived from the original on 2022-08-29. Retrieved 2022-05-12.
  14. 1 2 "DataFrame - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  15. 1 2 "Bag - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  16. 1 2 "Array - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  17. ↑ Eswaramoorthy, Pavithra. "What is Dask?". coiled.io. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  18. ↑ "Dask-ML". ml.dask.org. Archived from the original on 2022-05-17. Retrieved 2022-05-12.
  19. 1 2 "Parallel computing with Dask". docs.xarray.dev. Archived from the original on 2022-05-16. Retrieved 2022-05-12.
  20. 1 2 "Delayed - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  21. 1 2 "Futures - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  22. ↑ "Specification - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  23. ↑ "Dask scheduler - Dask documentation". docs.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  24. ↑ "Computing with scikit-learn". scikit-learn.org. Retrieved 2022-05-12.
  25. ↑ "Parallel Prediction and Transformation - Dask documentation". ml.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  26. ↑ "Incremental Hyperparameter Optimization - Dask documentation". ml.dask.org. Archived from the original on 2022-06-19. Retrieved 2022-05-12.
  27. ↑ "API Reference - Dask documentation". ml.dask.org. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  28. 1 2 "Distributed XGBoost with Dask". XGBoost Tutorials. Archived from the original on 2022-04-25. Retrieved 2022-05-12.
  29. ↑ "How Distributed LightGBM Works. Dask". LightGBM. Distributed Learning Guide. Archived from the original on 2022-05-21. Retrieved 2022-05-12.
  30. ↑ Rocklin, Matthew. "Dask and Pandas and XGBoost". matthewrocklin.com. Retrieved 2022-05-12.
  31. ↑ "Skorch documentation". Skorch. Archived from the original on 2022-06-16. Retrieved 2022-05-12.
  32. ↑ "SciKeras documentation". adriangb.com. Archived from the original on 2022-10-06. Retrieved 2022-05-12.
  33. ↑ "Dask Usage at Blue Yonder". tech.blueyonder.com. 19 June 2020. Archived from the original on 2021-04-23. Retrieved 2022-05-12.
  34. ↑ Bowne-Anderson, Hugo. "Search at Grubhub and User Intent". coiled.io. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  35. ↑ McEntee Ryan, McCarty Mike. "Dask & RAPIDS: The Next Big Thing for Data Science & ML". capitalone.com. Archived from the original on 2022-05-16. Retrieved 2022-05-12.
  36. ↑ Patel, Harshil. "Which library should I use? Apache Spark, Dask, and Pandas Performance Compared (With Benchmarks)". censius.ai. Retrieved 2022-05-12.
  37. ↑ "Adapting Dask to Data Intensive Geoscience Research". coiled.wistia.com. Retrieved 2022-05-12.
  38. ↑ "Met Office". metoffice.gov.uk. Archived from the original on 2022-01-28. Retrieved 2022-05-12.
  39. ↑ "Pangeo". pangeo.io. Archived from the original on 2022-05-23. Retrieved 2022-05-12.
  40. ↑ "Forecasting with HEAVY.AI and Prophet". docs.heavy.ai. Archived from the original on 2022-05-23. Retrieved 2022-05-12.
  41. ↑ "Dask - the simple way. Tsfresh documentation". tsfresh.readthedocs.io. Archived from the original on 2022-06-19. Retrieved 2022-05-12.
  42. 1 2 "scikit-learn". scikit-learn. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  43. ↑ "Scale Python with Dask on GPUs". rapids.ai. Retrieved 2022-05-12.
  44. ↑ "Dask Executor - Apache Airflow Documentation". airflow.apache.org. Archived from the original on 2022-05-11. Retrieved 2022-05-12.
  45. ↑ "Deployment: Dask. Prefect Docs". docs.prefect.io. Archived from the original on 2022-06-06. Retrieved 2022-05-12.
  46. ↑ "Anaconda". anaconda.com. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  47. ↑ "The Blaze Ecosystem". blaze.pydata.org. Archived from the original on 2022-04-09. Retrieved 2022-05-12.
  48. ↑ "DARPA". darpa. Archived from the original on 2020-01-15. Retrieved 2022-05-12.
  49. ↑ "Dask History". Coiled. YouTube. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  50. ↑ "Xarray". xarray.pydata.org. Archived from the original on 2022-05-17. Retrieved 2022-05-12.
  51. ↑ "Image processing in Python". scikit-image. Archived from the original on 2022-05-11. Retrieved 2022-05-12.
  52. ↑ "pandas". pandas. Archived from the original on 2012-02-13. Retrieved 2022-05-12.
  53. ↑ Rocklin, Matthew. "Funding Dask, a brief history". matthewrocklin.com. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
  54. ↑ "Coiled: Python for Data Science on the Cloud with Dask". coiled.io. Retrieved 2022-05-12.
  55. ↑ Wiggers, Kyle. "Data and AI operations startup Coiled nabs $21M". VentureBeat. Archived from the original on 2022-05-12. Retrieved 2022-05-12.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.