AoS and SoA

In computing, an array of structures (AoS), structure of arrays (SoA) or array of structures of arrays (AoSoA) are contrasting ways to arrange a sequence of records in memory, with regard to interleaving, and are of interest in SIMD and SIMT programming.

Structure of arrays

Main article: Parallel array

See also: Planar image format

Structure of arrays (SoA) is a layout separating elements of a record (or 'struct' in the C programming language) into one parallel array per field. ^[1] The motivation is easier manipulation with packed SIMD instructions in most instruction set architectures, since a single SIMD register can load homogeneous data, possibly transferred by a wide internal datapath (e.g. 128-bit). If only a specific part of the record is needed, only those parts need to be iterated over, allowing more data to fit onto a single cache line. The downside is requiring more cache ways when traversing data, and inefficient indexed addressing.

For example, to store N points in 3D space using a structure of arrays:

structpointlist3D{floatx[N];floaty[N];floatz[N];};structpointlist3Dpoints;floatget_point_x(inti){returnpoints.x[i];}

Array of structures

Array of structures (AoS) is the opposite (and more conventional) layout, in which data for different fields is interleaved. This is often more intuitive, and supported directly by most programming languages.

For example, to store N points in 3D space using an array of structures:

structpoint3D{floatx;floaty;floatz;};structpoint3Dpoints[N];floatget_point_x(inti){returnpoints[i].x;}

Array of structures of arrays

Array of structures of arrays (AoSoA) or tiled array of structs is a hybrid approach between the previous layouts, in which data for different fields is interleaved using tiles or blocks with size equal to the SIMD vector size. This is often less intuitive, but can achieve the memory throughput of the SoA approach, while being more friendly to the cache locality and load port architectures of modern processors. ^[2] In particular, memory requests in modern processors have to be fulfilled in fixed width (e.g., size of a cacheline ^[3] ). The tiled storage of AoSoA aligns the memory access pattern to the requests' fixed width, leading to fewer access operations to complete a memory request and thus increasing the efficiency. ^[4]

For example, to store N points in 3D space using an array of structures of arrays with a SIMD register width of 8 floats (or 8×32 = 256 bits):

structpoint3Dx8{floatx[8];floaty[8];floatz[8];};structpoint3Dx8points[(N+7)/8];floatget_point_x(inti){returnpoints[i/8].x[i%8];}

A different width may be needed depending on the actual SIMD register width. The interior arrays may be replaced with SIMD types such as float32x8 for languages with such support.

Alternatives

It is possible to split some subset of a structure (rather than each individual field) into a parallel array  – and this can actually improve locality of reference if different pieces of fields are used at different times in the program (see data oriented design).

Some SIMD architectures provide strided load/store instructions to load homogeneous data from the SoA format. Yet another option used in some Cell libraries is to de-interleave data from the AoS format when loading sources into registers, and interleave when writing out results (facilitated by the superscalar issue of permutes). Some vector maths libraries align floating point 4D vectors with the SIMD register to leverage the associated data path and instructions, while still providing programmer convenience, although this does not scale to SIMD units wider than four lanes.

4D vectors

AoS vs. SoA presents a choice when considering 3D or 4D vector data on machines with four-lane SIMD hardware. SIMD ISAs are usually designed for homogeneous data, however some provide a dot product instruction ^[5] and additional permutes, making the AoS case easier to handle.

Although most GPU hardware has moved away from 4D instructions to scalar SIMT pipelines, ^[6] modern compute kernels using SoA instead of AoS can still give better performance due to memory coalescing. ^[7]

Software support

Most languages support the AoS format more naturally by combining records and various array abstract data types.

SoA is mostly found in languages, libraries, or metaprogramming tools used to support a data-oriented design. Examples include:

• "Data frames," as implemented in R, Python's Pandas package, and Julia's DataFrames.jl package, are interfaces to access SoA like AoS.
• The Julia package StructArrays.jl allows for accessing SoA as AoS to combine the performance of SoA with the intuitiveness of AoS.
• Code generators for the C language, including Datadraw and the X Macro technique.

Automated creation of AoSoA is more complex. An example of AoSoA in metaprogramming is found in LANL's Cabana library written in C++; it assumes a vector width of 16 lanes by default. ^[8]

References

1. "How to Manipulate Data Structure to Optimize Memory Use". Intel. 2012-02-09. Retrieved 2019-03-17.
2. "Memory Layout Transformations". Intel. 2019-03-26. Retrieved 2019-06-02.
3. "Kernel Profiling Guide" (PDF). NVIDIA. 2022-12-01. Retrieved 2022-01-14.)
4. Fei, Yun (Raymond); Huang, Yuhan; Gao, Ming (2021), "Principles towards Real-Time Simulation of Material Point Method on Modern GPUs", pp. 1–16, arXiv: 2111.00699 [cs.GR]
5. "Intel SSE4 Floating Point Dot Product Intrinsics". Intel. Archived from the original on 2016-06-24. Retrieved 2019-03-17.
6. "Modern GPU Architecture (See Scalar Unified Pipelines)" (PDF). NVIDIA. Archived from the original (PDF) on 2018-05-17. Retrieved 2019-03-17.
7. Kim, Hyesoon (2010-02-08). "CUDA Optimization Strategies" (PDF). CS4803 Design Game Consoles. Retrieved 2019-03-17.
8. "ECP-copa/Cabana: AoSoA". GitHub.