Data Storage

The general structure of Data Storage is described in the section Terms & Structure: Data Storage. This page contains additional information about Data Storage.

Using the Interface

When a Data Storage interface concept is used in a function’s argument list, it describes the minimum requirements that Data Storage objects must meet in order to be used to call the function.

void func1(alpaka::concepts::IDataSource auto input)
{
    alpaka::unused(input);
}

void func2(alpaka::concepts::IView auto input)
{
    alpaka::unused(input);
}

int main()
{
    // fulfill the IDataSource interface, but not the IMdSpan interface
    alpaka::LinearizedIdxGenerator linearizedIdxGenerator = otherLinearizedIdxGenerator;
    // fulfil the IDataSource and the IMdSpan interface, but not the IView interface
    alpaka::MdSpan mdpsan = otherMdspan;
    // fulfil the IDataSource, the IMdSpan and the IView interface, but not the IBuffer interface
    alpaka::View view = otherView;
    // fulfil the IDataSource, the IMdSpan, the IView and the IBuffer interface
    alpaka::onHost::SharedBuffer sharedBuffer = otherSharedBuffer;

    func1(linearizedIdxGenerator);
    func1(mdpsan);
    func1(view);
    func1(sharedBuffer);

    // does not compile, linearizedIdxGenerator does not fulfil the IView interface
    // func2(linearizedIdxGenerator);
    // does not compile, mdspan does not fulfil the IView interface
    // func2(mdpsan);
    func2(view);
    func2(sharedBuffer);
    return 0;
}

Read-only access via const annotation

All objects that implement a Data Storage interface are const-correct. If either the value type is const (e.g., alpaka::MdSpan<float const, TExtent, TPitch, TAlignment>) or the Data Storage type itself is annotated with const (e.g., alpaka::MdSpan<float, TExtent, TPitch, TAlignment> const), the values in the storage cannot be changed.

Special case: mutable alpaka::concepts::IDataSource

alpaka::concepts::IDataSource only requires that data can be read from a Data Storage object, but not written to it, regardless of the const annotation. However, there are valid cases where a function argument only requires alpaka::concepts::IDataSource in the general case but depending on other arguments can require mutability as well.

// same interface like alpaka::onAcc::SimdAlgo.concurrent()
ALPAKA_FN_INLINE ALPAKA_FN_ACC constexpr void concurrent(
    auto const& acc,
    auto&& func,
    alpaka::concepts::IDataSource auto&& data0,
    alpaka::concepts::IDataSource auto&&... dataN)
{
    auto simdGrid = alpaka::onAcc::SimdAlgo{alpaka::onAcc::worker::threadsInGrid};
    simdGrid.concurrent(acc, data0.getExtents(), func, data0, dataN...);
}

// user defined
struct SimdCopyOp
{
    constexpr void operator()(auto const&, auto const a, auto c) const
    {
        c = a.load();
    }
};

// user defined
struct Kernel
{
    ALPAKA_FN_ACC void operator()(
        auto const& acc,
        auto const& func,
        alpaka::concepts::IDataSource auto const& in,
        alpaka::concepts::IMdSpan auto out) const
    {
        concurrent(acc, func, in, out);
    }
};

int main()
{
    // ...

    queue.enqueue(frameSpec, alpaka::KernelBundle{Kernel{}, SimdCopyOp{}, in, out});

    return 0;
}

For example, alpaka::onAcc::SimdAlgo.concurrent() requires the IDataSource interface for the Data Storage object. The IDataSource interface only supports reading data. Depending on the user-defined functor, some of the Data Storage objects must be writable, so they must implement the IMdSpan interface. However, the IMdSpan interface would prevent the user from using generators that only implement the IDataSource interface. Therefore, the minimum requirement must be the non-constant IDataSource interface.

Memory Layout of multidimensional Data Storage

There are several functions and parameters for improving the memory layout of multidimensional Data Storage to enhance application performance. alpaka supports Pitches, which optimize memory loads, and Alignment, which is required for vector operations such as AVX on CPUs. All alpaka functions automatically handle Pitches and Alignment during memory access. However, it is sometimes necessary to process raw memory, for example, when a memory pointer is passed from alpaka to non-alpaka code. The following section explains how alpaka implements Pitches and Alignment.

Pitches

Pitches are a mechanism that enables transparent, optimized data loading for multidimensional Data Storage. Depending on the hardware, a processor can load a specific number of bytes with a single load command. The size of the user data does not always correspond to the size of the load command. If less user data is required than the load command can load, so-called padding bytes are added to fill up the load command.

For example, an Nvidia GPU can load 128 bytes with a single load command. If we have a matrix with 32-bit integers (4 bytes) and 5 rows with 30 elements each, each row requires 120 bytes. To ensure that each row can be loaded with a single load command and that no data from a second row is loaded, which would result in some rows requiring multiple load commands, 8 bytes of padding bytes are added. The same principals are required for vectorization on CPU.

When the padding bytes are added, the Extents can no longer be used to calculate the memory location of a specific element. The Extents assume that the memory is contiguous. Pitches solves this problem: it is a multidimensional value where each component stores the number of bytes required to jump to the next element within the corresponding dimension, including the padding bytes. The innermost dimension (x) pitch is sizeof(value_type), the next dimension (y) pitch is the byte-stride of a full row including padding, and so on for higher dimensions. In general, given an N-dimensional zero-based index, the dot product (element-wise multiplication and sum) of the index with the pitches yields the byte offset from the start of the buffer to that data element. In the simplest case, when 1D Data Storage is used, the size of the value type in bytes is the pitch value. For example, if the value type is float (32 bits), the pitch is 4 bytes. With 2D Data Storage, the first pitch dimension, Pitch[0] (row stride), stores the size of a row in bytes, i.e., number_of_elements_in_x * sizeof(value_type) + padding_bytes, so adding Pitch[0] to a memory pointer jumps one row forward. Pitch[1] (column stride) is again the size of the value type.

Compared to the above example from Nvidia, we have chosen slightly different numbers for the example for illustrative purposes. The example has 3 rows with 5 elements each. Each element has a size of 4 bytes. 2 bytes of padding are added to each row. Therefore, the size of a row is 5 elements * 4 Byte/element + 2 Byte = 22 Byte.

../_images/2D_padding_example.svg — Matrix with [3, 5] elements, each element has a size of 4 bytes and 2 bytes of padding per row.

alpaka::Vec extents = alpaka::Vec{3u, 5u};
auto buffer = allocHostBufferWithPadding<int32_t>(extents);

REQUIRE(extents.x() == 5u);
REQUIRE(extents.y() == 3u);

// element size in byte (sizeof(int32_t))
REQUIRE(buffer.getPitches().x() == 4u);
// 5 elements with 4 bytes + 2 bytes padding
REQUIRE(buffer.getPitches().y() == 22u);

To manually calculate the address of a specific element in a Data Storage using a given memory pointer of element 0 and the Pitch, use the following code:

template<typename T>
T get_element(std::byte* dataPtr, alpaka::concepts::Vector auto idx, alpaka::concepts::Vector auto pitches)
{
    // (idx * pitches).sum() is a dot product
    return *(reinterpret_cast<T*>(dataPtr + (idx * pitches).sum()));
}

../_images/2D_padding_example_linearized.svg — The linearized memory layout of a matrix with [3, 5] elements, where each element has a size of 4 bytes and 2 bytes of padding per row.

The same applies to higher-dimensional memory. The Pitch for a dimension describes how many bytes must be added to skip a position in that dimension. The following example shows 3D memory and the corresponding values for the Extent and Pitch.

../_images/3D_padding_example.svg — Cube with [3, 3, 5] elements, each element has a size of 4 bytes and 2 bytes of padding per row and 22 bytes of padding per side.

alpaka::Vec extents = alpaka::Vec{3u, 3u, 5u};
auto buffer = allocHostBufferWithPadding<int32_t>(extents);

REQUIRE(extents.x() == 5u);
REQUIRE(extents.y() == 3u);
REQUIRE(extents.z() == 3u);

// element size in byte (sizeof(int32_t))
REQUIRE(buffer.getPitches().x() == 4u);
// 5 elements with 4 bytes + 2 bytes padding
REQUIRE(buffer.getPitches().y() == 22u);
// 3 rows with user data and padding, each 22 bytes long + 22 bytes padding for empty row
REQUIRE(buffer.getPitches().z() == 88u);