Allocate Memory

Now that we know how to get a device and create a queue, we can move on to memory allocation. To allocate memory, you need a device and sometimes a queue. alpaka’s memory allocation methods return a DataStorage object that implements the alpaka::concepts::IBuffer interface. The alpaka::concepts::IBuffer interface requires that the lifetime of the memory be tracked. One implementation of this interface is the alpaka::onHost::SharedBuffer handle. It behaves like a std::shared_ptr<> from the standard library. When the last instance going out of scope, the memory is freed.

  • If a DataStorage handle allow it, copying a handle is a shallow copy of the handle and does not duplicate the data.

  • A deep copy of the memory must be explicitly triggered using alpaka::onHost::memcpy().

  • A buffer is not initialized with default values.

  • The extents, which describe the number of elements per dimension, should be >=1. The extents can have any dimensionality.

  • If the extent requires the alpaka::concepts::VectorOrScalar concept, it is permissible to use a scalar instead of an alpaka vector type to allocate a one-dimensional buffer.

  • The extent object also determines the internal index type used for addressing the buffer.

Choosing Extents

The extent rules above are easiest to see in code:

auto oneDimBuffer = onHost::allocHost<int>(size_t{16});

auto twoDimExtents = Vec<uint32_t, 2u>{2u, 3u};
auto twoDimBuffer = onHost::allocHost<int>(twoDimExtents);

static_assert(std::same_as<ALPAKA_TYPEOF(twoDimBuffer)::index_type, uint32_t>);

This snippet shows the three extent-related points the chapter relies on:

  • a one-dimensional allocation can use a scalar extent,

  • a multidimensional allocation uses an alpaka vector extent,

  • and the extent’s scalar type becomes the buffer’s internal index_type.

The following examples show how to create memory which is only accessible on the device.

concepts::Vector auto extents = Vec{2u, 3u};
// the allocation is providing a shared buffer which will be
// automatically freed if the last handle runs out of a life-time
concepts::IBuffer auto devBuffer = onHost::alloc<int>(device, extents);

There is a type of memory called mapped memory, which is located on the CPU but is also accessible on the device. Explicit memory copies are not required to access the memory from the device or host. When using mapped memory, you must be careful not to access the memory of the host and the device in parallel. Accessing this type of memory from the device is usually associated with high latency and is slow.

concepts::Vector auto extents = Vec{2u, 3u};
// allocate memory which lives on the host but is accessible from the compute device too
concepts::IBuffer auto devMappedBuffer = onHost::allocMapped<int>(device, extents);

Unified memory is similar to mapped memory in that it does not require explicit memory copies. Depending on the API used, it is located on the host or device. It is transparently migrated page by page to the location from which it is accessed. You should not access unified memory in parallel from the host and the device. The first access to a memory location is often associated with high latencies, but once the page has been migrated, access is just as fast as direct access to the device memory.

concepts::Vector auto extents = Vec{2u, 3u};
// allocate memory can be accessed from host and device (unified memory),
// the real location depends on the native backend e.g. CUDA, OneApi, ...
concepts::IBuffer auto devUnifiedBuffer = onHost::allocUnified<int>(device, extents);

Very often, the typical pattern for memory allocation is that you create a buffer for the host and need a second buffer for the device with the same value type and dimensions. For this, you can use alpaka::onHost::allocLike(device, sourceBuffer) to adopt all properties except the target device. The data in the source buffer is not copied. This can only be done explicitly.

concepts::Vector auto extents = Vec{2u, 3u};
// short notation to allocate memory on the host without a host device as first argument
concepts::IBuffer auto hostBuffer = onHost::allocHost<int>(extents);
// inherits value type and extents but does NOT copy the data
concepts::IBuffer auto devDoubleBuffer = onHost::allocLike(computeDevice, hostBuffer);

Sometimes you want to allocate memory that is only used as a temporary buffer and is no longer needed after your tasks are complete. Since memory allocations are costly, you generally avoid allocating memory, for example, in a loop. Depending on the device or queue API, alpaka::onHost::allocDeferred() automatically uses an internal caching allocator to keep allocation as cost-effective as possible.

That kind of temporary buffer shows up naturally later for things such as scan scratch storage, intermediate image tiles, or one stage of a multi-step numerical pipeline.

concepts::Vector auto extents = Vec{2u, 3u};
{
    // The allocation is deferred.
    // It is only allowed to access the memory after the queue processed the allocation task.
    concepts::IBuffer auto devDeferredBuffer = onHost::allocDeferred<int>(queue, extents);
    // Call the kernel with the buffer. This is only a dummy call not a real kernel call.
    callKernel(devDeferredBuffer);
    // At the end of the scope the buffer will be destroyed, but it could be that the kernel is not finished yet.
    // This special allocation method take care that the buffer is waiting for the queue before the memory is
    // freed.
}

Complete Source File

040_memory_allocation.cpp
 1/* Copyright 2025 René Widera
 2 * SPDX-License-Identifier: ISC
 3 */
 4
 5#include "docsTest.hpp"
 6
 7#include <alpaka/alpaka.hpp>
 8
 9#include <catch2/catch_template_test_macros.hpp>
10#include <catch2/catch_test_macros.hpp>
11
12#include <algorithm>
13#include <iostream>
14#include <type_traits>
15#include <vector>
16
17using namespace alpaka;
18
19TEST_CASE("memory extent forms", "[docs]")
20{
21    auto oneDimBuffer = onHost::allocHost<int>(size_t{16});
22
23    auto twoDimExtents = Vec<uint32_t, 2u>{2u, 3u};
24    auto twoDimBuffer = onHost::allocHost<int>(twoDimExtents);
25
26    static_assert(std::same_as<ALPAKA_TYPEOF(twoDimBuffer)::index_type, uint32_t>);
27
28    CHECK(oneDimBuffer.getExtents().x() == 16u);
29    CHECK(twoDimBuffer.getExtents() == twoDimExtents);
30}
31
32TEST_CASE("memory allocations", "[docs]")
33{
34    onHost::concepts::Device auto device = onHost::makeHostDevice();
35    {
36        concepts::Vector auto extents = Vec{2u, 3u};
37        // the allocation is providing a shared buffer which will be
38        // automatically freed if the last handle runs out of a life-time
39        concepts::IBuffer auto devBuffer = onHost::alloc<int>(device, extents);
40        alpaka::unused(devBuffer);
41    }
42    {
43        concepts::Vector auto extents = Vec{2u, 3u};
44        // allocate memory which lives on the host but is accessible from the compute device too
45        concepts::IBuffer auto devMappedBuffer = onHost::allocMapped<int>(device, extents);
46        alpaka::unused(devMappedBuffer);
47    }
48    {
49        concepts::Vector auto extents = Vec{2u, 3u};
50        // allocate memory can be accessed from host and device (unified memory),
51        // the real location depends on the native backend e.g. CUDA, OneApi, ...
52        concepts::IBuffer auto devUnifiedBuffer = onHost::allocUnified<int>(device, extents);
53        alpaka::unused(devUnifiedBuffer);
54    }
55}
56
57void callKernel([[maybe_unused]] auto dummyMemory)
58{
59}
60
61TEST_CASE("memory allocations deferred", "[docs]")
62{
63    onHost::concepts::Device auto device = onHost::makeHostDevice();
64    onHost::Queue queue = device.makeQueue();
65    concepts::Vector auto extents = Vec{2u, 3u};
66    {
67        // The allocation is deferred.
68        // It is only allowed to access the memory after the queue processed the allocation task.
69        concepts::IBuffer auto devDeferredBuffer = onHost::allocDeferred<int>(queue, extents);
70        // Call the kernel with the buffer. This is only a dummy call not a real kernel call.
71        callKernel(devDeferredBuffer);
72        // At the end of the scope the buffer will be destroyed, but it could be that the kernel is not finished yet.
73        // This special allocation method take care that the buffer is waiting for the queue before the memory is
74        // freed.
75    }
76}
77
78TEST_CASE("memory allocations like", "[docs]")
79{
80    onHost::concepts::Device auto computeDevice = onHost::makeHostDevice();
81    concepts::Vector auto extents = Vec{2u, 3u};
82    // short notation to allocate memory on the host without a host device as first argument
83    concepts::IBuffer auto hostBuffer = onHost::allocHost<int>(extents);
84    // inherits value type and extents but does NOT copy the data
85    concepts::IBuffer auto devDoubleBuffer = onHost::allocLike(computeDevice, hostBuffer);
86
87    alpaka::unused(devDoubleBuffer);
88}