Kernel

After selecting a device, creating a queue, and allocating memory, the next step is to launch work on the device. In alpaka, kernels are implemented as function objects, known as functors. A functor is a C++ class or struct that contains at least the member function void operator() const, which implements the algorithm to be executed on the Device. A functor can be extended with additional functionality and is limited only by the requirement that it must be trivially copyable. In this tutorial, we will focus on a very simple Kernel.

The Kernel is implemented using the same principle as CUDA, HIP, or SYCL. The function specifies which part of a problem a single hardware thread processes based on its ID. To execute the algorithm, the Kernel is launched in parallel. Therefore, the same algorithm is executed n times, each time with a different thread ID.

A FrameSpec is used to describe the parallelism of a Kernel launch.

Writing the Kernel

An alpaka Kernel must meet the following requirements:

  • The kernel must be a function object. Therefore, it must implement the function call operator with the following signature: ALPAKA_FN_ACC void operator()(onAcc::concepts::Acc auto acc, ...) const or constexpr void operator()(onAcc::concepts::Acc auto acc, ...) const.

  • The functor must be trivially copyable.

  • Arguments of the functor must be trivially copyable and captured by value or const reference.

  • The first argument is the const reference to the accelerator handle acc, which follows the concepts onAcc::concepts::Acc.

  • The Device Memory handles are passed via the arguments [1]. Typically, output buffers use IMdSpan and input buffers use IDataSource.

struct VectorAddKernel
{
    ALPAKA_FN_ACC void operator()(
        onAcc::concepts::Acc auto const& acc,
        concepts::IMdSpan auto out,
        concepts::IDataSource auto const& lhs,
        concepts::IDataSource auto const& rhs) const
    {
        ALPAKA_ASSERT_ACC(out.getExtents() == lhs.getExtents());
        ALPAKA_ASSERT_ACC(out.getExtents() == rhs.getExtents());

        for(auto [i] : onAcc::makeIdxMap(acc, onAcc::worker::threadsInGrid, IdxRange{out.getExtents()}))
        {
            out[i] = lhs[i] + rhs[i];
        }
    }
};

alpaka offers many different ways to write a Kernel, but it also provides several functions to help you create a kernel easily and securely. onAcc::makeIdxMap is the most important helper function for writing a kernel. It distributes a range of indices among the workers. The second argument defines which worker group the executor threads belong to.

For the simplest kernel, we use the worker group onAcc::worker::threadsInGrid, which means that the index range is distributed across all global threads [2]. In the example, the range is the extents of out. We access all elements of out in parallel and write the sum of lhs and rhs to data position i. The parallelism is defined by the FrameSpec when the Kernel starts. Within the Kernel, we work with the parallelism defined by the ThreadSpec, which is derived from the FrameSpec and the used queue. The Launching the Kernel section explains how this derivation works.

The function call onAcc::makeIdxMap(acc, onAcc::worker::threadsInGrid, IdxRange{out.getExtents()}) provides us with certain guarantees:

  • All elements in out are processed by the worker group independently of the parallelism specified via FrameSpec [3].

  • This ensures that no out-of-range memory access happens.

  • Indies provided by onAcc::makeIdxMap for each worker group are optimized based on the used Device of the Queue and the Executor. They can be used for a one to one mapping to memory. Resulting in a contiguous, chunked access pattern on the CPU and a strided access pattern on GPU-like devices.

onAcc::makeIdxMap offers many more features needed for performance optimization. Once you’ve finished the tutorial, check out the chunked section to discover the full potential of this function.

Launching the Kernel

On the host side, we first need to create a FrameSpec. A FrameSpec describes the maximum possible parallelism. A FrameSpec contains the number of Frames and the Frame Extents. When the Kernel starts, a Thread Spec to execute the kernel is derived from the FrameSpec, the executor and queue properties. Therefore, the number of threads per thread block is derived from Frame Extents, and the number of thread blocks is derived from Frame count. The numbers from the Frame Extents can be directly transferred to the Thread Spec, but the number of blocks and/or threads may also be smaller than in the FrameSpec.

Let’s take, for example, a FrameSpec with the dimensions {10, 32} (number of Frames, Frame extents):

  • On a GPU, Thread Spec could take the value {10, 32} (blocks, threads), since we can launch 10 blocks and each GPU core has 32 threads.

  • On a CPU, the Thread Spec could be {10, 1} (blocks, threads), since we can run 10 blocks on 10 or fewer cores, but each core can have only one thread.

onAcc::makeIdxMap allows the use of any number of blocks and threads. Therefore, any FrameSpec will work. The size of the FrameSpec effects performance but not the correctness. To begin with, you should use the onHost::getFrameSpec function to create a FrameSpec. This function assumes that you are using the onAcc::makeIdxMap function in your Kernel and returns a well-functioning FrameSpec depending on the Device and Executor.

/* Let alpaka calculate a well-functioning `frameSpec` for you.
 * This assumes that you are using `onAcc::makeIdxMap` in the kernel.
 */
onHost::concepts::FrameSpec auto frameSpec = onHost::getFrameSpec(device, exec::anyExecutor, Vec{numElements});

/* Create a kernel object and enqueue it along with the `frameSpec` and kernel arguments.
 * Depending on how many tasks are still in the queue, the kernel may be executed immediately or after a delay.
 */
queue.enqueue(frameSpec, KernelBundle{VectorAddKernel{}, resultBuffer, lhsBuffer, rhsBuffer});
/* If you use a non-blocking queue, the kernel runs asynchronously with respect to the host.
 * To synchronize the kernel, you must call `onHost::wait(queue)`.
 */
onHost::wait(queue);

In the advanced area, section chunked, you’ll learn how onAcc::makeIdxMap works in detail. Once you understand this, you can manually set a FrameSpec that might work better than the FrameSpec returned by onHost::getFrameSpec.

Typical Beginner Mistakes

  • Forgetting to copy the inputs to the device before enqueueing the kernel.

  • Forgetting to copy the result back to the host after the kernel.

  • Forgetting to wait before reading host-side results from a non-blocking queue.

  • Choosing a one-dimensional frame for naturally multidimensional code and then reimplementing manual index arithmetic in the kernel.

  • Calculating your own thread ID and use it to access the data, rather than accessing the data via onAcc::makeIdxMap.

Complete Source File

050_kernel.cpp
 1/* Copyright 2026 René Widera
 2 * SPDX-License-Identifier: ISC
 3 */
 4
 5#include "docsTest.hpp"
 6
 7#include <alpaka/alpaka.hpp>
 8
 9#include <catch2/catch_template_test_macros.hpp>
10#include <catch2/catch_test_macros.hpp>
11
12#include <numeric>
13#include <vector>
14
15using namespace alpaka;
16
17struct VectorAddKernel
18{
19    ALPAKA_FN_ACC void operator()(
20        onAcc::concepts::Acc auto const& acc,
21        concepts::IMdSpan auto out,
22        concepts::IDataSource auto const& lhs,
23        concepts::IDataSource auto const& rhs) const
24    {
25        ALPAKA_ASSERT_ACC(out.getExtents() == lhs.getExtents());
26        ALPAKA_ASSERT_ACC(out.getExtents() == rhs.getExtents());
27
28        for(auto [i] : onAcc::makeIdxMap(acc, onAcc::worker::threadsInGrid, IdxRange{out.getExtents()}))
29        {
30            out[i] = lhs[i] + rhs[i];
31        }
32    }
33};
34
35
36TEMPLATE_LIST_TEST_CASE("tutorial kernel intro vector add", "[docs]", docs::test::TestBackends)
37{
38    auto selector = onHost::makeDeviceSelector(TestType::makeDict());
39    if(!selector.isAvailable())
40        return;
41    onHost::concepts::Device auto device = selector.makeDevice(0);
42    onHost::Queue queue = device.makeQueue();
43
44    /* When you use `makeIdxMap`, your algorithm is not subject to any restrictions regarding data size, such as the
45     * requirement that the data must be a power of two or a multiple of 100.
46     */
47    size_t numElements = 293u;
48    auto lhsBuffer = onHost::alloc<int>(device, numElements);
49    // Allocate buffers on the compute device.
50    auto rhsBuffer = onHost::allocLike(device, lhsBuffer);
51    auto resultBuffer = onHost::allocLike(device, lhsBuffer);
52
53    std::vector<int> lhs(numElements);
54    std::vector<int> rhs(numElements);
55    std::iota(lhs.begin(), lhs.end(), 0);
56    std::iota(rhs.begin(), rhs.end(), 1000);
57    std::vector<int> result(lhs.size(), -1);
58
59    // Copy input data to the device.
60    onHost::memcpy(queue, lhsBuffer, lhs);
61    onHost::memcpy(queue, rhsBuffer, rhs);
62    onHost::memset(queue, resultBuffer, 0x00);
63
64    /* Let alpaka calculate a well-functioning `frameSpec` for you.
65     * This assumes that you are using `onAcc::makeIdxMap` in the kernel.
66     */
67    onHost::concepts::FrameSpec auto frameSpec = onHost::getFrameSpec(device, exec::anyExecutor, Vec{numElements});
68
69    /* Create a kernel object and enqueue it along with the `frameSpec` and kernel arguments.
70     * Depending on how many tasks are still in the queue, the kernel may be executed immediately or after a delay.
71     */
72    queue.enqueue(frameSpec, KernelBundle{VectorAddKernel{}, resultBuffer, lhsBuffer, rhsBuffer});
73    /* If you use a non-blocking queue, the kernel runs asynchronously with respect to the host.
74     * To synchronize the kernel, you must call `onHost::wait(queue)`.
75     */
76    onHost::wait(queue);
77
78    // Copy the result back and wait for completion before reading it.
79    onHost::memcpy(queue, result, resultBuffer);
80    onHost::wait(queue);
81
82    for(size_t i = 0; i < result.size(); ++i)
83    {
84        CHECK(result[i] == lhs[i] + rhs[i]);
85    }
86}