System requirements

Validated hardware (MX1P)

Validated host platforms

Vendor Part Name CPU
Supermicro SYS-221H-TNR Intel Xeon6
Supermicro SYS-222HA-TN Intel Xeon6
Gigabyte G494-ZB0-AAP2 AMD EPYC 9005
Gigabyte G494-SB0-AAP2 Intel Xeon6
Dell R770 Intel Xeon6

Validated DDR5 RDIMM modules

Vendor Part Number Density Speed (MT/s)
Samsung M321R8GA0EB2-CCP 64GB 6400
Samsung M321R8GA0EB2-CCPKC 64GB 6400
Samsung M321RAJA0MB2-CCPWF 128GB 6400
SK hynix HMCG94AHBRA481N 64GB 6400
SK hynix HMCG94AGBRA181N-AA 64GB 5600
Micron MTC40F2046S1RC64BD2 64GB 6400
Micron MTC40F2047S1RC64BB1 128GB 6400

BIOS configuration

AMD platforms

  • SPM(Special Purpose Memory) configuration

    AMD CBS → CXL Common Options → CXL Memory Attribute : Enabled
    Warning
    If disabled, the CXL memory lacks EFI_MEMORY_SP attribute, causing the OS to boot it as standard system-ram(CPU-less NUMA node via SRAT).
    Once it is integrated into system memory, it cannot be reconfigured to DAX.

  • Use CPU address space

    AMD CBS → CXL Common Options → CXL Physical Addressing : System address
    Warning
    If not set to System address, a mismatch between SPA(System Physical Address) and HPA(Host Physical Address) occurs.
    It prevents the HDM(Host-managed Device Memory) Decoder from validating the memory range, making it impossible to create CXL Region.
    Without a valid region, the memory cannot be assigned to DAX.

  • Enable MSI multi vector

    AMD CBS → NBIO Common Options → IOMMU/Security → IOMMU : Enabled
    Warning
    If disabled, Interrupt Remapping is unavailable, limiting the CXL device to a single MSI vector.
    But our device is using multi vector MSI, so it should be enabled.

OS

OS Release status SDK tarball
Ubuntu 24.04 LTS Stable — officially supported xcena_sdk_<ver>_ubuntu_24.04.tar.xz
openSUSE Leap 16.0 Experimental xcena_sdk_<ver>_opensuse-leap_16.0.tar.xz
RHEL 9.6+ / Rocky Linux 9.6+ / AlmaLinux 9.6+ Experimental xcena_sdk_<ver>_el9.tar.xz

Experimental targets

openSUSE Leap and RHEL 9 support is preview-quality. The install and host-setup flow is verified on validated hardware — openSUSE Leap 16.0 and Rocky Linux 9.6 — but these targets are less battle-tested than Ubuntu; please report any issues you hit.

On Red Hat Enterprise Linux itself, enable CodeReady Builder and EPEL before installing; the SDK’s build dependencies (dkms, fmt-devel, gtest) live there, and unlike Rocky/AlmaLinux they are not reachable from the stock repositories:

sudo subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

CXL memory host setup

The SDK manages the device’s CXL memory through the ccma kernel module (CXL Contiguous Memory Allocator), which install.sh installs and configures automatically. The default setup needs no extra host configuration — install the SDK and reboot:

  • At boot the CXL memory onlines as system-ram (a CPU-less NUMA node). ccma is loaded before the dax kmem driver (via the boot config the installer writes) and reserves physically contiguous chunks the moment the node onlines — while the memory is still pristine and unfragmented.
  • By default ccma reserves 60% of each CXL node (sysram_reserve_pct=60) as contiguous chunks for device computing. The remaining 40% stays NUMA-allocatable, so applications can also place ordinary host allocations on the CXL node (e.g. numactl --membind) — the same device serves as compute memory and as a host NUMA memory node at once.
  • pxl_resourced detects ccma automatically and serves device memory allocations through it.

The installer persists this as boot configuration (/etc/modprobe.d/ccma.conf, /etc/modules-load.d/ccma.conf, and a udev rescan rule), so it stays correct across reboots even if dax device numbers change. The shipped module parameters are matched to the device — changing them (in particular chunk_size) can make the device memory unavailable, so leave them as installed.

devdax mode

Converting the CXL memory to devdax dedicates the whole region to device memory: no NUMA node is created for the host, and nearly the full region capacity becomes device compute memory instead of the 60% reserve above. Use it when:

  • Multi-host deployment — devdax mode is required: the multi-host coordinator accesses the shared region through /dev/daxN.N directly, and the shared region must not be managed by ccma (remove ccma from multi-host hosts with driver/ccma/uninstall.sh).
  • Maximum device-memory capacity matters more than using the CXL node as host NUMA memory.

For the CXL memory to be usable as devdax, the host needs three things: memhp_default_state=offline on the kernel cmdline, the distro’s memory auto-online udev rule neutralized, and a boot-time service that re-applies the devdax conversion (the kernel re-binds the memory to system-ram on every boot).

scripts/setup_host.sh (shipped with the SDK) does all three per-distro — the kernel cmdline (grubby on RHEL 9’s BLS, GRUB regeneration elsewhere), the udev override (Ubuntu 90-daxctl-device.rules, openSUSE 80-hotplug-cpu-mem.rules, RHEL 9 40-redhat.rules), and the boot-time xcena-devdax.service. Run it once and reboot:

sudo ./scripts/setup_host.sh
sudo reboot

install.sh does not run this by default; run the installer with ./install.sh --with-host-setup to include it. To go back to the default system-ram setup, undo all three changes together (remove memhp_default_state=offline from the cmdline, delete the udev override, disable xcena-devdax.service) and reboot — removing only the service while leaving the cmdline in place strands the memory offline, with no capacity in either mode.

Why devdax mode needs these

Once CXL memory is onlined as system-ram and the kernel starts using it as regular RAM, it can no longer be offlined or converted to devdax (daxctl reconfigure-device then fails with Device or resource busy). memhp_default_state=offline stops the kernel auto-onlining it, the udev override stops the distro force-onlining it anyway, and xcena-devdax.service performs the devdax conversion on each boot.

Note

xcena-devdax.service converts only dax devices under an XCENA CXL region (PCI vendor 0x20a6). If the XCENA memory instead surfaces as a soft-reserved hmem dax — a dax device not under /sys/bus/cxl/devices/region*, usually with the CXL region left disabled — the CXL region failed to form. Recheck the BIOS CXL Physical Addressing = System address / HDM decoder settings above.

Manual equivalent — what setup_host.sh does, for reference or advanced setups

1. Kernel cmdline — add memhp_default_state=offline (append to GRUB_CMDLINE_LINUX in /etc/default/grub, keeping existing parameters), then apply per bootloader and reboot:

sudo update-grub                                                     # Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg                          # openSUSE
sudo grubby --update-kernel=ALL --args=memhp_default_state=offline   # RHEL 9 (BLS)
sudo reboot

Verify after reboot: cat /proc/cmdline | grep memhp_default_state.

2. udev override — the distro rule that force-onlines hot-plugged memory (90-daxctl-device.rules / 80-hotplug-cpu-mem.rules / 40-redhat.rules) is shadowed under /etc/udev/rules.d/ with its online action disabled, so memhp_default_state=offline is not undone at boot.

3. boot-time conversion — xcena-devdax.service runs daxctl reconfigure-device --mode=devdax on the XCENA dax devices at each boot (the kernel re-binds them to system-ram otherwise).

Required system packages

Ubuntu (apt):

build-essential cmake ninja-build wget
python3-dev python3-pip python3-venv libncurses6
libglib2.0-0 libslirp0 liburing2 libcurl3-gnutls

openSUSE Leap 16 (zypper — daxctl is provided by the ndctl package):

gcc gcc-c++ make cmake ninja wget
python313-devel python313-pip libncurses6
libglib-2_0-0 libslirp0 liburing2 libcurl4
ndctl dkms kernel-default-devel

The full per-distro lists live in deps/*.list and are installed by install_dependencies.sh.

Kernel headers
The mx_dma DKMS build needs headers for the booted kernel (linux-headers-$(uname -r) / kernel-default-devel / kernel-devel-$(uname -r)). install_dependencies.sh installs them best-effort — they cannot be pinned in deps/*.list, which has no way to name the running kernel. When they are unavailable the install warns and skips the driver step instead of failing: see Install skips the driver. —

Required Python packages 

numpy tomli capstone cxxfilt pyelftools

Docker

  • Optional, recommended for consistent build environments Note
    For non-Ubuntu systems, it is recommended to use the official Docker image to ensure full compatibility.