Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
cxl: docs/devices - add cxl device and protocol reference
Add a simple device primer sufficient to understand the theory
of operation documentation.
Signed-off-by: Gregory Price <gourry@gourry.net>
Link: https://patch.msgid.link/20250512162134.3596150-4-gourry@gourry.net
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
authored by
Gregory Price
and committed by
Dave Jiang
750d662c
a3bf6b41
+171
+165
Documentation/driver-api/cxl/devices/device-types.rst
+165
Documentation/driver-api/cxl/devices/device-types.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
=====================
4
+
Devices and Protocols
5
+
=====================
6
+
7
+
The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section
8
+
covers some basic background on device types and on-device resources used by the platform and OS
9
+
which impact configuration.
10
+
11
+
Protocols
12
+
=========
13
+
14
+
There are three core protocols to CXL. For the purpose of this documentation,
15
+
we will only discuss very high level definitions as the specific hardware
16
+
details are largely abstracted away from Linux. See the CXL specification
17
+
for more details.
18
+
19
+
CXL.io
20
+
------
21
+
The basic interaction protocol, similar to PCIe configuration mechanisms.
22
+
Typically used for initialization, configuration, and I/O access for anything
23
+
other than memory (CXL.mem) or cache (CXL.cache) operations.
24
+
25
+
The Linux CXL driver exposes access to .io functionalty via the various sysfs
26
+
interfaces and /dev/cxl/ devices (which exposes direct access to device
27
+
mailboxes).
28
+
29
+
CXL.cache
30
+
---------
31
+
The mechanism by which a device may coherently access and cache host memory.
32
+
33
+
Largely transparent to Linux once configured.
34
+
35
+
CXL.mem
36
+
---------
37
+
The mechanism by which the CPU may coherently access and cache device memory.
38
+
39
+
Largely transparent to Linux once configured.
40
+
41
+
42
+
Device Types
43
+
============
44
+
45
+
Type-1
46
+
------
47
+
48
+
A Type-1 CXL device:
49
+
50
+
* Supports cxl.io and cxl.cache protocols
51
+
* Implements a fully coherent cache
52
+
* Allows Device-to-Host coherence and Host-to-Device snoops.
53
+
* Does NOT have host-managed device memory (HDM)
54
+
55
+
Typical examples of type-1 devices is a Smart NIC - which may want to
56
+
directly operate on host-memory (DMA) to store incoming packets. These
57
+
devices largely rely on CPU-attached memory.
58
+
59
+
Type-2
60
+
------
61
+
62
+
A Type-2 CXL Device:
63
+
64
+
* Supports cxl.io, cxl.cache, and cxl.mem protocols
65
+
* Optionally implements coherent cache and Host-Managed Device Memory
66
+
* Is typically an accelerator device w/ high bandwidth memory.
67
+
68
+
The primary difference between a type-1 and type-2 device is the presence
69
+
of host-managed device memory, which allows the device to operate on a
70
+
local memory bank - while the CPU sill has coherent DMA to the same memory.
71
+
72
+
The allows things like GPUs to expose their memory via DAX devices or file
73
+
descriptors, allows drivers and programs direct access to device memory
74
+
rather than use block-transfer semantics.
75
+
76
+
Type-3
77
+
------
78
+
79
+
A Type-3 CXL Device
80
+
81
+
* Supports cxl.io and cxl.mem
82
+
* Implements Host-Managed Device Memory
83
+
* May provide either Volatile or Persistent memory capacity (or both).
84
+
85
+
A basic example of a type-3 device is a simple memory expander, whose
86
+
local memory capacity is exposed to the CPU for access directly via
87
+
basic coherent DMA.
88
+
89
+
Switch
90
+
------
91
+
92
+
A CXL switch is a device capacity of routing any CXL (and by extension, PCIe)
93
+
protocol between an upstream, downstream, or peer devices. Many devices, such
94
+
as Multi-Logical Devices, imply the presence of switching in some manner.
95
+
96
+
Logical Devices and Heads
97
+
-------------------------
98
+
99
+
A CXL device may present one or more "Logical Devices" to one or more hosts
100
+
(via physical "Heads").
101
+
102
+
A Single-Logical Device (SLD) is a device which presents a single device to
103
+
one or more heads.
104
+
105
+
A Multi-Logical Device (MLD) is a device which may present multiple devices
106
+
to one or more devices.
107
+
108
+
A Single-Headed Device exposes only a single physical connection.
109
+
110
+
A Multi-Headed Device exposes multiple physical connections.
111
+
112
+
MHSLD
113
+
~~~~~
114
+
A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical
115
+
device to multiple heads which may be connected to one or more discrete
116
+
hosts. An example of this would be a simple memory-pool which may be
117
+
statically configured (prior to boot) to expose portions of its memory
118
+
to Linux via the CEDT ACPI table.
119
+
120
+
MHMLD
121
+
~~~~~
122
+
A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical
123
+
devices to multiple heads which may be connected to one or more discrete
124
+
hosts. An example of this would be a Dynamic Capacity Device or which
125
+
may be configured at runtime to expose portions of its memory to Linux.
126
+
127
+
Example Devices
128
+
===============
129
+
130
+
Memory Expander
131
+
---------------
132
+
The simplest form of Type-3 device is a memory expander. A memory expander
133
+
exposes Host-Managed Device Memory (HDM) to Linux. This memory may be
134
+
Volatile or Non-Volatile (Persistent).
135
+
136
+
Memory Expanders will typically be considered a form of Single-Headed,
137
+
Single-Logical Device - as its form factor will typically be an add-in-card
138
+
(AIC) or some other similar form-factor.
139
+
140
+
The Linux CXL driver provides support for static or dynamic configuration of
141
+
basic memory expanders. The platform may program decoders prior to OS init
142
+
(e.g. auto-decoders), or the user may program the fabric if the platform
143
+
defers these operations to the OS.
144
+
145
+
Multiple Memory Expanders may be added to an external chassis and exposed to
146
+
a host via a head attached to a CXL switch. This is a "memory pool", and
147
+
would be considered an MHSLD or MHMLD depending on the management capabilities
148
+
provided by the switch platform.
149
+
150
+
As of v6.14, Linux does not provide a formalized interface to manage non-DCD
151
+
MHSLD or MHMLD devices.
152
+
153
+
Dynamic Capacity Device (DCD)
154
+
-----------------------------
155
+
156
+
A Dynamic Capacity Device is a Type-3 device which provides dynamic management
157
+
of memory capacity. The basic premise of a DCD to provide an allocator-like
158
+
interface for physical memory capacity to a "Fabric Manager" (an external,
159
+
privileged host with privileges to change configurations for other hosts).
160
+
161
+
A DCD manages "Memory Extents", which may be volatile or persistent. Extents
162
+
may also be exclusive to a single host or shared across multiple hosts.
163
+
164
+
As of v6.14, Linux does not provide a formalized interface to manage DCD
165
+
devices, however there is active work on LKML targeting future release.