Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'cxl-for-5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl
Pull CXL (Compute Express Link) updates from Dan Williams:
"This subsystem is still in the build-out phase as the bulk of the
update is improvements to enumeration and fleshing out the device
model. In terms of new features, more mailbox commands have been added
to the allowed-list in support of persistent memory provisioning
support targeting v5.15.
The critical update from an enumeration perspective is support for the
CXL Fixed Memory Window Structure that indicates to Linux which system
physical address ranges decode to the CXL Host Bridges in the system.
This allows the driver to detect which address ranges have been mapped
by firmware and what address ranges are available for future hotplug.
So, again, mostly skeleton this round, with more meat targeting v5.15.
Summary:
- Add support for the CXL Fixed Memory Window Structure, a recent
extension of the ACPI CEDT (CXL Early Discovery Table)
- Add infrastructure for component registers
- Add HDM (Host-managed device memory) decoder definitions
- Define a device model for an HDM decoder tree
- Bridge CXL persistent memory capabilities to an NVDIMM bus /
device-model
- Switch to fine grained mapping of CXL MMIO registers to allow
different drivers / system software to own individual register
blocks
- Enable media provisioning commands, and publish the label storage
area size in sysfs
- Miscellaneous cleanups and fixes"
* tag 'cxl-for-5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl: (34 commits)
cxl/pci: Rename CXL REGLOC ID
cxl/acpi: Use the ACPI CFMWS to create static decoder objects
cxl/acpi: Add the Host Bridge base address to CXL port objects
cxl/pmem: Register 'pmem' / cxl_nvdimm devices
libnvdimm: Drop unused device power management support
libnvdimm: Export nvdimm shutdown helper, nvdimm_delete()
cxl/pmem: Add initial infrastructure for pmem support
cxl/core: Add cxl-bus driver infrastructure
cxl/pci: Add media provisioning required commands
cxl/component_regs: Fix offset
cxl/hdm: Fix decoder count calculation
cxl/acpi: Introduce cxl_decoder objects
cxl/acpi: Enumerate host bridge root ports
cxl/acpi: Add downstream port data to cxl_port instances
cxl/Kconfig: Default drivers to CONFIG_CXL_BUS
cxl/acpi: Introduce the root of a cxl_port topology
cxl/pci: Fixup devm_cxl_iomap_block() to take a 'struct device *'
cxl/pci: Add HDM decoder capabilities
cxl/pci: Reserve individual register block regions
cxl/pci: Map registers based on capabilities
...
+2599
-294
+103
Documentation/ABI/testing/sysfs-bus-cxl
+103
Documentation/ABI/testing/sysfs-bus-cxl
···
24
24
(RO) "Persistent Only Capacity" as bytes. Represents the
25
25
identically named field in the Identify Memory Device Output
26
26
Payload in the CXL-2.0 specification.
27
+
28
+
What: /sys/bus/cxl/devices/*/devtype
29
+
Date: June, 2021
30
+
KernelVersion: v5.14
31
+
Contact: linux-cxl@vger.kernel.org
32
+
Description:
33
+
CXL device objects export the devtype attribute which mirrors
34
+
the same value communicated in the DEVTYPE environment variable
35
+
for uevents for devices on the "cxl" bus.
36
+
37
+
What: /sys/bus/cxl/devices/portX/uport
38
+
Date: June, 2021
39
+
KernelVersion: v5.14
40
+
Contact: linux-cxl@vger.kernel.org
41
+
Description:
42
+
CXL port objects are enumerated from either a platform firmware
43
+
device (ACPI0017 and ACPI0016) or PCIe switch upstream port with
44
+
CXL component registers. The 'uport' symlink connects the CXL
45
+
portX object to the device that published the CXL port
46
+
capability.
47
+
48
+
What: /sys/bus/cxl/devices/portX/dportY
49
+
Date: June, 2021
50
+
KernelVersion: v5.14
51
+
Contact: linux-cxl@vger.kernel.org
52
+
Description:
53
+
CXL port objects are enumerated from either a platform firmware
54
+
device (ACPI0017 and ACPI0016) or PCIe switch upstream port with
55
+
CXL component registers. The 'dportY' symlink identifies one or
56
+
more downstream ports that the upstream port may target in its
57
+
decode of CXL memory resources. The 'Y' integer reflects the
58
+
hardware port unique-id used in the hardware decoder target
59
+
list.
60
+
61
+
What: /sys/bus/cxl/devices/decoderX.Y
62
+
Date: June, 2021
63
+
KernelVersion: v5.14
64
+
Contact: linux-cxl@vger.kernel.org
65
+
Description:
66
+
CXL decoder objects are enumerated from either a platform
67
+
firmware description, or a CXL HDM decoder register set in a
68
+
PCIe device (see CXL 2.0 section 8.2.5.12 CXL HDM Decoder
69
+
Capability Structure). The 'X' in decoderX.Y represents the
70
+
cxl_port container of this decoder, and 'Y' represents the
71
+
instance id of a given decoder resource.
72
+
73
+
What: /sys/bus/cxl/devices/decoderX.Y/{start,size}
74
+
Date: June, 2021
75
+
KernelVersion: v5.14
76
+
Contact: linux-cxl@vger.kernel.org
77
+
Description:
78
+
The 'start' and 'size' attributes together convey the physical
79
+
address base and number of bytes mapped in the decoder's decode
80
+
window. For decoders of devtype "cxl_decoder_root" the address
81
+
range is fixed. For decoders of devtype "cxl_decoder_switch" the
82
+
address is bounded by the decode range of the cxl_port ancestor
83
+
of the decoder's cxl_port, and dynamically updates based on the
84
+
active memory regions in that address space.
85
+
86
+
What: /sys/bus/cxl/devices/decoderX.Y/locked
87
+
Date: June, 2021
88
+
KernelVersion: v5.14
89
+
Contact: linux-cxl@vger.kernel.org
90
+
Description:
91
+
CXL HDM decoders have the capability to lock the configuration
92
+
until the next device reset. For decoders of devtype
93
+
"cxl_decoder_root" there is no standard facility to unlock them.
94
+
For decoders of devtype "cxl_decoder_switch" a secondary bus
95
+
reset, of the PCIe bridge that provides the bus for this
96
+
decoders uport, unlocks / resets the decoder.
97
+
98
+
What: /sys/bus/cxl/devices/decoderX.Y/target_list
99
+
Date: June, 2021
100
+
KernelVersion: v5.14
101
+
Contact: linux-cxl@vger.kernel.org
102
+
Description:
103
+
Display a comma separated list of the current decoder target
104
+
configuration. The list is ordered by the current configured
105
+
interleave order of the decoder's dport instances. Each entry in
106
+
the list is a dport id.
107
+
108
+
What: /sys/bus/cxl/devices/decoderX.Y/cap_{pmem,ram,type2,type3}
109
+
Date: June, 2021
110
+
KernelVersion: v5.14
111
+
Contact: linux-cxl@vger.kernel.org
112
+
Description:
113
+
When a CXL decoder is of devtype "cxl_decoder_root", it
114
+
represents a fixed memory window identified by platform
115
+
firmware. A fixed window may only support a subset of memory
116
+
types. The 'cap_*' attributes indicate whether persistent
117
+
memory, volatile memory, accelerator memory, and / or expander
118
+
memory may be mapped behind this decoder's memory window.
119
+
120
+
What: /sys/bus/cxl/devices/decoderX.Y/target_type
121
+
Date: June, 2021
122
+
KernelVersion: v5.14
123
+
Contact: linux-cxl@vger.kernel.org
124
+
Description:
125
+
When a CXL decoder is of devtype "cxl_decoder_switch", it can
126
+
optionally decode either accelerator memory (type-2) or expander
127
+
memory (type-3). The 'target_type' attribute indicates the
128
+
current setting which may dynamically change based on what
129
+
memory regions are activated in this decode hierarchy.
+13
-7
Documentation/driver-api/cxl/memory-devices.rst
+13
-7
Documentation/driver-api/cxl/memory-devices.rst
···
22
22
CXL Memory Device
23
23
-----------------
24
24
25
-
.. kernel-doc:: drivers/cxl/mem.c
26
-
:doc: cxl mem
25
+
.. kernel-doc:: drivers/cxl/pci.c
26
+
:doc: cxl pci
27
27
28
-
.. kernel-doc:: drivers/cxl/mem.c
28
+
.. kernel-doc:: drivers/cxl/pci.c
29
29
:internal:
30
30
31
-
CXL Bus
32
-
-------
33
-
.. kernel-doc:: drivers/cxl/bus.c
34
-
:doc: cxl bus
31
+
CXL Core
32
+
--------
33
+
.. kernel-doc:: drivers/cxl/cxl.h
34
+
:doc: cxl objects
35
+
36
+
.. kernel-doc:: drivers/cxl/cxl.h
37
+
:internal:
38
+
39
+
.. kernel-doc:: drivers/cxl/core.c
40
+
:doc: cxl core
35
41
36
42
External Interfaces
37
43
===================
+34
-9
drivers/cxl/Kconfig
+34
-9
drivers/cxl/Kconfig
···
15
15
16
16
config CXL_MEM
17
17
tristate "CXL.mem: Memory Devices"
18
+
default CXL_BUS
18
19
help
19
20
The CXL.mem protocol allows a device to act as a provider of
20
21
"System RAM" and/or "Persistent Memory" that is fully coherent
21
22
as if the memory was attached to the typical CPU memory
22
23
controller.
23
24
24
-
Say 'y/m' to enable a driver (named "cxl_mem.ko" when built as
25
-
a module) that will attach to CXL.mem devices for
26
-
configuration, provisioning, and health monitoring. This
27
-
driver is required for dynamic provisioning of CXL.mem
28
-
attached memory which is a prerequisite for persistent memory
29
-
support. Typically volatile memory is mapped by platform
30
-
firmware and included in the platform memory map, but in some
31
-
cases the OS is responsible for mapping that memory. See
32
-
Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
25
+
Say 'y/m' to enable a driver that will attach to CXL.mem devices for
26
+
configuration and management primarily via the mailbox interface. See
27
+
Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification for more
28
+
details.
33
29
34
30
If unsure say 'm'.
35
31
···
46
50
potential impact to memory currently in use by the kernel.
47
51
48
52
If developing CXL hardware or the driver say Y, otherwise say N.
53
+
54
+
config CXL_ACPI
55
+
tristate "CXL ACPI: Platform Support"
56
+
depends on ACPI
57
+
default CXL_BUS
58
+
help
59
+
Enable support for host managed device memory (HDM) resources
60
+
published by a platform's ACPI CXL memory layout description. See
61
+
Chapter 9.14.1 CXL Early Discovery Table (CEDT) in the CXL 2.0
62
+
specification, and CXL Fixed Memory Window Structures (CEDT.CFMWS)
63
+
(https://www.computeexpresslink.org/spec-landing). The CXL core
64
+
consumes these resource to publish the root of a cxl_port decode
65
+
hierarchy to map regions that represent System RAM, or Persistent
66
+
Memory regions to be managed by LIBNVDIMM.
67
+
68
+
If unsure say 'm'.
69
+
70
+
config CXL_PMEM
71
+
tristate "CXL PMEM: Persistent Memory Support"
72
+
depends on LIBNVDIMM
73
+
default CXL_BUS
74
+
help
75
+
In addition to typical memory resources a platform may also advertise
76
+
support for persistent memory attached via CXL. This support is
77
+
managed via a bridge driver from CXL to the LIBNVDIMM system
78
+
subsystem. Say 'y/m' to enable support for enumerating and
79
+
provisioning the persistent memory capacity of CXL memory expanders.
80
+
81
+
If unsure say 'm'.
49
82
endif
+8
-4
drivers/cxl/Makefile
+8
-4
drivers/cxl/Makefile
···
1
1
# SPDX-License-Identifier: GPL-2.0
2
-
obj-$(CONFIG_CXL_BUS) += cxl_bus.o
3
-
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
2
+
obj-$(CONFIG_CXL_BUS) += cxl_core.o
3
+
obj-$(CONFIG_CXL_MEM) += cxl_pci.o
4
+
obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
5
+
obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
4
6
5
7
ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
6
-
cxl_bus-y := bus.o
7
-
cxl_mem-y := mem.o
8
+
cxl_core-y := core.o
9
+
cxl_pci-y := pci.o
10
+
cxl_acpi-y := acpi.o
11
+
cxl_pmem-y := pmem.o
+434
drivers/cxl/acpi.c
+434
drivers/cxl/acpi.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
3
+
#include <linux/platform_device.h>
4
+
#include <linux/module.h>
5
+
#include <linux/device.h>
6
+
#include <linux/kernel.h>
7
+
#include <linux/acpi.h>
8
+
#include <linux/pci.h>
9
+
#include "cxl.h"
10
+
11
+
static struct acpi_table_header *acpi_cedt;
12
+
13
+
/* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */
14
+
#define CFMWS_INTERLEAVE_WAYS(x) (1 << (x)->interleave_ways)
15
+
#define CFMWS_INTERLEAVE_GRANULARITY(x) ((x)->granularity + 8)
16
+
17
+
static unsigned long cfmws_to_decoder_flags(int restrictions)
18
+
{
19
+
unsigned long flags = 0;
20
+
21
+
if (restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE2)
22
+
flags |= CXL_DECODER_F_TYPE2;
23
+
if (restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE3)
24
+
flags |= CXL_DECODER_F_TYPE3;
25
+
if (restrictions & ACPI_CEDT_CFMWS_RESTRICT_VOLATILE)
26
+
flags |= CXL_DECODER_F_RAM;
27
+
if (restrictions & ACPI_CEDT_CFMWS_RESTRICT_PMEM)
28
+
flags |= CXL_DECODER_F_PMEM;
29
+
if (restrictions & ACPI_CEDT_CFMWS_RESTRICT_FIXED)
30
+
flags |= CXL_DECODER_F_LOCK;
31
+
32
+
return flags;
33
+
}
34
+
35
+
static int cxl_acpi_cfmws_verify(struct device *dev,
36
+
struct acpi_cedt_cfmws *cfmws)
37
+
{
38
+
int expected_len;
39
+
40
+
if (cfmws->interleave_arithmetic != ACPI_CEDT_CFMWS_ARITHMETIC_MODULO) {
41
+
dev_err(dev, "CFMWS Unsupported Interleave Arithmetic\n");
42
+
return -EINVAL;
43
+
}
44
+
45
+
if (!IS_ALIGNED(cfmws->base_hpa, SZ_256M)) {
46
+
dev_err(dev, "CFMWS Base HPA not 256MB aligned\n");
47
+
return -EINVAL;
48
+
}
49
+
50
+
if (!IS_ALIGNED(cfmws->window_size, SZ_256M)) {
51
+
dev_err(dev, "CFMWS Window Size not 256MB aligned\n");
52
+
return -EINVAL;
53
+
}
54
+
55
+
expected_len = struct_size((cfmws), interleave_targets,
56
+
CFMWS_INTERLEAVE_WAYS(cfmws));
57
+
58
+
if (cfmws->header.length < expected_len) {
59
+
dev_err(dev, "CFMWS length %d less than expected %d\n",
60
+
cfmws->header.length, expected_len);
61
+
return -EINVAL;
62
+
}
63
+
64
+
if (cfmws->header.length > expected_len)
65
+
dev_dbg(dev, "CFMWS length %d greater than expected %d\n",
66
+
cfmws->header.length, expected_len);
67
+
68
+
return 0;
69
+
}
70
+
71
+
static void cxl_add_cfmws_decoders(struct device *dev,
72
+
struct cxl_port *root_port)
73
+
{
74
+
struct acpi_cedt_cfmws *cfmws;
75
+
struct cxl_decoder *cxld;
76
+
acpi_size len, cur = 0;
77
+
void *cedt_subtable;
78
+
unsigned long flags;
79
+
int rc;
80
+
81
+
len = acpi_cedt->length - sizeof(*acpi_cedt);
82
+
cedt_subtable = acpi_cedt + 1;
83
+
84
+
while (cur < len) {
85
+
struct acpi_cedt_header *c = cedt_subtable + cur;
86
+
87
+
if (c->type != ACPI_CEDT_TYPE_CFMWS) {
88
+
cur += c->length;
89
+
continue;
90
+
}
91
+
92
+
cfmws = cedt_subtable + cur;
93
+
94
+
if (cfmws->header.length < sizeof(*cfmws)) {
95
+
dev_warn_once(dev,
96
+
"CFMWS entry skipped:invalid length:%u\n",
97
+
cfmws->header.length);
98
+
cur += c->length;
99
+
continue;
100
+
}
101
+
102
+
rc = cxl_acpi_cfmws_verify(dev, cfmws);
103
+
if (rc) {
104
+
dev_err(dev, "CFMWS range %#llx-%#llx not registered\n",
105
+
cfmws->base_hpa, cfmws->base_hpa +
106
+
cfmws->window_size - 1);
107
+
cur += c->length;
108
+
continue;
109
+
}
110
+
111
+
flags = cfmws_to_decoder_flags(cfmws->restrictions);
112
+
cxld = devm_cxl_add_decoder(dev, root_port,
113
+
CFMWS_INTERLEAVE_WAYS(cfmws),
114
+
cfmws->base_hpa, cfmws->window_size,
115
+
CFMWS_INTERLEAVE_WAYS(cfmws),
116
+
CFMWS_INTERLEAVE_GRANULARITY(cfmws),
117
+
CXL_DECODER_EXPANDER,
118
+
flags);
119
+
120
+
if (IS_ERR(cxld)) {
121
+
dev_err(dev, "Failed to add decoder for %#llx-%#llx\n",
122
+
cfmws->base_hpa, cfmws->base_hpa +
123
+
cfmws->window_size - 1);
124
+
} else {
125
+
dev_dbg(dev, "add: %s range %#llx-%#llx\n",
126
+
dev_name(&cxld->dev), cfmws->base_hpa,
127
+
cfmws->base_hpa + cfmws->window_size - 1);
128
+
}
129
+
cur += c->length;
130
+
}
131
+
}
132
+
133
+
static struct acpi_cedt_chbs *cxl_acpi_match_chbs(struct device *dev, u32 uid)
134
+
{
135
+
struct acpi_cedt_chbs *chbs, *chbs_match = NULL;
136
+
acpi_size len, cur = 0;
137
+
void *cedt_subtable;
138
+
139
+
len = acpi_cedt->length - sizeof(*acpi_cedt);
140
+
cedt_subtable = acpi_cedt + 1;
141
+
142
+
while (cur < len) {
143
+
struct acpi_cedt_header *c = cedt_subtable + cur;
144
+
145
+
if (c->type != ACPI_CEDT_TYPE_CHBS) {
146
+
cur += c->length;
147
+
continue;
148
+
}
149
+
150
+
chbs = cedt_subtable + cur;
151
+
152
+
if (chbs->header.length < sizeof(*chbs)) {
153
+
dev_warn_once(dev,
154
+
"CHBS entry skipped: invalid length:%u\n",
155
+
chbs->header.length);
156
+
cur += c->length;
157
+
continue;
158
+
}
159
+
160
+
if (chbs->uid != uid) {
161
+
cur += c->length;
162
+
continue;
163
+
}
164
+
165
+
if (chbs_match) {
166
+
dev_warn_once(dev,
167
+
"CHBS entry skipped: duplicate UID:%u\n",
168
+
uid);
169
+
cur += c->length;
170
+
continue;
171
+
}
172
+
173
+
chbs_match = chbs;
174
+
cur += c->length;
175
+
}
176
+
177
+
return chbs_match ? chbs_match : ERR_PTR(-ENODEV);
178
+
}
179
+
180
+
static resource_size_t get_chbcr(struct acpi_cedt_chbs *chbs)
181
+
{
182
+
return IS_ERR(chbs) ? CXL_RESOURCE_NONE : chbs->base;
183
+
}
184
+
185
+
struct cxl_walk_context {
186
+
struct device *dev;
187
+
struct pci_bus *root;
188
+
struct cxl_port *port;
189
+
int error;
190
+
int count;
191
+
};
192
+
193
+
static int match_add_root_ports(struct pci_dev *pdev, void *data)
194
+
{
195
+
struct cxl_walk_context *ctx = data;
196
+
struct pci_bus *root_bus = ctx->root;
197
+
struct cxl_port *port = ctx->port;
198
+
int type = pci_pcie_type(pdev);
199
+
struct device *dev = ctx->dev;
200
+
u32 lnkcap, port_num;
201
+
int rc;
202
+
203
+
if (pdev->bus != root_bus)
204
+
return 0;
205
+
if (!pci_is_pcie(pdev))
206
+
return 0;
207
+
if (type != PCI_EXP_TYPE_ROOT_PORT)
208
+
return 0;
209
+
if (pci_read_config_dword(pdev, pci_pcie_cap(pdev) + PCI_EXP_LNKCAP,
210
+
&lnkcap) != PCIBIOS_SUCCESSFUL)
211
+
return 0;
212
+
213
+
/* TODO walk DVSEC to find component register base */
214
+
port_num = FIELD_GET(PCI_EXP_LNKCAP_PN, lnkcap);
215
+
rc = cxl_add_dport(port, &pdev->dev, port_num, CXL_RESOURCE_NONE);
216
+
if (rc) {
217
+
ctx->error = rc;
218
+
return rc;
219
+
}
220
+
ctx->count++;
221
+
222
+
dev_dbg(dev, "add dport%d: %s\n", port_num, dev_name(&pdev->dev));
223
+
224
+
return 0;
225
+
}
226
+
227
+
static struct cxl_dport *find_dport_by_dev(struct cxl_port *port, struct device *dev)
228
+
{
229
+
struct cxl_dport *dport;
230
+
231
+
device_lock(&port->dev);
232
+
list_for_each_entry(dport, &port->dports, list)
233
+
if (dport->dport == dev) {
234
+
device_unlock(&port->dev);
235
+
return dport;
236
+
}
237
+
238
+
device_unlock(&port->dev);
239
+
return NULL;
240
+
}
241
+
242
+
static struct acpi_device *to_cxl_host_bridge(struct device *dev)
243
+
{
244
+
struct acpi_device *adev = to_acpi_device(dev);
245
+
246
+
if (strcmp(acpi_device_hid(adev), "ACPI0016") == 0)
247
+
return adev;
248
+
return NULL;
249
+
}
250
+
251
+
/*
252
+
* A host bridge is a dport to a CFMWS decode and it is a uport to the
253
+
* dport (PCIe Root Ports) in the host bridge.
254
+
*/
255
+
static int add_host_bridge_uport(struct device *match, void *arg)
256
+
{
257
+
struct acpi_device *bridge = to_cxl_host_bridge(match);
258
+
struct cxl_port *root_port = arg;
259
+
struct device *host = root_port->dev.parent;
260
+
struct acpi_pci_root *pci_root;
261
+
struct cxl_walk_context ctx;
262
+
struct cxl_decoder *cxld;
263
+
struct cxl_dport *dport;
264
+
struct cxl_port *port;
265
+
266
+
if (!bridge)
267
+
return 0;
268
+
269
+
pci_root = acpi_pci_find_root(bridge->handle);
270
+
if (!pci_root)
271
+
return -ENXIO;
272
+
273
+
dport = find_dport_by_dev(root_port, match);
274
+
if (!dport) {
275
+
dev_dbg(host, "host bridge expected and not found\n");
276
+
return -ENODEV;
277
+
}
278
+
279
+
port = devm_cxl_add_port(host, match, dport->component_reg_phys,
280
+
root_port);
281
+
if (IS_ERR(port))
282
+
return PTR_ERR(port);
283
+
dev_dbg(host, "%s: add: %s\n", dev_name(match), dev_name(&port->dev));
284
+
285
+
ctx = (struct cxl_walk_context){
286
+
.dev = host,
287
+
.root = pci_root->bus,
288
+
.port = port,
289
+
};
290
+
pci_walk_bus(pci_root->bus, match_add_root_ports, &ctx);
291
+
292
+
if (ctx.count == 0)
293
+
return -ENODEV;
294
+
if (ctx.error)
295
+
return ctx.error;
296
+
297
+
/* TODO: Scan CHBCR for HDM Decoder resources */
298
+
299
+
/*
300
+
* In the single-port host-bridge case there are no HDM decoders
301
+
* in the CHBCR and a 1:1 passthrough decode is implied.
302
+
*/
303
+
if (ctx.count == 1) {
304
+
cxld = devm_cxl_add_passthrough_decoder(host, port);
305
+
if (IS_ERR(cxld))
306
+
return PTR_ERR(cxld);
307
+
308
+
dev_dbg(host, "add: %s\n", dev_name(&cxld->dev));
309
+
}
310
+
311
+
return 0;
312
+
}
313
+
314
+
static int add_host_bridge_dport(struct device *match, void *arg)
315
+
{
316
+
int rc;
317
+
acpi_status status;
318
+
unsigned long long uid;
319
+
struct acpi_cedt_chbs *chbs;
320
+
struct cxl_port *root_port = arg;
321
+
struct device *host = root_port->dev.parent;
322
+
struct acpi_device *bridge = to_cxl_host_bridge(match);
323
+
324
+
if (!bridge)
325
+
return 0;
326
+
327
+
status = acpi_evaluate_integer(bridge->handle, METHOD_NAME__UID, NULL,
328
+
&uid);
329
+
if (status != AE_OK) {
330
+
dev_err(host, "unable to retrieve _UID of %s\n",
331
+
dev_name(match));
332
+
return -ENODEV;
333
+
}
334
+
335
+
chbs = cxl_acpi_match_chbs(host, uid);
336
+
if (IS_ERR(chbs))
337
+
dev_dbg(host, "No CHBS found for Host Bridge: %s\n",
338
+
dev_name(match));
339
+
340
+
rc = cxl_add_dport(root_port, match, uid, get_chbcr(chbs));
341
+
if (rc) {
342
+
dev_err(host, "failed to add downstream port: %s\n",
343
+
dev_name(match));
344
+
return rc;
345
+
}
346
+
dev_dbg(host, "add dport%llu: %s\n", uid, dev_name(match));
347
+
return 0;
348
+
}
349
+
350
+
static int add_root_nvdimm_bridge(struct device *match, void *data)
351
+
{
352
+
struct cxl_decoder *cxld;
353
+
struct cxl_port *root_port = data;
354
+
struct cxl_nvdimm_bridge *cxl_nvb;
355
+
struct device *host = root_port->dev.parent;
356
+
357
+
if (!is_root_decoder(match))
358
+
return 0;
359
+
360
+
cxld = to_cxl_decoder(match);
361
+
if (!(cxld->flags & CXL_DECODER_F_PMEM))
362
+
return 0;
363
+
364
+
cxl_nvb = devm_cxl_add_nvdimm_bridge(host, root_port);
365
+
if (IS_ERR(cxl_nvb)) {
366
+
dev_dbg(host, "failed to register pmem\n");
367
+
return PTR_ERR(cxl_nvb);
368
+
}
369
+
dev_dbg(host, "%s: add: %s\n", dev_name(&root_port->dev),
370
+
dev_name(&cxl_nvb->dev));
371
+
return 1;
372
+
}
373
+
374
+
static int cxl_acpi_probe(struct platform_device *pdev)
375
+
{
376
+
int rc;
377
+
acpi_status status;
378
+
struct cxl_port *root_port;
379
+
struct device *host = &pdev->dev;
380
+
struct acpi_device *adev = ACPI_COMPANION(host);
381
+
382
+
root_port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL);
383
+
if (IS_ERR(root_port))
384
+
return PTR_ERR(root_port);
385
+
dev_dbg(host, "add: %s\n", dev_name(&root_port->dev));
386
+
387
+
status = acpi_get_table(ACPI_SIG_CEDT, 0, &acpi_cedt);
388
+
if (ACPI_FAILURE(status))
389
+
return -ENXIO;
390
+
391
+
rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
392
+
add_host_bridge_dport);
393
+
if (rc)
394
+
goto out;
395
+
396
+
cxl_add_cfmws_decoders(host, root_port);
397
+
398
+
/*
399
+
* Root level scanned with host-bridge as dports, now scan host-bridges
400
+
* for their role as CXL uports to their CXL-capable PCIe Root Ports.
401
+
*/
402
+
rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
403
+
add_host_bridge_uport);
404
+
if (rc)
405
+
goto out;
406
+
407
+
if (IS_ENABLED(CONFIG_CXL_PMEM))
408
+
rc = device_for_each_child(&root_port->dev, root_port,
409
+
add_root_nvdimm_bridge);
410
+
411
+
out:
412
+
acpi_put_table(acpi_cedt);
413
+
if (rc < 0)
414
+
return rc;
415
+
return 0;
416
+
}
417
+
418
+
static const struct acpi_device_id cxl_acpi_ids[] = {
419
+
{ "ACPI0017", 0 },
420
+
{ "", 0 },
421
+
};
422
+
MODULE_DEVICE_TABLE(acpi, cxl_acpi_ids);
423
+
424
+
static struct platform_driver cxl_acpi_driver = {
425
+
.probe = cxl_acpi_probe,
426
+
.driver = {
427
+
.name = KBUILD_MODNAME,
428
+
.acpi_match_table = cxl_acpi_ids,
429
+
},
430
+
};
431
+
432
+
module_platform_driver(cxl_acpi_driver);
433
+
MODULE_LICENSE("GPL v2");
434
+
MODULE_IMPORT_NS(CXL);
-29
drivers/cxl/bus.c
-29
drivers/cxl/bus.c
···
1
-
// SPDX-License-Identifier: GPL-2.0-only
2
-
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
3
-
#include <linux/device.h>
4
-
#include <linux/module.h>
5
-
6
-
/**
7
-
* DOC: cxl bus
8
-
*
9
-
* The CXL bus provides namespace for control devices and a rendezvous
10
-
* point for cross-device interleave coordination.
11
-
*/
12
-
struct bus_type cxl_bus_type = {
13
-
.name = "cxl",
14
-
};
15
-
EXPORT_SYMBOL_GPL(cxl_bus_type);
16
-
17
-
static __init int cxl_bus_init(void)
18
-
{
19
-
return bus_register(&cxl_bus_type);
20
-
}
21
-
22
-
static void cxl_bus_exit(void)
23
-
{
24
-
bus_unregister(&cxl_bus_type);
25
-
}
26
-
27
-
module_init(cxl_bus_init);
28
-
module_exit(cxl_bus_exit);
29
-
MODULE_LICENSE("GPL v2");
+1067
drivers/cxl/core.c
+1067
drivers/cxl/core.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
3
+
#include <linux/io-64-nonatomic-lo-hi.h>
4
+
#include <linux/device.h>
5
+
#include <linux/module.h>
6
+
#include <linux/pci.h>
7
+
#include <linux/slab.h>
8
+
#include <linux/idr.h>
9
+
#include "cxl.h"
10
+
#include "mem.h"
11
+
12
+
/**
13
+
* DOC: cxl core
14
+
*
15
+
* The CXL core provides a sysfs hierarchy for control devices and a rendezvous
16
+
* point for cross-device interleave coordination through cxl ports.
17
+
*/
18
+
19
+
static DEFINE_IDA(cxl_port_ida);
20
+
21
+
static ssize_t devtype_show(struct device *dev, struct device_attribute *attr,
22
+
char *buf)
23
+
{
24
+
return sysfs_emit(buf, "%s\n", dev->type->name);
25
+
}
26
+
static DEVICE_ATTR_RO(devtype);
27
+
28
+
static struct attribute *cxl_base_attributes[] = {
29
+
&dev_attr_devtype.attr,
30
+
NULL,
31
+
};
32
+
33
+
static struct attribute_group cxl_base_attribute_group = {
34
+
.attrs = cxl_base_attributes,
35
+
};
36
+
37
+
static ssize_t start_show(struct device *dev, struct device_attribute *attr,
38
+
char *buf)
39
+
{
40
+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
41
+
42
+
return sysfs_emit(buf, "%#llx\n", cxld->range.start);
43
+
}
44
+
static DEVICE_ATTR_RO(start);
45
+
46
+
static ssize_t size_show(struct device *dev, struct device_attribute *attr,
47
+
char *buf)
48
+
{
49
+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
50
+
51
+
return sysfs_emit(buf, "%#llx\n", range_len(&cxld->range));
52
+
}
53
+
static DEVICE_ATTR_RO(size);
54
+
55
+
#define CXL_DECODER_FLAG_ATTR(name, flag) \
56
+
static ssize_t name##_show(struct device *dev, \
57
+
struct device_attribute *attr, char *buf) \
58
+
{ \
59
+
struct cxl_decoder *cxld = to_cxl_decoder(dev); \
60
+
\
61
+
return sysfs_emit(buf, "%s\n", \
62
+
(cxld->flags & (flag)) ? "1" : "0"); \
63
+
} \
64
+
static DEVICE_ATTR_RO(name)
65
+
66
+
CXL_DECODER_FLAG_ATTR(cap_pmem, CXL_DECODER_F_PMEM);
67
+
CXL_DECODER_FLAG_ATTR(cap_ram, CXL_DECODER_F_RAM);
68
+
CXL_DECODER_FLAG_ATTR(cap_type2, CXL_DECODER_F_TYPE2);
69
+
CXL_DECODER_FLAG_ATTR(cap_type3, CXL_DECODER_F_TYPE3);
70
+
CXL_DECODER_FLAG_ATTR(locked, CXL_DECODER_F_LOCK);
71
+
72
+
static ssize_t target_type_show(struct device *dev,
73
+
struct device_attribute *attr, char *buf)
74
+
{
75
+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
76
+
77
+
switch (cxld->target_type) {
78
+
case CXL_DECODER_ACCELERATOR:
79
+
return sysfs_emit(buf, "accelerator\n");
80
+
case CXL_DECODER_EXPANDER:
81
+
return sysfs_emit(buf, "expander\n");
82
+
}
83
+
return -ENXIO;
84
+
}
85
+
static DEVICE_ATTR_RO(target_type);
86
+
87
+
static ssize_t target_list_show(struct device *dev,
88
+
struct device_attribute *attr, char *buf)
89
+
{
90
+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
91
+
ssize_t offset = 0;
92
+
int i, rc = 0;
93
+
94
+
device_lock(dev);
95
+
for (i = 0; i < cxld->interleave_ways; i++) {
96
+
struct cxl_dport *dport = cxld->target[i];
97
+
struct cxl_dport *next = NULL;
98
+
99
+
if (!dport)
100
+
break;
101
+
102
+
if (i + 1 < cxld->interleave_ways)
103
+
next = cxld->target[i + 1];
104
+
rc = sysfs_emit_at(buf, offset, "%d%s", dport->port_id,
105
+
next ? "," : "");
106
+
if (rc < 0)
107
+
break;
108
+
offset += rc;
109
+
}
110
+
device_unlock(dev);
111
+
112
+
if (rc < 0)
113
+
return rc;
114
+
115
+
rc = sysfs_emit_at(buf, offset, "\n");
116
+
if (rc < 0)
117
+
return rc;
118
+
119
+
return offset + rc;
120
+
}
121
+
static DEVICE_ATTR_RO(target_list);
122
+
123
+
static struct attribute *cxl_decoder_base_attrs[] = {
124
+
&dev_attr_start.attr,
125
+
&dev_attr_size.attr,
126
+
&dev_attr_locked.attr,
127
+
&dev_attr_target_list.attr,
128
+
NULL,
129
+
};
130
+
131
+
static struct attribute_group cxl_decoder_base_attribute_group = {
132
+
.attrs = cxl_decoder_base_attrs,
133
+
};
134
+
135
+
static struct attribute *cxl_decoder_root_attrs[] = {
136
+
&dev_attr_cap_pmem.attr,
137
+
&dev_attr_cap_ram.attr,
138
+
&dev_attr_cap_type2.attr,
139
+
&dev_attr_cap_type3.attr,
140
+
NULL,
141
+
};
142
+
143
+
static struct attribute_group cxl_decoder_root_attribute_group = {
144
+
.attrs = cxl_decoder_root_attrs,
145
+
};
146
+
147
+
static const struct attribute_group *cxl_decoder_root_attribute_groups[] = {
148
+
&cxl_decoder_root_attribute_group,
149
+
&cxl_decoder_base_attribute_group,
150
+
&cxl_base_attribute_group,
151
+
NULL,
152
+
};
153
+
154
+
static struct attribute *cxl_decoder_switch_attrs[] = {
155
+
&dev_attr_target_type.attr,
156
+
NULL,
157
+
};
158
+
159
+
static struct attribute_group cxl_decoder_switch_attribute_group = {
160
+
.attrs = cxl_decoder_switch_attrs,
161
+
};
162
+
163
+
static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = {
164
+
&cxl_decoder_switch_attribute_group,
165
+
&cxl_decoder_base_attribute_group,
166
+
&cxl_base_attribute_group,
167
+
NULL,
168
+
};
169
+
170
+
static void cxl_decoder_release(struct device *dev)
171
+
{
172
+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
173
+
struct cxl_port *port = to_cxl_port(dev->parent);
174
+
175
+
ida_free(&port->decoder_ida, cxld->id);
176
+
kfree(cxld);
177
+
}
178
+
179
+
static const struct device_type cxl_decoder_switch_type = {
180
+
.name = "cxl_decoder_switch",
181
+
.release = cxl_decoder_release,
182
+
.groups = cxl_decoder_switch_attribute_groups,
183
+
};
184
+
185
+
static const struct device_type cxl_decoder_root_type = {
186
+
.name = "cxl_decoder_root",
187
+
.release = cxl_decoder_release,
188
+
.groups = cxl_decoder_root_attribute_groups,
189
+
};
190
+
191
+
bool is_root_decoder(struct device *dev)
192
+
{
193
+
return dev->type == &cxl_decoder_root_type;
194
+
}
195
+
EXPORT_SYMBOL_GPL(is_root_decoder);
196
+
197
+
struct cxl_decoder *to_cxl_decoder(struct device *dev)
198
+
{
199
+
if (dev_WARN_ONCE(dev, dev->type->release != cxl_decoder_release,
200
+
"not a cxl_decoder device\n"))
201
+
return NULL;
202
+
return container_of(dev, struct cxl_decoder, dev);
203
+
}
204
+
EXPORT_SYMBOL_GPL(to_cxl_decoder);
205
+
206
+
static void cxl_dport_release(struct cxl_dport *dport)
207
+
{
208
+
list_del(&dport->list);
209
+
put_device(dport->dport);
210
+
kfree(dport);
211
+
}
212
+
213
+
static void cxl_port_release(struct device *dev)
214
+
{
215
+
struct cxl_port *port = to_cxl_port(dev);
216
+
struct cxl_dport *dport, *_d;
217
+
218
+
device_lock(dev);
219
+
list_for_each_entry_safe(dport, _d, &port->dports, list)
220
+
cxl_dport_release(dport);
221
+
device_unlock(dev);
222
+
ida_free(&cxl_port_ida, port->id);
223
+
kfree(port);
224
+
}
225
+
226
+
static const struct attribute_group *cxl_port_attribute_groups[] = {
227
+
&cxl_base_attribute_group,
228
+
NULL,
229
+
};
230
+
231
+
static const struct device_type cxl_port_type = {
232
+
.name = "cxl_port",
233
+
.release = cxl_port_release,
234
+
.groups = cxl_port_attribute_groups,
235
+
};
236
+
237
+
struct cxl_port *to_cxl_port(struct device *dev)
238
+
{
239
+
if (dev_WARN_ONCE(dev, dev->type != &cxl_port_type,
240
+
"not a cxl_port device\n"))
241
+
return NULL;
242
+
return container_of(dev, struct cxl_port, dev);
243
+
}
244
+
245
+
static void unregister_port(void *_port)
246
+
{
247
+
struct cxl_port *port = _port;
248
+
struct cxl_dport *dport;
249
+
250
+
device_lock(&port->dev);
251
+
list_for_each_entry(dport, &port->dports, list) {
252
+
char link_name[CXL_TARGET_STRLEN];
253
+
254
+
if (snprintf(link_name, CXL_TARGET_STRLEN, "dport%d",
255
+
dport->port_id) >= CXL_TARGET_STRLEN)
256
+
continue;
257
+
sysfs_remove_link(&port->dev.kobj, link_name);
258
+
}
259
+
device_unlock(&port->dev);
260
+
device_unregister(&port->dev);
261
+
}
262
+
263
+
static void cxl_unlink_uport(void *_port)
264
+
{
265
+
struct cxl_port *port = _port;
266
+
267
+
sysfs_remove_link(&port->dev.kobj, "uport");
268
+
}
269
+
270
+
static int devm_cxl_link_uport(struct device *host, struct cxl_port *port)
271
+
{
272
+
int rc;
273
+
274
+
rc = sysfs_create_link(&port->dev.kobj, &port->uport->kobj, "uport");
275
+
if (rc)
276
+
return rc;
277
+
return devm_add_action_or_reset(host, cxl_unlink_uport, port);
278
+
}
279
+
280
+
static struct cxl_port *cxl_port_alloc(struct device *uport,
281
+
resource_size_t component_reg_phys,
282
+
struct cxl_port *parent_port)
283
+
{
284
+
struct cxl_port *port;
285
+
struct device *dev;
286
+
int rc;
287
+
288
+
port = kzalloc(sizeof(*port), GFP_KERNEL);
289
+
if (!port)
290
+
return ERR_PTR(-ENOMEM);
291
+
292
+
rc = ida_alloc(&cxl_port_ida, GFP_KERNEL);
293
+
if (rc < 0)
294
+
goto err;
295
+
port->id = rc;
296
+
297
+
/*
298
+
* The top-level cxl_port "cxl_root" does not have a cxl_port as
299
+
* its parent and it does not have any corresponding component
300
+
* registers as its decode is described by a fixed platform
301
+
* description.
302
+
*/
303
+
dev = &port->dev;
304
+
if (parent_port)
305
+
dev->parent = &parent_port->dev;
306
+
else
307
+
dev->parent = uport;
308
+
309
+
port->uport = uport;
310
+
port->component_reg_phys = component_reg_phys;
311
+
ida_init(&port->decoder_ida);
312
+
INIT_LIST_HEAD(&port->dports);
313
+
314
+
device_initialize(dev);
315
+
device_set_pm_not_required(dev);
316
+
dev->bus = &cxl_bus_type;
317
+
dev->type = &cxl_port_type;
318
+
319
+
return port;
320
+
321
+
err:
322
+
kfree(port);
323
+
return ERR_PTR(rc);
324
+
}
325
+
326
+
/**
327
+
* devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy
328
+
* @host: host device for devm operations
329
+
* @uport: "physical" device implementing this upstream port
330
+
* @component_reg_phys: (optional) for configurable cxl_port instances
331
+
* @parent_port: next hop up in the CXL memory decode hierarchy
332
+
*/
333
+
struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,
334
+
resource_size_t component_reg_phys,
335
+
struct cxl_port *parent_port)
336
+
{
337
+
struct cxl_port *port;
338
+
struct device *dev;
339
+
int rc;
340
+
341
+
port = cxl_port_alloc(uport, component_reg_phys, parent_port);
342
+
if (IS_ERR(port))
343
+
return port;
344
+
345
+
dev = &port->dev;
346
+
if (parent_port)
347
+
rc = dev_set_name(dev, "port%d", port->id);
348
+
else
349
+
rc = dev_set_name(dev, "root%d", port->id);
350
+
if (rc)
351
+
goto err;
352
+
353
+
rc = device_add(dev);
354
+
if (rc)
355
+
goto err;
356
+
357
+
rc = devm_add_action_or_reset(host, unregister_port, port);
358
+
if (rc)
359
+
return ERR_PTR(rc);
360
+
361
+
rc = devm_cxl_link_uport(host, port);
362
+
if (rc)
363
+
return ERR_PTR(rc);
364
+
365
+
return port;
366
+
367
+
err:
368
+
put_device(dev);
369
+
return ERR_PTR(rc);
370
+
}
371
+
EXPORT_SYMBOL_GPL(devm_cxl_add_port);
372
+
373
+
static struct cxl_dport *find_dport(struct cxl_port *port, int id)
374
+
{
375
+
struct cxl_dport *dport;
376
+
377
+
device_lock_assert(&port->dev);
378
+
list_for_each_entry (dport, &port->dports, list)
379
+
if (dport->port_id == id)
380
+
return dport;
381
+
return NULL;
382
+
}
383
+
384
+
static int add_dport(struct cxl_port *port, struct cxl_dport *new)
385
+
{
386
+
struct cxl_dport *dup;
387
+
388
+
device_lock(&port->dev);
389
+
dup = find_dport(port, new->port_id);
390
+
if (dup)
391
+
dev_err(&port->dev,
392
+
"unable to add dport%d-%s non-unique port id (%s)\n",
393
+
new->port_id, dev_name(new->dport),
394
+
dev_name(dup->dport));
395
+
else
396
+
list_add_tail(&new->list, &port->dports);
397
+
device_unlock(&port->dev);
398
+
399
+
return dup ? -EEXIST : 0;
400
+
}
401
+
402
+
/**
403
+
* cxl_add_dport - append downstream port data to a cxl_port
404
+
* @port: the cxl_port that references this dport
405
+
* @dport_dev: firmware or PCI device representing the dport
406
+
* @port_id: identifier for this dport in a decoder's target list
407
+
* @component_reg_phys: optional location of CXL component registers
408
+
*
409
+
* Note that all allocations and links are undone by cxl_port deletion
410
+
* and release.
411
+
*/
412
+
int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,
413
+
resource_size_t component_reg_phys)
414
+
{
415
+
char link_name[CXL_TARGET_STRLEN];
416
+
struct cxl_dport *dport;
417
+
int rc;
418
+
419
+
if (snprintf(link_name, CXL_TARGET_STRLEN, "dport%d", port_id) >=
420
+
CXL_TARGET_STRLEN)
421
+
return -EINVAL;
422
+
423
+
dport = kzalloc(sizeof(*dport), GFP_KERNEL);
424
+
if (!dport)
425
+
return -ENOMEM;
426
+
427
+
INIT_LIST_HEAD(&dport->list);
428
+
dport->dport = get_device(dport_dev);
429
+
dport->port_id = port_id;
430
+
dport->component_reg_phys = component_reg_phys;
431
+
dport->port = port;
432
+
433
+
rc = add_dport(port, dport);
434
+
if (rc)
435
+
goto err;
436
+
437
+
rc = sysfs_create_link(&port->dev.kobj, &dport_dev->kobj, link_name);
438
+
if (rc)
439
+
goto err;
440
+
441
+
return 0;
442
+
err:
443
+
cxl_dport_release(dport);
444
+
return rc;
445
+
}
446
+
EXPORT_SYMBOL_GPL(cxl_add_dport);
447
+
448
+
static struct cxl_decoder *
449
+
cxl_decoder_alloc(struct cxl_port *port, int nr_targets, resource_size_t base,
450
+
resource_size_t len, int interleave_ways,
451
+
int interleave_granularity, enum cxl_decoder_type type,
452
+
unsigned long flags)
453
+
{
454
+
struct cxl_decoder *cxld;
455
+
struct device *dev;
456
+
int rc = 0;
457
+
458
+
if (interleave_ways < 1)
459
+
return ERR_PTR(-EINVAL);
460
+
461
+
device_lock(&port->dev);
462
+
if (list_empty(&port->dports))
463
+
rc = -EINVAL;
464
+
device_unlock(&port->dev);
465
+
if (rc)
466
+
return ERR_PTR(rc);
467
+
468
+
cxld = kzalloc(struct_size(cxld, target, nr_targets), GFP_KERNEL);
469
+
if (!cxld)
470
+
return ERR_PTR(-ENOMEM);
471
+
472
+
rc = ida_alloc(&port->decoder_ida, GFP_KERNEL);
473
+
if (rc < 0)
474
+
goto err;
475
+
476
+
*cxld = (struct cxl_decoder) {
477
+
.id = rc,
478
+
.range = {
479
+
.start = base,
480
+
.end = base + len - 1,
481
+
},
482
+
.flags = flags,
483
+
.interleave_ways = interleave_ways,
484
+
.interleave_granularity = interleave_granularity,
485
+
.target_type = type,
486
+
};
487
+
488
+
/* handle implied target_list */
489
+
if (interleave_ways == 1)
490
+
cxld->target[0] =
491
+
list_first_entry(&port->dports, struct cxl_dport, list);
492
+
dev = &cxld->dev;
493
+
device_initialize(dev);
494
+
device_set_pm_not_required(dev);
495
+
dev->parent = &port->dev;
496
+
dev->bus = &cxl_bus_type;
497
+
498
+
/* root ports do not have a cxl_port_type parent */
499
+
if (port->dev.parent->type == &cxl_port_type)
500
+
dev->type = &cxl_decoder_switch_type;
501
+
else
502
+
dev->type = &cxl_decoder_root_type;
503
+
504
+
return cxld;
505
+
err:
506
+
kfree(cxld);
507
+
return ERR_PTR(rc);
508
+
}
509
+
510
+
static void unregister_dev(void *dev)
511
+
{
512
+
device_unregister(dev);
513
+
}
514
+
515
+
struct cxl_decoder *
516
+
devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets,
517
+
resource_size_t base, resource_size_t len,
518
+
int interleave_ways, int interleave_granularity,
519
+
enum cxl_decoder_type type, unsigned long flags)
520
+
{
521
+
struct cxl_decoder *cxld;
522
+
struct device *dev;
523
+
int rc;
524
+
525
+
cxld = cxl_decoder_alloc(port, nr_targets, base, len, interleave_ways,
526
+
interleave_granularity, type, flags);
527
+
if (IS_ERR(cxld))
528
+
return cxld;
529
+
530
+
dev = &cxld->dev;
531
+
rc = dev_set_name(dev, "decoder%d.%d", port->id, cxld->id);
532
+
if (rc)
533
+
goto err;
534
+
535
+
rc = device_add(dev);
536
+
if (rc)
537
+
goto err;
538
+
539
+
rc = devm_add_action_or_reset(host, unregister_dev, dev);
540
+
if (rc)
541
+
return ERR_PTR(rc);
542
+
return cxld;
543
+
544
+
err:
545
+
put_device(dev);
546
+
return ERR_PTR(rc);
547
+
}
548
+
EXPORT_SYMBOL_GPL(devm_cxl_add_decoder);
549
+
550
+
/**
551
+
* cxl_probe_component_regs() - Detect CXL Component register blocks
552
+
* @dev: Host device of the @base mapping
553
+
* @base: Mapping containing the HDM Decoder Capability Header
554
+
* @map: Map object describing the register block information found
555
+
*
556
+
* See CXL 2.0 8.2.4 Component Register Layout and Definition
557
+
* See CXL 2.0 8.2.5.5 CXL Device Register Interface
558
+
*
559
+
* Probe for component register information and return it in map object.
560
+
*/
561
+
void cxl_probe_component_regs(struct device *dev, void __iomem *base,
562
+
struct cxl_component_reg_map *map)
563
+
{
564
+
int cap, cap_count;
565
+
u64 cap_array;
566
+
567
+
*map = (struct cxl_component_reg_map) { 0 };
568
+
569
+
/*
570
+
* CXL.cache and CXL.mem registers are at offset 0x1000 as defined in
571
+
* CXL 2.0 8.2.4 Table 141.
572
+
*/
573
+
base += CXL_CM_OFFSET;
574
+
575
+
cap_array = readq(base + CXL_CM_CAP_HDR_OFFSET);
576
+
577
+
if (FIELD_GET(CXL_CM_CAP_HDR_ID_MASK, cap_array) != CM_CAP_HDR_CAP_ID) {
578
+
dev_err(dev,
579
+
"Couldn't locate the CXL.cache and CXL.mem capability array header./n");
580
+
return;
581
+
}
582
+
583
+
/* It's assumed that future versions will be backward compatible */
584
+
cap_count = FIELD_GET(CXL_CM_CAP_HDR_ARRAY_SIZE_MASK, cap_array);
585
+
586
+
for (cap = 1; cap <= cap_count; cap++) {
587
+
void __iomem *register_block;
588
+
u32 hdr;
589
+
int decoder_cnt;
590
+
u16 cap_id, offset;
591
+
u32 length;
592
+
593
+
hdr = readl(base + cap * 0x4);
594
+
595
+
cap_id = FIELD_GET(CXL_CM_CAP_HDR_ID_MASK, hdr);
596
+
offset = FIELD_GET(CXL_CM_CAP_PTR_MASK, hdr);
597
+
register_block = base + offset;
598
+
599
+
switch (cap_id) {
600
+
case CXL_CM_CAP_CAP_ID_HDM:
601
+
dev_dbg(dev, "found HDM decoder capability (0x%x)\n",
602
+
offset);
603
+
604
+
hdr = readl(register_block);
605
+
606
+
decoder_cnt = cxl_hdm_decoder_count(hdr);
607
+
length = 0x20 * decoder_cnt + 0x10;
608
+
609
+
map->hdm_decoder.valid = true;
610
+
map->hdm_decoder.offset = CXL_CM_OFFSET + offset;
611
+
map->hdm_decoder.size = length;
612
+
break;
613
+
default:
614
+
dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
615
+
offset);
616
+
break;
617
+
}
618
+
}
619
+
}
620
+
EXPORT_SYMBOL_GPL(cxl_probe_component_regs);
621
+
622
+
static void cxl_nvdimm_bridge_release(struct device *dev)
623
+
{
624
+
struct cxl_nvdimm_bridge *cxl_nvb = to_cxl_nvdimm_bridge(dev);
625
+
626
+
kfree(cxl_nvb);
627
+
}
628
+
629
+
static const struct attribute_group *cxl_nvdimm_bridge_attribute_groups[] = {
630
+
&cxl_base_attribute_group,
631
+
NULL,
632
+
};
633
+
634
+
static const struct device_type cxl_nvdimm_bridge_type = {
635
+
.name = "cxl_nvdimm_bridge",
636
+
.release = cxl_nvdimm_bridge_release,
637
+
.groups = cxl_nvdimm_bridge_attribute_groups,
638
+
};
639
+
640
+
struct cxl_nvdimm_bridge *to_cxl_nvdimm_bridge(struct device *dev)
641
+
{
642
+
if (dev_WARN_ONCE(dev, dev->type != &cxl_nvdimm_bridge_type,
643
+
"not a cxl_nvdimm_bridge device\n"))
644
+
return NULL;
645
+
return container_of(dev, struct cxl_nvdimm_bridge, dev);
646
+
}
647
+
EXPORT_SYMBOL_GPL(to_cxl_nvdimm_bridge);
648
+
649
+
static struct cxl_nvdimm_bridge *
650
+
cxl_nvdimm_bridge_alloc(struct cxl_port *port)
651
+
{
652
+
struct cxl_nvdimm_bridge *cxl_nvb;
653
+
struct device *dev;
654
+
655
+
cxl_nvb = kzalloc(sizeof(*cxl_nvb), GFP_KERNEL);
656
+
if (!cxl_nvb)
657
+
return ERR_PTR(-ENOMEM);
658
+
659
+
dev = &cxl_nvb->dev;
660
+
cxl_nvb->port = port;
661
+
cxl_nvb->state = CXL_NVB_NEW;
662
+
device_initialize(dev);
663
+
device_set_pm_not_required(dev);
664
+
dev->parent = &port->dev;
665
+
dev->bus = &cxl_bus_type;
666
+
dev->type = &cxl_nvdimm_bridge_type;
667
+
668
+
return cxl_nvb;
669
+
}
670
+
671
+
static void unregister_nvb(void *_cxl_nvb)
672
+
{
673
+
struct cxl_nvdimm_bridge *cxl_nvb = _cxl_nvb;
674
+
bool flush;
675
+
676
+
/*
677
+
* If the bridge was ever activated then there might be in-flight state
678
+
* work to flush. Once the state has been changed to 'dead' then no new
679
+
* work can be queued by user-triggered bind.
680
+
*/
681
+
device_lock(&cxl_nvb->dev);
682
+
flush = cxl_nvb->state != CXL_NVB_NEW;
683
+
cxl_nvb->state = CXL_NVB_DEAD;
684
+
device_unlock(&cxl_nvb->dev);
685
+
686
+
/*
687
+
* Even though the device core will trigger device_release_driver()
688
+
* before the unregister, it does not know about the fact that
689
+
* cxl_nvdimm_bridge_driver defers ->remove() work. So, do the driver
690
+
* release not and flush it before tearing down the nvdimm device
691
+
* hierarchy.
692
+
*/
693
+
device_release_driver(&cxl_nvb->dev);
694
+
if (flush)
695
+
flush_work(&cxl_nvb->state_work);
696
+
device_unregister(&cxl_nvb->dev);
697
+
}
698
+
699
+
struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
700
+
struct cxl_port *port)
701
+
{
702
+
struct cxl_nvdimm_bridge *cxl_nvb;
703
+
struct device *dev;
704
+
int rc;
705
+
706
+
if (!IS_ENABLED(CONFIG_CXL_PMEM))
707
+
return ERR_PTR(-ENXIO);
708
+
709
+
cxl_nvb = cxl_nvdimm_bridge_alloc(port);
710
+
if (IS_ERR(cxl_nvb))
711
+
return cxl_nvb;
712
+
713
+
dev = &cxl_nvb->dev;
714
+
rc = dev_set_name(dev, "nvdimm-bridge");
715
+
if (rc)
716
+
goto err;
717
+
718
+
rc = device_add(dev);
719
+
if (rc)
720
+
goto err;
721
+
722
+
rc = devm_add_action_or_reset(host, unregister_nvb, cxl_nvb);
723
+
if (rc)
724
+
return ERR_PTR(rc);
725
+
726
+
return cxl_nvb;
727
+
728
+
err:
729
+
put_device(dev);
730
+
return ERR_PTR(rc);
731
+
}
732
+
EXPORT_SYMBOL_GPL(devm_cxl_add_nvdimm_bridge);
733
+
734
+
static void cxl_nvdimm_release(struct device *dev)
735
+
{
736
+
struct cxl_nvdimm *cxl_nvd = to_cxl_nvdimm(dev);
737
+
738
+
kfree(cxl_nvd);
739
+
}
740
+
741
+
static const struct attribute_group *cxl_nvdimm_attribute_groups[] = {
742
+
&cxl_base_attribute_group,
743
+
NULL,
744
+
};
745
+
746
+
static const struct device_type cxl_nvdimm_type = {
747
+
.name = "cxl_nvdimm",
748
+
.release = cxl_nvdimm_release,
749
+
.groups = cxl_nvdimm_attribute_groups,
750
+
};
751
+
752
+
bool is_cxl_nvdimm(struct device *dev)
753
+
{
754
+
return dev->type == &cxl_nvdimm_type;
755
+
}
756
+
EXPORT_SYMBOL_GPL(is_cxl_nvdimm);
757
+
758
+
struct cxl_nvdimm *to_cxl_nvdimm(struct device *dev)
759
+
{
760
+
if (dev_WARN_ONCE(dev, !is_cxl_nvdimm(dev),
761
+
"not a cxl_nvdimm device\n"))
762
+
return NULL;
763
+
return container_of(dev, struct cxl_nvdimm, dev);
764
+
}
765
+
EXPORT_SYMBOL_GPL(to_cxl_nvdimm);
766
+
767
+
static struct cxl_nvdimm *cxl_nvdimm_alloc(struct cxl_memdev *cxlmd)
768
+
{
769
+
struct cxl_nvdimm *cxl_nvd;
770
+
struct device *dev;
771
+
772
+
cxl_nvd = kzalloc(sizeof(*cxl_nvd), GFP_KERNEL);
773
+
if (!cxl_nvd)
774
+
return ERR_PTR(-ENOMEM);
775
+
776
+
dev = &cxl_nvd->dev;
777
+
cxl_nvd->cxlmd = cxlmd;
778
+
device_initialize(dev);
779
+
device_set_pm_not_required(dev);
780
+
dev->parent = &cxlmd->dev;
781
+
dev->bus = &cxl_bus_type;
782
+
dev->type = &cxl_nvdimm_type;
783
+
784
+
return cxl_nvd;
785
+
}
786
+
787
+
int devm_cxl_add_nvdimm(struct device *host, struct cxl_memdev *cxlmd)
788
+
{
789
+
struct cxl_nvdimm *cxl_nvd;
790
+
struct device *dev;
791
+
int rc;
792
+
793
+
cxl_nvd = cxl_nvdimm_alloc(cxlmd);
794
+
if (IS_ERR(cxl_nvd))
795
+
return PTR_ERR(cxl_nvd);
796
+
797
+
dev = &cxl_nvd->dev;
798
+
rc = dev_set_name(dev, "pmem%d", cxlmd->id);
799
+
if (rc)
800
+
goto err;
801
+
802
+
rc = device_add(dev);
803
+
if (rc)
804
+
goto err;
805
+
806
+
dev_dbg(host, "%s: register %s\n", dev_name(dev->parent),
807
+
dev_name(dev));
808
+
809
+
return devm_add_action_or_reset(host, unregister_dev, dev);
810
+
811
+
err:
812
+
put_device(dev);
813
+
return rc;
814
+
}
815
+
EXPORT_SYMBOL_GPL(devm_cxl_add_nvdimm);
816
+
817
+
/**
818
+
* cxl_probe_device_regs() - Detect CXL Device register blocks
819
+
* @dev: Host device of the @base mapping
820
+
* @base: Mapping of CXL 2.0 8.2.8 CXL Device Register Interface
821
+
* @map: Map object describing the register block information found
822
+
*
823
+
* Probe for device register information and return it in map object.
824
+
*/
825
+
void cxl_probe_device_regs(struct device *dev, void __iomem *base,
826
+
struct cxl_device_reg_map *map)
827
+
{
828
+
int cap, cap_count;
829
+
u64 cap_array;
830
+
831
+
*map = (struct cxl_device_reg_map){ 0 };
832
+
833
+
cap_array = readq(base + CXLDEV_CAP_ARRAY_OFFSET);
834
+
if (FIELD_GET(CXLDEV_CAP_ARRAY_ID_MASK, cap_array) !=
835
+
CXLDEV_CAP_ARRAY_CAP_ID)
836
+
return;
837
+
838
+
cap_count = FIELD_GET(CXLDEV_CAP_ARRAY_COUNT_MASK, cap_array);
839
+
840
+
for (cap = 1; cap <= cap_count; cap++) {
841
+
u32 offset, length;
842
+
u16 cap_id;
843
+
844
+
cap_id = FIELD_GET(CXLDEV_CAP_HDR_CAP_ID_MASK,
845
+
readl(base + cap * 0x10));
846
+
offset = readl(base + cap * 0x10 + 0x4);
847
+
length = readl(base + cap * 0x10 + 0x8);
848
+
849
+
switch (cap_id) {
850
+
case CXLDEV_CAP_CAP_ID_DEVICE_STATUS:
851
+
dev_dbg(dev, "found Status capability (0x%x)\n", offset);
852
+
853
+
map->status.valid = true;
854
+
map->status.offset = offset;
855
+
map->status.size = length;
856
+
break;
857
+
case CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX:
858
+
dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);
859
+
map->mbox.valid = true;
860
+
map->mbox.offset = offset;
861
+
map->mbox.size = length;
862
+
break;
863
+
case CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX:
864
+
dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);
865
+
break;
866
+
case CXLDEV_CAP_CAP_ID_MEMDEV:
867
+
dev_dbg(dev, "found Memory Device capability (0x%x)\n", offset);
868
+
map->memdev.valid = true;
869
+
map->memdev.offset = offset;
870
+
map->memdev.size = length;
871
+
break;
872
+
default:
873
+
if (cap_id >= 0x8000)
874
+
dev_dbg(dev, "Vendor cap ID: %#x offset: %#x\n", cap_id, offset);
875
+
else
876
+
dev_dbg(dev, "Unknown cap ID: %#x offset: %#x\n", cap_id, offset);
877
+
break;
878
+
}
879
+
}
880
+
}
881
+
EXPORT_SYMBOL_GPL(cxl_probe_device_regs);
882
+
883
+
static void __iomem *devm_cxl_iomap_block(struct device *dev,
884
+
resource_size_t addr,
885
+
resource_size_t length)
886
+
{
887
+
void __iomem *ret_val;
888
+
struct resource *res;
889
+
890
+
res = devm_request_mem_region(dev, addr, length, dev_name(dev));
891
+
if (!res) {
892
+
resource_size_t end = addr + length - 1;
893
+
894
+
dev_err(dev, "Failed to request region %pa-%pa\n", &addr, &end);
895
+
return NULL;
896
+
}
897
+
898
+
ret_val = devm_ioremap(dev, addr, length);
899
+
if (!ret_val)
900
+
dev_err(dev, "Failed to map region %pr\n", res);
901
+
902
+
return ret_val;
903
+
}
904
+
905
+
int cxl_map_component_regs(struct pci_dev *pdev,
906
+
struct cxl_component_regs *regs,
907
+
struct cxl_register_map *map)
908
+
{
909
+
struct device *dev = &pdev->dev;
910
+
resource_size_t phys_addr;
911
+
resource_size_t length;
912
+
913
+
phys_addr = pci_resource_start(pdev, map->barno);
914
+
phys_addr += map->block_offset;
915
+
916
+
phys_addr += map->component_map.hdm_decoder.offset;
917
+
length = map->component_map.hdm_decoder.size;
918
+
regs->hdm_decoder = devm_cxl_iomap_block(dev, phys_addr, length);
919
+
if (!regs->hdm_decoder)
920
+
return -ENOMEM;
921
+
922
+
return 0;
923
+
}
924
+
EXPORT_SYMBOL_GPL(cxl_map_component_regs);
925
+
926
+
int cxl_map_device_regs(struct pci_dev *pdev,
927
+
struct cxl_device_regs *regs,
928
+
struct cxl_register_map *map)
929
+
{
930
+
struct device *dev = &pdev->dev;
931
+
resource_size_t phys_addr;
932
+
933
+
phys_addr = pci_resource_start(pdev, map->barno);
934
+
phys_addr += map->block_offset;
935
+
936
+
if (map->device_map.status.valid) {
937
+
resource_size_t addr;
938
+
resource_size_t length;
939
+
940
+
addr = phys_addr + map->device_map.status.offset;
941
+
length = map->device_map.status.size;
942
+
regs->status = devm_cxl_iomap_block(dev, addr, length);
943
+
if (!regs->status)
944
+
return -ENOMEM;
945
+
}
946
+
947
+
if (map->device_map.mbox.valid) {
948
+
resource_size_t addr;
949
+
resource_size_t length;
950
+
951
+
addr = phys_addr + map->device_map.mbox.offset;
952
+
length = map->device_map.mbox.size;
953
+
regs->mbox = devm_cxl_iomap_block(dev, addr, length);
954
+
if (!regs->mbox)
955
+
return -ENOMEM;
956
+
}
957
+
958
+
if (map->device_map.memdev.valid) {
959
+
resource_size_t addr;
960
+
resource_size_t length;
961
+
962
+
addr = phys_addr + map->device_map.memdev.offset;
963
+
length = map->device_map.memdev.size;
964
+
regs->memdev = devm_cxl_iomap_block(dev, addr, length);
965
+
if (!regs->memdev)
966
+
return -ENOMEM;
967
+
}
968
+
969
+
return 0;
970
+
}
971
+
EXPORT_SYMBOL_GPL(cxl_map_device_regs);
972
+
973
+
/**
974
+
* __cxl_driver_register - register a driver for the cxl bus
975
+
* @cxl_drv: cxl driver structure to attach
976
+
* @owner: owning module/driver
977
+
* @modname: KBUILD_MODNAME for parent driver
978
+
*/
979
+
int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
980
+
const char *modname)
981
+
{
982
+
if (!cxl_drv->probe) {
983
+
pr_debug("%s ->probe() must be specified\n", modname);
984
+
return -EINVAL;
985
+
}
986
+
987
+
if (!cxl_drv->name) {
988
+
pr_debug("%s ->name must be specified\n", modname);
989
+
return -EINVAL;
990
+
}
991
+
992
+
if (!cxl_drv->id) {
993
+
pr_debug("%s ->id must be specified\n", modname);
994
+
return -EINVAL;
995
+
}
996
+
997
+
cxl_drv->drv.bus = &cxl_bus_type;
998
+
cxl_drv->drv.owner = owner;
999
+
cxl_drv->drv.mod_name = modname;
1000
+
cxl_drv->drv.name = cxl_drv->name;
1001
+
1002
+
return driver_register(&cxl_drv->drv);
1003
+
}
1004
+
EXPORT_SYMBOL_GPL(__cxl_driver_register);
1005
+
1006
+
void cxl_driver_unregister(struct cxl_driver *cxl_drv)
1007
+
{
1008
+
driver_unregister(&cxl_drv->drv);
1009
+
}
1010
+
EXPORT_SYMBOL_GPL(cxl_driver_unregister);
1011
+
1012
+
static int cxl_device_id(struct device *dev)
1013
+
{
1014
+
if (dev->type == &cxl_nvdimm_bridge_type)
1015
+
return CXL_DEVICE_NVDIMM_BRIDGE;
1016
+
if (dev->type == &cxl_nvdimm_type)
1017
+
return CXL_DEVICE_NVDIMM;
1018
+
return 0;
1019
+
}
1020
+
1021
+
static int cxl_bus_uevent(struct device *dev, struct kobj_uevent_env *env)
1022
+
{
1023
+
return add_uevent_var(env, "MODALIAS=" CXL_MODALIAS_FMT,
1024
+
cxl_device_id(dev));
1025
+
}
1026
+
1027
+
static int cxl_bus_match(struct device *dev, struct device_driver *drv)
1028
+
{
1029
+
return cxl_device_id(dev) == to_cxl_drv(drv)->id;
1030
+
}
1031
+
1032
+
static int cxl_bus_probe(struct device *dev)
1033
+
{
1034
+
return to_cxl_drv(dev->driver)->probe(dev);
1035
+
}
1036
+
1037
+
static int cxl_bus_remove(struct device *dev)
1038
+
{
1039
+
struct cxl_driver *cxl_drv = to_cxl_drv(dev->driver);
1040
+
1041
+
if (cxl_drv->remove)
1042
+
cxl_drv->remove(dev);
1043
+
return 0;
1044
+
}
1045
+
1046
+
struct bus_type cxl_bus_type = {
1047
+
.name = "cxl",
1048
+
.uevent = cxl_bus_uevent,
1049
+
.match = cxl_bus_match,
1050
+
.probe = cxl_bus_probe,
1051
+
.remove = cxl_bus_remove,
1052
+
};
1053
+
EXPORT_SYMBOL_GPL(cxl_bus_type);
1054
+
1055
+
static __init int cxl_core_init(void)
1056
+
{
1057
+
return bus_register(&cxl_bus_type);
1058
+
}
1059
+
1060
+
static void cxl_core_exit(void)
1061
+
{
1062
+
bus_unregister(&cxl_bus_type);
1063
+
}
1064
+
1065
+
module_init(cxl_core_init);
1066
+
module_exit(cxl_core_exit);
1067
+
MODULE_LICENSE("GPL v2");
+285
-53
drivers/cxl/cxl.h
+285
-53
drivers/cxl/cxl.h
···
4
4
#ifndef __CXL_H__
5
5
#define __CXL_H__
6
6
7
+
#include <linux/libnvdimm.h>
7
8
#include <linux/bitfield.h>
8
9
#include <linux/bitops.h>
9
10
#include <linux/io.h>
11
+
12
+
/**
13
+
* DOC: cxl objects
14
+
*
15
+
* The CXL core objects like ports, decoders, and regions are shared
16
+
* between the subsystem drivers cxl_acpi, cxl_pci, and core drivers
17
+
* (port-driver, region-driver, nvdimm object-drivers... etc).
18
+
*/
19
+
20
+
/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
21
+
#define CXL_CM_OFFSET 0x1000
22
+
#define CXL_CM_CAP_HDR_OFFSET 0x0
23
+
#define CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0)
24
+
#define CM_CAP_HDR_CAP_ID 1
25
+
#define CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16)
26
+
#define CM_CAP_HDR_CAP_VERSION 1
27
+
#define CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20)
28
+
#define CM_CAP_HDR_CACHE_MEM_VERSION 1
29
+
#define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
30
+
#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
31
+
32
+
#define CXL_CM_CAP_CAP_ID_HDM 0x5
33
+
#define CXL_CM_CAP_CAP_HDM_VERSION 1
34
+
35
+
/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
36
+
#define CXL_HDM_DECODER_CAP_OFFSET 0x0
37
+
#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
38
+
#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
39
+
#define CXL_HDM_DECODER0_BASE_LOW_OFFSET 0x10
40
+
#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET 0x14
41
+
#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET 0x18
42
+
#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET 0x1c
43
+
#define CXL_HDM_DECODER0_CTRL_OFFSET 0x20
44
+
45
+
static inline int cxl_hdm_decoder_count(u32 cap_hdr)
46
+
{
47
+
int val = FIELD_GET(CXL_HDM_DECODER_COUNT_MASK, cap_hdr);
48
+
49
+
return val ? val * 2 : 1;
50
+
}
10
51
11
52
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
12
53
#define CXLDEV_CAP_ARRAY_OFFSET 0x0
···
75
34
#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18
76
35
#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
77
36
78
-
/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
79
-
#define CXLMDEV_STATUS_OFFSET 0x0
80
-
#define CXLMDEV_DEV_FATAL BIT(0)
81
-
#define CXLMDEV_FW_HALT BIT(1)
82
-
#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
83
-
#define CXLMDEV_MS_NOT_READY 0
84
-
#define CXLMDEV_MS_READY 1
85
-
#define CXLMDEV_MS_ERROR 2
86
-
#define CXLMDEV_MS_DISABLED 3
87
-
#define CXLMDEV_READY(status) \
88
-
(FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) == \
89
-
CXLMDEV_MS_READY)
90
-
#define CXLMDEV_MBOX_IF_READY BIT(4)
91
-
#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
92
-
#define CXLMDEV_RESET_NEEDED_NOT 0
93
-
#define CXLMDEV_RESET_NEEDED_COLD 1
94
-
#define CXLMDEV_RESET_NEEDED_WARM 2
95
-
#define CXLMDEV_RESET_NEEDED_HOT 3
96
-
#define CXLMDEV_RESET_NEEDED_CXL 4
97
-
#define CXLMDEV_RESET_NEEDED(status) \
98
-
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
99
-
CXLMDEV_RESET_NEEDED_NOT)
37
+
#define CXL_COMPONENT_REGS() \
38
+
void __iomem *hdm_decoder
100
39
101
-
struct cxl_memdev;
102
-
/**
103
-
* struct cxl_mem - A CXL memory device
104
-
* @pdev: The PCI device associated with this CXL device.
105
-
* @regs: IO mappings to the device's MMIO
106
-
* @status_regs: CXL 2.0 8.2.8.3 Device Status Registers
107
-
* @mbox_regs: CXL 2.0 8.2.8.4 Mailbox Registers
108
-
* @memdev_regs: CXL 2.0 8.2.8.5 Memory Device Registers
109
-
* @payload_size: Size of space for payload
110
-
* (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
111
-
* @mbox_mutex: Mutex to synchronize mailbox access.
112
-
* @firmware_version: Firmware version for the memory device.
113
-
* @enabled_commands: Hardware commands found enabled in CEL.
114
-
* @pmem_range: Persistent memory capacity information.
115
-
* @ram_range: Volatile memory capacity information.
40
+
#define CXL_DEVICE_REGS() \
41
+
void __iomem *status; \
42
+
void __iomem *mbox; \
43
+
void __iomem *memdev
44
+
45
+
/* See note for 'struct cxl_regs' for the rationale of this organization */
46
+
/*
47
+
* CXL_COMPONENT_REGS - Common set of CXL Component register block base pointers
48
+
* @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure
116
49
*/
117
-
struct cxl_mem {
118
-
struct pci_dev *pdev;
119
-
void __iomem *regs;
120
-
struct cxl_memdev *cxlmd;
121
-
122
-
void __iomem *status_regs;
123
-
void __iomem *mbox_regs;
124
-
void __iomem *memdev_regs;
125
-
126
-
size_t payload_size;
127
-
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
128
-
char firmware_version[0x10];
129
-
unsigned long *enabled_cmds;
130
-
131
-
struct range pmem_range;
132
-
struct range ram_range;
50
+
struct cxl_component_regs {
51
+
CXL_COMPONENT_REGS();
133
52
};
134
53
54
+
/* See note for 'struct cxl_regs' for the rationale of this organization */
55
+
/*
56
+
* CXL_DEVICE_REGS - Common set of CXL Device register block base pointers
57
+
* @status: CXL 2.0 8.2.8.3 Device Status Registers
58
+
* @mbox: CXL 2.0 8.2.8.4 Mailbox Registers
59
+
* @memdev: CXL 2.0 8.2.8.5 Memory Device Registers
60
+
*/
61
+
struct cxl_device_regs {
62
+
CXL_DEVICE_REGS();
63
+
};
64
+
65
+
/*
66
+
* Note, the anonymous union organization allows for per
67
+
* register-block-type helper routines, without requiring block-type
68
+
* agnostic code to include the prefix.
69
+
*/
70
+
struct cxl_regs {
71
+
union {
72
+
struct {
73
+
CXL_COMPONENT_REGS();
74
+
};
75
+
struct cxl_component_regs component;
76
+
};
77
+
union {
78
+
struct {
79
+
CXL_DEVICE_REGS();
80
+
};
81
+
struct cxl_device_regs device_regs;
82
+
};
83
+
};
84
+
85
+
struct cxl_reg_map {
86
+
bool valid;
87
+
unsigned long offset;
88
+
unsigned long size;
89
+
};
90
+
91
+
struct cxl_component_reg_map {
92
+
struct cxl_reg_map hdm_decoder;
93
+
};
94
+
95
+
struct cxl_device_reg_map {
96
+
struct cxl_reg_map status;
97
+
struct cxl_reg_map mbox;
98
+
struct cxl_reg_map memdev;
99
+
};
100
+
101
+
struct cxl_register_map {
102
+
struct list_head list;
103
+
u64 block_offset;
104
+
u8 reg_type;
105
+
u8 barno;
106
+
union {
107
+
struct cxl_component_reg_map component_map;
108
+
struct cxl_device_reg_map device_map;
109
+
};
110
+
};
111
+
112
+
void cxl_probe_component_regs(struct device *dev, void __iomem *base,
113
+
struct cxl_component_reg_map *map);
114
+
void cxl_probe_device_regs(struct device *dev, void __iomem *base,
115
+
struct cxl_device_reg_map *map);
116
+
int cxl_map_component_regs(struct pci_dev *pdev,
117
+
struct cxl_component_regs *regs,
118
+
struct cxl_register_map *map);
119
+
int cxl_map_device_regs(struct pci_dev *pdev,
120
+
struct cxl_device_regs *regs,
121
+
struct cxl_register_map *map);
122
+
123
+
#define CXL_RESOURCE_NONE ((resource_size_t) -1)
124
+
#define CXL_TARGET_STRLEN 20
125
+
126
+
/*
127
+
* cxl_decoder flags that define the type of memory / devices this
128
+
* decoder supports as well as configuration lock status See "CXL 2.0
129
+
* 8.2.5.12.7 CXL HDM Decoder 0 Control Register" for details.
130
+
*/
131
+
#define CXL_DECODER_F_RAM BIT(0)
132
+
#define CXL_DECODER_F_PMEM BIT(1)
133
+
#define CXL_DECODER_F_TYPE2 BIT(2)
134
+
#define CXL_DECODER_F_TYPE3 BIT(3)
135
+
#define CXL_DECODER_F_LOCK BIT(4)
136
+
#define CXL_DECODER_F_MASK GENMASK(4, 0)
137
+
138
+
enum cxl_decoder_type {
139
+
CXL_DECODER_ACCELERATOR = 2,
140
+
CXL_DECODER_EXPANDER = 3,
141
+
};
142
+
143
+
/**
144
+
* struct cxl_decoder - CXL address range decode configuration
145
+
* @dev: this decoder's device
146
+
* @id: kernel device name id
147
+
* @range: address range considered by this decoder
148
+
* @interleave_ways: number of cxl_dports in this decode
149
+
* @interleave_granularity: data stride per dport
150
+
* @target_type: accelerator vs expander (type2 vs type3) selector
151
+
* @flags: memory type capabilities and locking
152
+
* @target: active ordered target list in current decoder configuration
153
+
*/
154
+
struct cxl_decoder {
155
+
struct device dev;
156
+
int id;
157
+
struct range range;
158
+
int interleave_ways;
159
+
int interleave_granularity;
160
+
enum cxl_decoder_type target_type;
161
+
unsigned long flags;
162
+
struct cxl_dport *target[];
163
+
};
164
+
165
+
166
+
enum cxl_nvdimm_brige_state {
167
+
CXL_NVB_NEW,
168
+
CXL_NVB_DEAD,
169
+
CXL_NVB_ONLINE,
170
+
CXL_NVB_OFFLINE,
171
+
};
172
+
173
+
struct cxl_nvdimm_bridge {
174
+
struct device dev;
175
+
struct cxl_port *port;
176
+
struct nvdimm_bus *nvdimm_bus;
177
+
struct nvdimm_bus_descriptor nd_desc;
178
+
struct work_struct state_work;
179
+
enum cxl_nvdimm_brige_state state;
180
+
};
181
+
182
+
struct cxl_nvdimm {
183
+
struct device dev;
184
+
struct cxl_memdev *cxlmd;
185
+
struct nvdimm *nvdimm;
186
+
};
187
+
188
+
/**
189
+
* struct cxl_port - logical collection of upstream port devices and
190
+
* downstream port devices to construct a CXL memory
191
+
* decode hierarchy.
192
+
* @dev: this port's device
193
+
* @uport: PCI or platform device implementing the upstream port capability
194
+
* @id: id for port device-name
195
+
* @dports: cxl_dport instances referenced by decoders
196
+
* @decoder_ida: allocator for decoder ids
197
+
* @component_reg_phys: component register capability base address (optional)
198
+
*/
199
+
struct cxl_port {
200
+
struct device dev;
201
+
struct device *uport;
202
+
int id;
203
+
struct list_head dports;
204
+
struct ida decoder_ida;
205
+
resource_size_t component_reg_phys;
206
+
};
207
+
208
+
/**
209
+
* struct cxl_dport - CXL downstream port
210
+
* @dport: PCI bridge or firmware device representing the downstream link
211
+
* @port_id: unique hardware identifier for dport in decoder target list
212
+
* @component_reg_phys: downstream port component registers
213
+
* @port: reference to cxl_port that contains this downstream port
214
+
* @list: node for a cxl_port's list of cxl_dport instances
215
+
*/
216
+
struct cxl_dport {
217
+
struct device *dport;
218
+
int port_id;
219
+
resource_size_t component_reg_phys;
220
+
struct cxl_port *port;
221
+
struct list_head list;
222
+
};
223
+
224
+
struct cxl_port *to_cxl_port(struct device *dev);
225
+
struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,
226
+
resource_size_t component_reg_phys,
227
+
struct cxl_port *parent_port);
228
+
229
+
int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id,
230
+
resource_size_t component_reg_phys);
231
+
232
+
struct cxl_decoder *to_cxl_decoder(struct device *dev);
233
+
bool is_root_decoder(struct device *dev);
234
+
struct cxl_decoder *
235
+
devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets,
236
+
resource_size_t base, resource_size_t len,
237
+
int interleave_ways, int interleave_granularity,
238
+
enum cxl_decoder_type type, unsigned long flags);
239
+
240
+
/*
241
+
* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability Structure)
242
+
* single ported host-bridges need not publish a decoder capability when a
243
+
* passthrough decode can be assumed, i.e. all transactions that the uport sees
244
+
* are claimed and passed to the single dport. Default the range a 0-base
245
+
* 0-length until the first CXL region is activated.
246
+
*/
247
+
static inline struct cxl_decoder *
248
+
devm_cxl_add_passthrough_decoder(struct device *host, struct cxl_port *port)
249
+
{
250
+
return devm_cxl_add_decoder(host, port, 1, 0, 0, 1, PAGE_SIZE,
251
+
CXL_DECODER_EXPANDER, 0);
252
+
}
253
+
135
254
extern struct bus_type cxl_bus_type;
255
+
256
+
struct cxl_driver {
257
+
const char *name;
258
+
int (*probe)(struct device *dev);
259
+
void (*remove)(struct device *dev);
260
+
struct device_driver drv;
261
+
int id;
262
+
};
263
+
264
+
static inline struct cxl_driver *to_cxl_drv(struct device_driver *drv)
265
+
{
266
+
return container_of(drv, struct cxl_driver, drv);
267
+
}
268
+
269
+
int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
270
+
const char *modname);
271
+
#define cxl_driver_register(x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)
272
+
void cxl_driver_unregister(struct cxl_driver *cxl_drv);
273
+
274
+
#define CXL_DEVICE_NVDIMM_BRIDGE 1
275
+
#define CXL_DEVICE_NVDIMM 2
276
+
277
+
#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
278
+
#define CXL_MODALIAS_FMT "cxl:t%d"
279
+
280
+
struct cxl_nvdimm_bridge *to_cxl_nvdimm_bridge(struct device *dev);
281
+
struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
282
+
struct cxl_port *port);
283
+
struct cxl_nvdimm *to_cxl_nvdimm(struct device *dev);
284
+
bool is_cxl_nvdimm(struct device *dev);
285
+
int devm_cxl_add_nvdimm(struct device *host, struct cxl_memdev *cxlmd);
136
286
#endif /* __CXL_H__ */
+271
-168
drivers/cxl/mem.c
drivers/cxl/pci.c
+271
-168
drivers/cxl/mem.c
drivers/cxl/pci.c
···
6
6
#include <linux/module.h>
7
7
#include <linux/sizes.h>
8
8
#include <linux/mutex.h>
9
+
#include <linux/list.h>
9
10
#include <linux/cdev.h>
10
11
#include <linux/idr.h>
11
12
#include <linux/pci.h>
···
14
13
#include <linux/io-64-nonatomic-lo-hi.h>
15
14
#include "pci.h"
16
15
#include "cxl.h"
16
+
#include "mem.h"
17
17
18
18
/**
19
-
* DOC: cxl mem
19
+
* DOC: cxl pci
20
20
*
21
-
* This implements a CXL memory device ("type-3") as it is defined by the
22
-
* Compute Express Link specification.
21
+
* This implements the PCI exclusive functionality for a CXL device as it is
22
+
* defined by the Compute Express Link specification. CXL devices may surface
23
+
* certain functionality even if it isn't CXL enabled.
23
24
*
24
25
* The driver has several responsibilities, mainly:
25
26
* - Create the memX device and register on the CXL bus.
···
29
26
* - Probe the device attributes to establish sysfs interface.
30
27
* - Provide an IOCTL interface to userspace to communicate with the device for
31
28
* things like firmware update.
32
-
* - Support management of interleave sets.
33
-
* - Handle and manage error conditions.
34
29
*/
35
-
36
-
/*
37
-
* An entire PCI topology full of devices should be enough for any
38
-
* config
39
-
*/
40
-
#define CXL_MEM_MAX_DEVS 65536
41
30
42
31
#define cxl_doorbell_busy(cxlm) \
43
-
(readl((cxlm)->mbox_regs + CXLDEV_MBOX_CTRL_OFFSET) & \
32
+
(readl((cxlm)->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET) & \
44
33
CXLDEV_MBOX_CTRL_DOORBELL)
45
34
46
35
/* CXL 2.0 - 8.2.8.4 */
···
51
56
CXL_MBOX_OP_GET_LSA = 0x4102,
52
57
CXL_MBOX_OP_SET_LSA = 0x4103,
53
58
CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
59
+
CXL_MBOX_OP_GET_ALERT_CONFIG = 0x4201,
60
+
CXL_MBOX_OP_SET_ALERT_CONFIG = 0x4202,
61
+
CXL_MBOX_OP_GET_SHUTDOWN_STATE = 0x4203,
54
62
CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
63
+
CXL_MBOX_OP_GET_POISON = 0x4300,
64
+
CXL_MBOX_OP_INJECT_POISON = 0x4301,
65
+
CXL_MBOX_OP_CLEAR_POISON = 0x4302,
66
+
CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS = 0x4303,
55
67
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
56
68
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
57
69
CXL_MBOX_OP_MAX = 0x10000
···
92
90
size_t size_out;
93
91
u16 return_code;
94
92
#define CXL_MBOX_SUCCESS 0
95
-
};
96
-
97
-
/**
98
-
* struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
99
-
* @dev: driver core device object
100
-
* @cdev: char dev core object for ioctl operations
101
-
* @cxlm: pointer to the parent device driver data
102
-
* @id: id number of this memdev instance.
103
-
*/
104
-
struct cxl_memdev {
105
-
struct device dev;
106
-
struct cdev cdev;
107
-
struct cxl_mem *cxlm;
108
-
int id;
109
93
};
110
94
111
95
static int cxl_mem_major;
···
166
178
CXL_CMD(GET_LSA, 0x8, ~0, 0),
167
179
CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
168
180
CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
181
+
CXL_CMD(SET_PARTITION_INFO, 0x0a, 0, 0),
182
+
CXL_CMD(SET_LSA, ~0, 0, 0),
183
+
CXL_CMD(GET_ALERT_CONFIG, 0, 0x10, 0),
184
+
CXL_CMD(SET_ALERT_CONFIG, 0xc, 0, 0),
185
+
CXL_CMD(GET_SHUTDOWN_STATE, 0, 0x1, 0),
186
+
CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
187
+
CXL_CMD(GET_POISON, 0x10, ~0, 0),
188
+
CXL_CMD(INJECT_POISON, 0x8, 0, 0),
189
+
CXL_CMD(CLEAR_POISON, 0x48, 0, 0),
190
+
CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
191
+
CXL_CMD(SCAN_MEDIA, 0x11, 0, 0),
192
+
CXL_CMD(GET_SCAN_MEDIA, 0, ~0, 0),
169
193
};
170
194
171
195
/*
···
292
292
static int __cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
293
293
struct mbox_cmd *mbox_cmd)
294
294
{
295
-
void __iomem *payload = cxlm->mbox_regs + CXLDEV_MBOX_PAYLOAD_OFFSET;
295
+
void __iomem *payload = cxlm->regs.mbox + CXLDEV_MBOX_PAYLOAD_OFFSET;
296
296
u64 cmd_reg, status_reg;
297
297
size_t out_len;
298
298
int rc;
···
335
335
}
336
336
337
337
/* #2, #3 */
338
-
writeq(cmd_reg, cxlm->mbox_regs + CXLDEV_MBOX_CMD_OFFSET);
338
+
writeq(cmd_reg, cxlm->regs.mbox + CXLDEV_MBOX_CMD_OFFSET);
339
339
340
340
/* #4 */
341
341
dev_dbg(&cxlm->pdev->dev, "Sending command\n");
342
342
writel(CXLDEV_MBOX_CTRL_DOORBELL,
343
-
cxlm->mbox_regs + CXLDEV_MBOX_CTRL_OFFSET);
343
+
cxlm->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET);
344
344
345
345
/* #5 */
346
346
rc = cxl_mem_wait_for_doorbell(cxlm);
···
350
350
}
351
351
352
352
/* #6 */
353
-
status_reg = readq(cxlm->mbox_regs + CXLDEV_MBOX_STATUS_OFFSET);
353
+
status_reg = readq(cxlm->regs.mbox + CXLDEV_MBOX_STATUS_OFFSET);
354
354
mbox_cmd->return_code =
355
355
FIELD_GET(CXLDEV_MBOX_STATUS_RET_CODE_MASK, status_reg);
356
356
···
360
360
}
361
361
362
362
/* #7 */
363
-
cmd_reg = readq(cxlm->mbox_regs + CXLDEV_MBOX_CMD_OFFSET);
363
+
cmd_reg = readq(cxlm->regs.mbox + CXLDEV_MBOX_CMD_OFFSET);
364
364
out_len = FIELD_GET(CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK, cmd_reg);
365
365
366
366
/* #8 */
···
421
421
goto out;
422
422
}
423
423
424
-
md_status = readq(cxlm->memdev_regs + CXLMDEV_STATUS_OFFSET);
424
+
md_status = readq(cxlm->regs.memdev + CXLMDEV_STATUS_OFFSET);
425
425
if (!(md_status & CXLMDEV_MBOX_IF_READY && CXLMDEV_READY(md_status))) {
426
426
dev_err(dev, "mbox: reported doorbell ready, but not mbox ready\n");
427
427
rc = -EBUSY;
···
890
890
return 0;
891
891
}
892
892
893
-
/**
894
-
* cxl_mem_setup_regs() - Setup necessary MMIO.
895
-
* @cxlm: The CXL memory device to communicate with.
896
-
*
897
-
* Return: 0 if all necessary registers mapped.
898
-
*
899
-
* A memory device is required by spec to implement a certain set of MMIO
900
-
* regions. The purpose of this function is to enumerate and map those
901
-
* registers.
902
-
*/
903
-
static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
904
-
{
905
-
struct device *dev = &cxlm->pdev->dev;
906
-
int cap, cap_count;
907
-
u64 cap_array;
908
-
909
-
cap_array = readq(cxlm->regs + CXLDEV_CAP_ARRAY_OFFSET);
910
-
if (FIELD_GET(CXLDEV_CAP_ARRAY_ID_MASK, cap_array) !=
911
-
CXLDEV_CAP_ARRAY_CAP_ID)
912
-
return -ENODEV;
913
-
914
-
cap_count = FIELD_GET(CXLDEV_CAP_ARRAY_COUNT_MASK, cap_array);
915
-
916
-
for (cap = 1; cap <= cap_count; cap++) {
917
-
void __iomem *register_block;
918
-
u32 offset;
919
-
u16 cap_id;
920
-
921
-
cap_id = FIELD_GET(CXLDEV_CAP_HDR_CAP_ID_MASK,
922
-
readl(cxlm->regs + cap * 0x10));
923
-
offset = readl(cxlm->regs + cap * 0x10 + 0x4);
924
-
register_block = cxlm->regs + offset;
925
-
926
-
switch (cap_id) {
927
-
case CXLDEV_CAP_CAP_ID_DEVICE_STATUS:
928
-
dev_dbg(dev, "found Status capability (0x%x)\n", offset);
929
-
cxlm->status_regs = register_block;
930
-
break;
931
-
case CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX:
932
-
dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);
933
-
cxlm->mbox_regs = register_block;
934
-
break;
935
-
case CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX:
936
-
dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);
937
-
break;
938
-
case CXLDEV_CAP_CAP_ID_MEMDEV:
939
-
dev_dbg(dev, "found Memory Device capability (0x%x)\n", offset);
940
-
cxlm->memdev_regs = register_block;
941
-
break;
942
-
default:
943
-
dev_dbg(dev, "Unknown cap ID: %d (0x%x)\n", cap_id, offset);
944
-
break;
945
-
}
946
-
}
947
-
948
-
if (!cxlm->status_regs || !cxlm->mbox_regs || !cxlm->memdev_regs) {
949
-
dev_err(dev, "registers not found: %s%s%s\n",
950
-
!cxlm->status_regs ? "status " : "",
951
-
!cxlm->mbox_regs ? "mbox " : "",
952
-
!cxlm->memdev_regs ? "memdev" : "");
953
-
return -ENXIO;
954
-
}
955
-
956
-
return 0;
957
-
}
958
-
959
893
static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
960
894
{
961
-
const int cap = readl(cxlm->mbox_regs + CXLDEV_MBOX_CAPS_OFFSET);
895
+
const int cap = readl(cxlm->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET);
962
896
963
897
cxlm->payload_size =
964
898
1 << FIELD_GET(CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK, cap);
···
917
983
return 0;
918
984
}
919
985
920
-
static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
921
-
u32 reg_hi)
986
+
static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev)
922
987
{
923
988
struct device *dev = &pdev->dev;
924
989
struct cxl_mem *cxlm;
925
-
void __iomem *regs;
926
-
u64 offset;
927
-
u8 bar;
928
-
int rc;
929
990
930
-
cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
991
+
cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
931
992
if (!cxlm) {
932
993
dev_err(dev, "No memory available\n");
933
-
return NULL;
994
+
return ERR_PTR(-ENOMEM);
934
995
}
935
-
936
-
offset = ((u64)reg_hi << 32) | (reg_lo & CXL_REGLOC_ADDR_MASK);
937
-
bar = FIELD_GET(CXL_REGLOC_BIR_MASK, reg_lo);
938
-
939
-
/* Basic sanity check that BAR is big enough */
940
-
if (pci_resource_len(pdev, bar) < offset) {
941
-
dev_err(dev, "BAR%d: %pr: too small (offset: %#llx)\n", bar,
942
-
&pdev->resource[bar], (unsigned long long)offset);
943
-
return NULL;
944
-
}
945
-
946
-
rc = pcim_iomap_regions(pdev, BIT(bar), pci_name(pdev));
947
-
if (rc) {
948
-
dev_err(dev, "failed to map registers\n");
949
-
return NULL;
950
-
}
951
-
regs = pcim_iomap_table(pdev)[bar];
952
996
953
997
mutex_init(&cxlm->mbox_mutex);
954
998
cxlm->pdev = pdev;
955
-
cxlm->regs = regs + offset;
956
999
cxlm->enabled_cmds =
957
1000
devm_kmalloc_array(dev, BITS_TO_LONGS(cxl_cmd_count),
958
1001
sizeof(unsigned long),
959
1002
GFP_KERNEL | __GFP_ZERO);
960
1003
if (!cxlm->enabled_cmds) {
961
1004
dev_err(dev, "No memory available for bitmap\n");
962
-
return NULL;
1005
+
return ERR_PTR(-ENOMEM);
963
1006
}
964
1007
965
-
dev_dbg(dev, "Mapped CXL Memory Device resource\n");
966
1008
return cxlm;
1009
+
}
1010
+
1011
+
static void __iomem *cxl_mem_map_regblock(struct cxl_mem *cxlm,
1012
+
u8 bar, u64 offset)
1013
+
{
1014
+
struct pci_dev *pdev = cxlm->pdev;
1015
+
struct device *dev = &pdev->dev;
1016
+
void __iomem *addr;
1017
+
1018
+
/* Basic sanity check that BAR is big enough */
1019
+
if (pci_resource_len(pdev, bar) < offset) {
1020
+
dev_err(dev, "BAR%d: %pr: too small (offset: %#llx)\n", bar,
1021
+
&pdev->resource[bar], (unsigned long long)offset);
1022
+
return IOMEM_ERR_PTR(-ENXIO);
1023
+
}
1024
+
1025
+
addr = pci_iomap(pdev, bar, 0);
1026
+
if (!addr) {
1027
+
dev_err(dev, "failed to map registers\n");
1028
+
return addr;
1029
+
}
1030
+
1031
+
dev_dbg(dev, "Mapped CXL Memory Device resource bar %u @ %#llx\n",
1032
+
bar, offset);
1033
+
1034
+
return addr;
1035
+
}
1036
+
1037
+
static void cxl_mem_unmap_regblock(struct cxl_mem *cxlm, void __iomem *base)
1038
+
{
1039
+
pci_iounmap(cxlm->pdev, base);
967
1040
}
968
1041
969
1042
static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
···
994
1053
}
995
1054
996
1055
return 0;
1056
+
}
1057
+
1058
+
static int cxl_probe_regs(struct cxl_mem *cxlm, void __iomem *base,
1059
+
struct cxl_register_map *map)
1060
+
{
1061
+
struct pci_dev *pdev = cxlm->pdev;
1062
+
struct device *dev = &pdev->dev;
1063
+
struct cxl_component_reg_map *comp_map;
1064
+
struct cxl_device_reg_map *dev_map;
1065
+
1066
+
switch (map->reg_type) {
1067
+
case CXL_REGLOC_RBI_COMPONENT:
1068
+
comp_map = &map->component_map;
1069
+
cxl_probe_component_regs(dev, base, comp_map);
1070
+
if (!comp_map->hdm_decoder.valid) {
1071
+
dev_err(dev, "HDM decoder registers not found\n");
1072
+
return -ENXIO;
1073
+
}
1074
+
1075
+
dev_dbg(dev, "Set up component registers\n");
1076
+
break;
1077
+
case CXL_REGLOC_RBI_MEMDEV:
1078
+
dev_map = &map->device_map;
1079
+
cxl_probe_device_regs(dev, base, dev_map);
1080
+
if (!dev_map->status.valid || !dev_map->mbox.valid ||
1081
+
!dev_map->memdev.valid) {
1082
+
dev_err(dev, "registers not found: %s%s%s\n",
1083
+
!dev_map->status.valid ? "status " : "",
1084
+
!dev_map->mbox.valid ? "status " : "",
1085
+
!dev_map->memdev.valid ? "status " : "");
1086
+
return -ENXIO;
1087
+
}
1088
+
1089
+
dev_dbg(dev, "Probing device registers...\n");
1090
+
break;
1091
+
default:
1092
+
break;
1093
+
}
1094
+
1095
+
return 0;
1096
+
}
1097
+
1098
+
static int cxl_map_regs(struct cxl_mem *cxlm, struct cxl_register_map *map)
1099
+
{
1100
+
struct pci_dev *pdev = cxlm->pdev;
1101
+
struct device *dev = &pdev->dev;
1102
+
1103
+
switch (map->reg_type) {
1104
+
case CXL_REGLOC_RBI_COMPONENT:
1105
+
cxl_map_component_regs(pdev, &cxlm->regs.component, map);
1106
+
dev_dbg(dev, "Mapping component registers...\n");
1107
+
break;
1108
+
case CXL_REGLOC_RBI_MEMDEV:
1109
+
cxl_map_device_regs(pdev, &cxlm->regs.device_regs, map);
1110
+
dev_dbg(dev, "Probing device registers...\n");
1111
+
break;
1112
+
default:
1113
+
break;
1114
+
}
1115
+
1116
+
return 0;
1117
+
}
1118
+
1119
+
static void cxl_decode_register_block(u32 reg_lo, u32 reg_hi,
1120
+
u8 *bar, u64 *offset, u8 *reg_type)
1121
+
{
1122
+
*offset = ((u64)reg_hi << 32) | (reg_lo & CXL_REGLOC_ADDR_MASK);
1123
+
*bar = FIELD_GET(CXL_REGLOC_BIR_MASK, reg_lo);
1124
+
*reg_type = FIELD_GET(CXL_REGLOC_RBI_MASK, reg_lo);
1125
+
}
1126
+
1127
+
/**
1128
+
* cxl_mem_setup_regs() - Setup necessary MMIO.
1129
+
* @cxlm: The CXL memory device to communicate with.
1130
+
*
1131
+
* Return: 0 if all necessary registers mapped.
1132
+
*
1133
+
* A memory device is required by spec to implement a certain set of MMIO
1134
+
* regions. The purpose of this function is to enumerate and map those
1135
+
* registers.
1136
+
*/
1137
+
static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
1138
+
{
1139
+
struct pci_dev *pdev = cxlm->pdev;
1140
+
struct device *dev = &pdev->dev;
1141
+
u32 regloc_size, regblocks;
1142
+
void __iomem *base;
1143
+
int regloc, i;
1144
+
struct cxl_register_map *map, *n;
1145
+
LIST_HEAD(register_maps);
1146
+
int ret = 0;
1147
+
1148
+
regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID);
1149
+
if (!regloc) {
1150
+
dev_err(dev, "register location dvsec not found\n");
1151
+
return -ENXIO;
1152
+
}
1153
+
1154
+
if (pci_request_mem_regions(pdev, pci_name(pdev)))
1155
+
return -ENODEV;
1156
+
1157
+
/* Get the size of the Register Locator DVSEC */
1158
+
pci_read_config_dword(pdev, regloc + PCI_DVSEC_HEADER1, ®loc_size);
1159
+
regloc_size = FIELD_GET(PCI_DVSEC_HEADER1_LENGTH_MASK, regloc_size);
1160
+
1161
+
regloc += PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET;
1162
+
regblocks = (regloc_size - PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET) / 8;
1163
+
1164
+
for (i = 0; i < regblocks; i++, regloc += 8) {
1165
+
u32 reg_lo, reg_hi;
1166
+
u8 reg_type;
1167
+
u64 offset;
1168
+
u8 bar;
1169
+
1170
+
map = kzalloc(sizeof(*map), GFP_KERNEL);
1171
+
if (!map) {
1172
+
ret = -ENOMEM;
1173
+
goto free_maps;
1174
+
}
1175
+
1176
+
list_add(&map->list, ®ister_maps);
1177
+
1178
+
pci_read_config_dword(pdev, regloc, ®_lo);
1179
+
pci_read_config_dword(pdev, regloc + 4, ®_hi);
1180
+
1181
+
cxl_decode_register_block(reg_lo, reg_hi, &bar, &offset,
1182
+
®_type);
1183
+
1184
+
dev_dbg(dev, "Found register block in bar %u @ 0x%llx of type %u\n",
1185
+
bar, offset, reg_type);
1186
+
1187
+
base = cxl_mem_map_regblock(cxlm, bar, offset);
1188
+
if (!base) {
1189
+
ret = -ENOMEM;
1190
+
goto free_maps;
1191
+
}
1192
+
1193
+
map->barno = bar;
1194
+
map->block_offset = offset;
1195
+
map->reg_type = reg_type;
1196
+
1197
+
ret = cxl_probe_regs(cxlm, base + offset, map);
1198
+
1199
+
/* Always unmap the regblock regardless of probe success */
1200
+
cxl_mem_unmap_regblock(cxlm, base);
1201
+
1202
+
if (ret)
1203
+
goto free_maps;
1204
+
}
1205
+
1206
+
pci_release_mem_regions(pdev);
1207
+
1208
+
list_for_each_entry(map, ®ister_maps, list) {
1209
+
ret = cxl_map_regs(cxlm, map);
1210
+
if (ret)
1211
+
goto free_maps;
1212
+
}
1213
+
1214
+
free_maps:
1215
+
list_for_each_entry_safe(map, n, ®ister_maps, list) {
1216
+
list_del(&map->list);
1217
+
kfree(map);
1218
+
}
1219
+
1220
+
return ret;
997
1221
}
998
1222
999
1223
static struct cxl_memdev *to_cxl_memdev(struct device *dev)
···
1200
1094
}
1201
1095
static DEVICE_ATTR_RO(payload_max);
1202
1096
1097
+
static ssize_t label_storage_size_show(struct device *dev,
1098
+
struct device_attribute *attr, char *buf)
1099
+
{
1100
+
struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
1101
+
struct cxl_mem *cxlm = cxlmd->cxlm;
1102
+
1103
+
return sysfs_emit(buf, "%zu\n", cxlm->lsa_size);
1104
+
}
1105
+
static DEVICE_ATTR_RO(label_storage_size);
1106
+
1203
1107
static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr,
1204
1108
char *buf)
1205
1109
{
···
1239
1123
static struct attribute *cxl_memdev_attributes[] = {
1240
1124
&dev_attr_firmware_version.attr,
1241
1125
&dev_attr_payload_max.attr,
1126
+
&dev_attr_label_storage_size.attr,
1242
1127
NULL,
1243
1128
};
1244
1129
···
1332
1215
return ERR_PTR(rc);
1333
1216
}
1334
1217
1335
-
static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
1218
+
static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
1219
+
struct cxl_mem *cxlm)
1336
1220
{
1337
1221
struct cxl_memdev *cxlmd;
1338
1222
struct device *dev;
···
1342
1224
1343
1225
cxlmd = cxl_memdev_alloc(cxlm);
1344
1226
if (IS_ERR(cxlmd))
1345
-
return PTR_ERR(cxlmd);
1227
+
return cxlmd;
1346
1228
1347
1229
dev = &cxlmd->dev;
1348
1230
rc = dev_set_name(dev, "mem%d", cxlmd->id);
···
1360
1242
if (rc)
1361
1243
goto err;
1362
1244
1363
-
return devm_add_action_or_reset(dev->parent, cxl_memdev_unregister,
1364
-
cxlmd);
1245
+
rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
1246
+
if (rc)
1247
+
return ERR_PTR(rc);
1248
+
return cxlmd;
1365
1249
1366
1250
err:
1367
1251
/*
···
1372
1252
*/
1373
1253
cxl_memdev_shutdown(cxlmd);
1374
1254
put_device(dev);
1375
-
return rc;
1255
+
return ERR_PTR(rc);
1376
1256
}
1377
1257
1378
1258
static int cxl_xfer_log(struct cxl_mem *cxlm, uuid_t *uuid, u32 size, u8 *out)
···
1575
1455
cxlm->pmem_range.end =
1576
1456
le64_to_cpu(id.persistent_capacity) * SZ_256M - 1;
1577
1457
1458
+
cxlm->lsa_size = le32_to_cpu(id.lsa_size);
1578
1459
memcpy(cxlm->firmware_version, id.fw_revision, sizeof(id.fw_revision));
1579
1460
1580
1461
return 0;
···
1583
1462
1584
1463
static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
1585
1464
{
1586
-
struct device *dev = &pdev->dev;
1587
-
struct cxl_mem *cxlm = NULL;
1588
-
u32 regloc_size, regblocks;
1589
-
int rc, regloc, i;
1465
+
struct cxl_memdev *cxlmd;
1466
+
struct cxl_mem *cxlm;
1467
+
int rc;
1590
1468
1591
1469
rc = pcim_enable_device(pdev);
1592
1470
if (rc)
1593
1471
return rc;
1594
1472
1595
-
regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC_OFFSET);
1596
-
if (!regloc) {
1597
-
dev_err(dev, "register location dvsec not found\n");
1598
-
return -ENXIO;
1599
-
}
1600
-
1601
-
/* Get the size of the Register Locator DVSEC */
1602
-
pci_read_config_dword(pdev, regloc + PCI_DVSEC_HEADER1, ®loc_size);
1603
-
regloc_size = FIELD_GET(PCI_DVSEC_HEADER1_LENGTH_MASK, regloc_size);
1604
-
1605
-
regloc += PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET;
1606
-
regblocks = (regloc_size - PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET) / 8;
1607
-
1608
-
for (i = 0; i < regblocks; i++, regloc += 8) {
1609
-
u32 reg_lo, reg_hi;
1610
-
u8 reg_type;
1611
-
1612
-
/* "register low and high" contain other bits */
1613
-
pci_read_config_dword(pdev, regloc, ®_lo);
1614
-
pci_read_config_dword(pdev, regloc + 4, ®_hi);
1615
-
1616
-
reg_type = FIELD_GET(CXL_REGLOC_RBI_MASK, reg_lo);
1617
-
1618
-
if (reg_type == CXL_REGLOC_RBI_MEMDEV) {
1619
-
cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
1620
-
break;
1621
-
}
1622
-
}
1623
-
1624
-
if (!cxlm)
1625
-
return -ENODEV;
1473
+
cxlm = cxl_mem_create(pdev);
1474
+
if (IS_ERR(cxlm))
1475
+
return PTR_ERR(cxlm);
1626
1476
1627
1477
rc = cxl_mem_setup_regs(cxlm);
1628
1478
if (rc)
···
1611
1519
if (rc)
1612
1520
return rc;
1613
1521
1614
-
return cxl_mem_add_memdev(cxlm);
1522
+
cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm);
1523
+
if (IS_ERR(cxlmd))
1524
+
return PTR_ERR(cxlmd);
1525
+
1526
+
if (range_len(&cxlm->pmem_range) && IS_ENABLED(CONFIG_CXL_PMEM))
1527
+
rc = devm_cxl_add_nvdimm(&pdev->dev, cxlmd);
1528
+
1529
+
return rc;
1615
1530
}
1616
1531
1617
1532
static const struct pci_device_id cxl_mem_pci_tbl[] = {
···
1642
1543
struct dentry *mbox_debugfs;
1643
1544
dev_t devt;
1644
1545
int rc;
1546
+
1547
+
/* Double check the anonymous union trickery in struct cxl_regs */
1548
+
BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
1549
+
offsetof(struct cxl_regs, device_regs.memdev));
1645
1550
1646
1551
rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
1647
1552
if (rc)
+81
drivers/cxl/mem.h
+81
drivers/cxl/mem.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/* Copyright(c) 2020-2021 Intel Corporation. */
3
+
#ifndef __CXL_MEM_H__
4
+
#define __CXL_MEM_H__
5
+
#include <linux/cdev.h>
6
+
#include "cxl.h"
7
+
8
+
/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
9
+
#define CXLMDEV_STATUS_OFFSET 0x0
10
+
#define CXLMDEV_DEV_FATAL BIT(0)
11
+
#define CXLMDEV_FW_HALT BIT(1)
12
+
#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
13
+
#define CXLMDEV_MS_NOT_READY 0
14
+
#define CXLMDEV_MS_READY 1
15
+
#define CXLMDEV_MS_ERROR 2
16
+
#define CXLMDEV_MS_DISABLED 3
17
+
#define CXLMDEV_READY(status) \
18
+
(FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) == \
19
+
CXLMDEV_MS_READY)
20
+
#define CXLMDEV_MBOX_IF_READY BIT(4)
21
+
#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
22
+
#define CXLMDEV_RESET_NEEDED_NOT 0
23
+
#define CXLMDEV_RESET_NEEDED_COLD 1
24
+
#define CXLMDEV_RESET_NEEDED_WARM 2
25
+
#define CXLMDEV_RESET_NEEDED_HOT 3
26
+
#define CXLMDEV_RESET_NEEDED_CXL 4
27
+
#define CXLMDEV_RESET_NEEDED(status) \
28
+
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
29
+
CXLMDEV_RESET_NEEDED_NOT)
30
+
31
+
/*
32
+
* An entire PCI topology full of devices should be enough for any
33
+
* config
34
+
*/
35
+
#define CXL_MEM_MAX_DEVS 65536
36
+
37
+
/**
38
+
* struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
39
+
* @dev: driver core device object
40
+
* @cdev: char dev core object for ioctl operations
41
+
* @cxlm: pointer to the parent device driver data
42
+
* @id: id number of this memdev instance.
43
+
*/
44
+
struct cxl_memdev {
45
+
struct device dev;
46
+
struct cdev cdev;
47
+
struct cxl_mem *cxlm;
48
+
int id;
49
+
};
50
+
51
+
/**
52
+
* struct cxl_mem - A CXL memory device
53
+
* @pdev: The PCI device associated with this CXL device.
54
+
* @cxlmd: Logical memory device chardev / interface
55
+
* @regs: Parsed register blocks
56
+
* @payload_size: Size of space for payload
57
+
* (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
58
+
* @lsa_size: Size of Label Storage Area
59
+
* (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
60
+
* @mbox_mutex: Mutex to synchronize mailbox access.
61
+
* @firmware_version: Firmware version for the memory device.
62
+
* @enabled_cmds: Hardware commands found enabled in CEL.
63
+
* @pmem_range: Persistent memory capacity information.
64
+
* @ram_range: Volatile memory capacity information.
65
+
*/
66
+
struct cxl_mem {
67
+
struct pci_dev *pdev;
68
+
struct cxl_memdev *cxlmd;
69
+
70
+
struct cxl_regs regs;
71
+
72
+
size_t payload_size;
73
+
size_t lsa_size;
74
+
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
75
+
char firmware_version[0x10];
76
+
unsigned long *enabled_cmds;
77
+
78
+
struct range pmem_range;
79
+
struct range ram_range;
80
+
};
81
+
#endif /* __CXL_MEM_H__ */
+1
-1
drivers/cxl/pci.h
+1
-1
drivers/cxl/pci.h
+230
drivers/cxl/pmem.c
+230
drivers/cxl/pmem.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
3
+
#include <linux/libnvdimm.h>
4
+
#include <linux/device.h>
5
+
#include <linux/module.h>
6
+
#include <linux/ndctl.h>
7
+
#include <linux/async.h>
8
+
#include <linux/slab.h>
9
+
#include "mem.h"
10
+
#include "cxl.h"
11
+
12
+
/*
13
+
* Ordered workqueue for cxl nvdimm device arrival and departure
14
+
* to coordinate bus rescans when a bridge arrives and trigger remove
15
+
* operations when the bridge is removed.
16
+
*/
17
+
static struct workqueue_struct *cxl_pmem_wq;
18
+
19
+
static void unregister_nvdimm(void *nvdimm)
20
+
{
21
+
nvdimm_delete(nvdimm);
22
+
}
23
+
24
+
static int match_nvdimm_bridge(struct device *dev, const void *data)
25
+
{
26
+
return strcmp(dev_name(dev), "nvdimm-bridge") == 0;
27
+
}
28
+
29
+
static struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(void)
30
+
{
31
+
struct device *dev;
32
+
33
+
dev = bus_find_device(&cxl_bus_type, NULL, NULL, match_nvdimm_bridge);
34
+
if (!dev)
35
+
return NULL;
36
+
return to_cxl_nvdimm_bridge(dev);
37
+
}
38
+
39
+
static int cxl_nvdimm_probe(struct device *dev)
40
+
{
41
+
struct cxl_nvdimm *cxl_nvd = to_cxl_nvdimm(dev);
42
+
struct cxl_nvdimm_bridge *cxl_nvb;
43
+
unsigned long flags = 0;
44
+
struct nvdimm *nvdimm;
45
+
int rc = -ENXIO;
46
+
47
+
cxl_nvb = cxl_find_nvdimm_bridge();
48
+
if (!cxl_nvb)
49
+
return -ENXIO;
50
+
51
+
device_lock(&cxl_nvb->dev);
52
+
if (!cxl_nvb->nvdimm_bus)
53
+
goto out;
54
+
55
+
set_bit(NDD_LABELING, &flags);
56
+
nvdimm = nvdimm_create(cxl_nvb->nvdimm_bus, cxl_nvd, NULL, flags, 0, 0,
57
+
NULL);
58
+
if (!nvdimm)
59
+
goto out;
60
+
61
+
rc = devm_add_action_or_reset(dev, unregister_nvdimm, nvdimm);
62
+
out:
63
+
device_unlock(&cxl_nvb->dev);
64
+
put_device(&cxl_nvb->dev);
65
+
66
+
return rc;
67
+
}
68
+
69
+
static struct cxl_driver cxl_nvdimm_driver = {
70
+
.name = "cxl_nvdimm",
71
+
.probe = cxl_nvdimm_probe,
72
+
.id = CXL_DEVICE_NVDIMM,
73
+
};
74
+
75
+
static int cxl_pmem_ctl(struct nvdimm_bus_descriptor *nd_desc,
76
+
struct nvdimm *nvdimm, unsigned int cmd, void *buf,
77
+
unsigned int buf_len, int *cmd_rc)
78
+
{
79
+
return -ENOTTY;
80
+
}
81
+
82
+
static bool online_nvdimm_bus(struct cxl_nvdimm_bridge *cxl_nvb)
83
+
{
84
+
if (cxl_nvb->nvdimm_bus)
85
+
return true;
86
+
cxl_nvb->nvdimm_bus =
87
+
nvdimm_bus_register(&cxl_nvb->dev, &cxl_nvb->nd_desc);
88
+
return cxl_nvb->nvdimm_bus != NULL;
89
+
}
90
+
91
+
static int cxl_nvdimm_release_driver(struct device *dev, void *data)
92
+
{
93
+
if (!is_cxl_nvdimm(dev))
94
+
return 0;
95
+
device_release_driver(dev);
96
+
return 0;
97
+
}
98
+
99
+
static void offline_nvdimm_bus(struct nvdimm_bus *nvdimm_bus)
100
+
{
101
+
if (!nvdimm_bus)
102
+
return;
103
+
104
+
/*
105
+
* Set the state of cxl_nvdimm devices to unbound / idle before
106
+
* nvdimm_bus_unregister() rips the nvdimm objects out from
107
+
* underneath them.
108
+
*/
109
+
bus_for_each_dev(&cxl_bus_type, NULL, NULL, cxl_nvdimm_release_driver);
110
+
nvdimm_bus_unregister(nvdimm_bus);
111
+
}
112
+
113
+
static void cxl_nvb_update_state(struct work_struct *work)
114
+
{
115
+
struct cxl_nvdimm_bridge *cxl_nvb =
116
+
container_of(work, typeof(*cxl_nvb), state_work);
117
+
struct nvdimm_bus *victim_bus = NULL;
118
+
bool release = false, rescan = false;
119
+
120
+
device_lock(&cxl_nvb->dev);
121
+
switch (cxl_nvb->state) {
122
+
case CXL_NVB_ONLINE:
123
+
if (!online_nvdimm_bus(cxl_nvb)) {
124
+
dev_err(&cxl_nvb->dev,
125
+
"failed to establish nvdimm bus\n");
126
+
release = true;
127
+
} else
128
+
rescan = true;
129
+
break;
130
+
case CXL_NVB_OFFLINE:
131
+
case CXL_NVB_DEAD:
132
+
victim_bus = cxl_nvb->nvdimm_bus;
133
+
cxl_nvb->nvdimm_bus = NULL;
134
+
break;
135
+
default:
136
+
break;
137
+
}
138
+
device_unlock(&cxl_nvb->dev);
139
+
140
+
if (release)
141
+
device_release_driver(&cxl_nvb->dev);
142
+
if (rescan) {
143
+
int rc = bus_rescan_devices(&cxl_bus_type);
144
+
145
+
dev_dbg(&cxl_nvb->dev, "rescan: %d\n", rc);
146
+
}
147
+
offline_nvdimm_bus(victim_bus);
148
+
149
+
put_device(&cxl_nvb->dev);
150
+
}
151
+
152
+
static void cxl_nvdimm_bridge_remove(struct device *dev)
153
+
{
154
+
struct cxl_nvdimm_bridge *cxl_nvb = to_cxl_nvdimm_bridge(dev);
155
+
156
+
if (cxl_nvb->state == CXL_NVB_ONLINE)
157
+
cxl_nvb->state = CXL_NVB_OFFLINE;
158
+
if (queue_work(cxl_pmem_wq, &cxl_nvb->state_work))
159
+
get_device(&cxl_nvb->dev);
160
+
}
161
+
162
+
static int cxl_nvdimm_bridge_probe(struct device *dev)
163
+
{
164
+
struct cxl_nvdimm_bridge *cxl_nvb = to_cxl_nvdimm_bridge(dev);
165
+
166
+
if (cxl_nvb->state == CXL_NVB_DEAD)
167
+
return -ENXIO;
168
+
169
+
if (cxl_nvb->state == CXL_NVB_NEW) {
170
+
cxl_nvb->nd_desc = (struct nvdimm_bus_descriptor) {
171
+
.provider_name = "CXL",
172
+
.module = THIS_MODULE,
173
+
.ndctl = cxl_pmem_ctl,
174
+
};
175
+
176
+
INIT_WORK(&cxl_nvb->state_work, cxl_nvb_update_state);
177
+
}
178
+
179
+
cxl_nvb->state = CXL_NVB_ONLINE;
180
+
if (queue_work(cxl_pmem_wq, &cxl_nvb->state_work))
181
+
get_device(&cxl_nvb->dev);
182
+
183
+
return 0;
184
+
}
185
+
186
+
static struct cxl_driver cxl_nvdimm_bridge_driver = {
187
+
.name = "cxl_nvdimm_bridge",
188
+
.probe = cxl_nvdimm_bridge_probe,
189
+
.remove = cxl_nvdimm_bridge_remove,
190
+
.id = CXL_DEVICE_NVDIMM_BRIDGE,
191
+
};
192
+
193
+
static __init int cxl_pmem_init(void)
194
+
{
195
+
int rc;
196
+
197
+
cxl_pmem_wq = alloc_ordered_workqueue("cxl_pmem", 0);
198
+
if (!cxl_pmem_wq)
199
+
return -ENXIO;
200
+
201
+
rc = cxl_driver_register(&cxl_nvdimm_bridge_driver);
202
+
if (rc)
203
+
goto err_bridge;
204
+
205
+
rc = cxl_driver_register(&cxl_nvdimm_driver);
206
+
if (rc)
207
+
goto err_nvdimm;
208
+
209
+
return 0;
210
+
211
+
err_nvdimm:
212
+
cxl_driver_unregister(&cxl_nvdimm_bridge_driver);
213
+
err_bridge:
214
+
destroy_workqueue(cxl_pmem_wq);
215
+
return rc;
216
+
}
217
+
218
+
static __exit void cxl_pmem_exit(void)
219
+
{
220
+
cxl_driver_unregister(&cxl_nvdimm_driver);
221
+
cxl_driver_unregister(&cxl_nvdimm_bridge_driver);
222
+
destroy_workqueue(cxl_pmem_wq);
223
+
}
224
+
225
+
MODULE_LICENSE("GPL v2");
226
+
module_init(cxl_pmem_init);
227
+
module_exit(cxl_pmem_exit);
228
+
MODULE_IMPORT_NS(CXL);
229
+
MODULE_ALIAS_CXL(CXL_DEVICE_NVDIMM_BRIDGE);
230
+
MODULE_ALIAS_CXL(CXL_DEVICE_NVDIMM);
+41
-23
drivers/nvdimm/bus.c
+41
-23
drivers/nvdimm/bus.c
···
363
363
nvdimm_bus->dev.groups = nd_desc->attr_groups;
364
364
nvdimm_bus->dev.bus = &nvdimm_bus_type;
365
365
nvdimm_bus->dev.of_node = nd_desc->of_node;
366
-
dev_set_name(&nvdimm_bus->dev, "ndbus%d", nvdimm_bus->id);
367
-
rc = device_register(&nvdimm_bus->dev);
366
+
device_initialize(&nvdimm_bus->dev);
367
+
device_set_pm_not_required(&nvdimm_bus->dev);
368
+
rc = dev_set_name(&nvdimm_bus->dev, "ndbus%d", nvdimm_bus->id);
369
+
if (rc)
370
+
goto err;
371
+
372
+
rc = device_add(&nvdimm_bus->dev);
368
373
if (rc) {
369
374
dev_dbg(&nvdimm_bus->dev, "registration failed: %d\n", rc);
370
375
goto err;
···
401
396
if (dev->class)
402
397
return 0;
403
398
404
-
if (is_nvdimm(dev)) {
405
-
struct nvdimm *nvdimm = to_nvdimm(dev);
406
-
bool dev_put = false;
407
-
408
-
/* We are shutting down. Make state frozen artificially. */
409
-
nvdimm_bus_lock(dev);
410
-
set_bit(NVDIMM_SECURITY_FROZEN, &nvdimm->sec.flags);
411
-
if (test_and_clear_bit(NDD_WORK_PENDING, &nvdimm->flags))
412
-
dev_put = true;
413
-
nvdimm_bus_unlock(dev);
414
-
cancel_delayed_work_sync(&nvdimm->dwork);
415
-
if (dev_put)
416
-
put_device(dev);
417
-
}
418
-
nd_device_unregister(dev, ND_SYNC);
399
+
if (is_nvdimm(dev))
400
+
nvdimm_delete(to_nvdimm(dev));
401
+
else
402
+
nd_device_unregister(dev, ND_SYNC);
419
403
420
404
return 0;
421
405
}
···
530
536
set_dev_node(dev, to_nd_region(dev)->numa_node);
531
537
532
538
dev->bus = &nvdimm_bus_type;
539
+
device_set_pm_not_required(dev);
533
540
if (dev->parent) {
534
541
get_device(dev->parent);
535
542
if (dev_to_node(dev) == NUMA_NO_NODE)
···
723
728
.is_visible = nd_numa_attr_visible,
724
729
};
725
730
731
+
static void ndctl_release(struct device *dev)
732
+
{
733
+
kfree(dev);
734
+
}
735
+
726
736
int nvdimm_bus_create_ndctl(struct nvdimm_bus *nvdimm_bus)
727
737
{
728
738
dev_t devt = MKDEV(nvdimm_bus_major, nvdimm_bus->id);
729
739
struct device *dev;
740
+
int rc;
730
741
731
-
dev = device_create(nd_class, &nvdimm_bus->dev, devt, nvdimm_bus,
732
-
"ndctl%d", nvdimm_bus->id);
742
+
dev = kzalloc(sizeof(*dev), GFP_KERNEL);
743
+
if (!dev)
744
+
return -ENOMEM;
745
+
device_initialize(dev);
746
+
device_set_pm_not_required(dev);
747
+
dev->class = nd_class;
748
+
dev->parent = &nvdimm_bus->dev;
749
+
dev->devt = devt;
750
+
dev->release = ndctl_release;
751
+
rc = dev_set_name(dev, "ndctl%d", nvdimm_bus->id);
752
+
if (rc)
753
+
goto err;
733
754
734
-
if (IS_ERR(dev))
735
-
dev_dbg(&nvdimm_bus->dev, "failed to register ndctl%d: %ld\n",
736
-
nvdimm_bus->id, PTR_ERR(dev));
737
-
return PTR_ERR_OR_ZERO(dev);
755
+
rc = device_add(dev);
756
+
if (rc) {
757
+
dev_dbg(&nvdimm_bus->dev, "failed to register ndctl%d: %d\n",
758
+
nvdimm_bus->id, rc);
759
+
goto err;
760
+
}
761
+
return 0;
762
+
763
+
err:
764
+
put_device(dev);
765
+
return rc;
738
766
}
739
767
740
768
void nvdimm_bus_destroy_ndctl(struct nvdimm_bus *nvdimm_bus)
+18
drivers/nvdimm/dimm_devs.c
+18
drivers/nvdimm/dimm_devs.c
···
642
642
}
643
643
EXPORT_SYMBOL_GPL(__nvdimm_create);
644
644
645
+
void nvdimm_delete(struct nvdimm *nvdimm)
646
+
{
647
+
struct device *dev = &nvdimm->dev;
648
+
bool dev_put = false;
649
+
650
+
/* We are shutting down. Make state frozen artificially. */
651
+
nvdimm_bus_lock(dev);
652
+
set_bit(NVDIMM_SECURITY_FROZEN, &nvdimm->sec.flags);
653
+
if (test_and_clear_bit(NDD_WORK_PENDING, &nvdimm->flags))
654
+
dev_put = true;
655
+
nvdimm_bus_unlock(dev);
656
+
cancel_delayed_work_sync(&nvdimm->dwork);
657
+
if (dev_put)
658
+
put_device(dev);
659
+
nd_device_unregister(dev, ND_SYNC);
660
+
}
661
+
EXPORT_SYMBOL_GPL(nvdimm_delete);
662
+
645
663
static void shutdown_security_notify(void *data)
646
664
{
647
665
struct nvdimm *nvdimm = data;
+1
include/linux/libnvdimm.h
+1
include/linux/libnvdimm.h
···
278
278
return __nvdimm_create(nvdimm_bus, provider_data, groups, flags,
279
279
cmd_mask, num_flush, flush_wpq, NULL, NULL, NULL);
280
280
}
281
+
void nvdimm_delete(struct nvdimm *nvdimm);
281
282
282
283
const struct nd_cmd_desc *nd_cmd_dimm_desc(int cmd);
283
284
const struct nd_cmd_desc *nd_cmd_bus_desc(int cmd);
+12
include/uapi/linux/cxl_mem.h
+12
include/uapi/linux/cxl_mem.h
···
29
29
___C(GET_LSA, "Get Label Storage Area"), \
30
30
___C(GET_HEALTH_INFO, "Get Health Info"), \
31
31
___C(GET_LOG, "Get Log"), \
32
+
___C(SET_PARTITION_INFO, "Set Partition Information"), \
33
+
___C(SET_LSA, "Set Label Storage Area"), \
34
+
___C(GET_ALERT_CONFIG, "Get Alert Configuration"), \
35
+
___C(SET_ALERT_CONFIG, "Set Alert Configuration"), \
36
+
___C(GET_SHUTDOWN_STATE, "Get Shutdown State"), \
37
+
___C(SET_SHUTDOWN_STATE, "Set Shutdown State"), \
38
+
___C(GET_POISON, "Get Poison List"), \
39
+
___C(INJECT_POISON, "Inject Poison"), \
40
+
___C(CLEAR_POISON, "Clear Poison"), \
41
+
___C(GET_SCAN_MEDIA_CAPS, "Get Scan Media Capabilities"), \
42
+
___C(SCAN_MEDIA, "Scan Media"), \
43
+
___C(GET_SCAN_MEDIA, "Get Scan Media Results"), \
32
44
___C(MAX, "invalid / last command")
33
45
34
46
#define ___C(a, b) CXL_MEM_COMMAND_ID_##a