Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'cxl-for-7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl
Pull CXL updates from Dave Jiang:
- Introduce cxl_memdev_attach and pave way for soft reserved handling,
type2 accelerator enabling, and LSA 2.0 enabling. All these series
require the endpoint driver to settle before continuing the memdev
driver probe.
- Address CXL port error protocol handling and reporting.
The large patch series was split into three parts. The first two
parts are included here with the final part coming later.
The first part consists of a series of code refactoring to PCI AER
sub-system that addresses CXL and also CXL RAS code to prepare for
port error handling.
The second part refactors the CXL code to move management of
component registers to cxl_port objects to allow all CXL AER errors
to be handled through the cxl_port hierarchy.
- Provide AMD Zen5 platform address translation for CXL using ACPI
PRMT. This includes a conventions document to explain why this is
needed and how it's implemented.
- Misc CXL patches of fixes, cleanups, and updates. Including CXL
address translation for unaligned MOD3 regions.
[ TLA service: CXL is "Compute Express Link" ]
* tag 'cxl-for-7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl: (59 commits)
cxl: Disable HPA/SPA translation handlers for Normalized Addressing
cxl/region: Factor out code into cxl_region_setup_poison()
cxl/atl: Lock decoders that need address translation
cxl: Enable AMD Zen5 address translation using ACPI PRMT
cxl/acpi: Prepare use of EFI runtime services
cxl: Introduce callback for HPA address ranges translation
cxl/region: Use region data to get the root decoder
cxl/region: Add @hpa_range argument to function cxl_calc_interleave_pos()
cxl/region: Separate region parameter setup and region construction
cxl: Simplify cxl_root_ops allocation and handling
cxl/region: Store HPA range in struct cxl_region
cxl/region: Store root decoder in struct cxl_region
cxl/region: Rename misleading variable name @hpa to @hpa_range
Documentation/driver-api/cxl: ACPI PRM Address Translation Support and AMD Zen5 enablement
cxl, doc: Moving conventions in separate files
cxl, doc: Remove isonum.txt inclusion
cxl/port: Unify endpoint and switch port lookup
cxl/port: Move endpoint component register management to cxl_port
cxl/port: Map Port RAS registers
cxl/port: Move dport RAS setup to dport add time
...
+2528
-1246
+7
-171
Documentation/driver-api/cxl/conventions.rst
+7
-171
Documentation/driver-api/cxl/conventions.rst
···
1
1
.. SPDX-License-Identifier: GPL-2.0
2
-
.. include:: <isonum.txt>
3
2
4
-
=======================================
5
3
Compute Express Link: Linux Conventions
6
-
=======================================
4
+
#######################################
7
5
8
6
There exists shipping platforms that bend or break CXL specification
9
7
expectations. Record the details and the rationale for those deviations.
···
9
11
and tradeoffs such that multiple platform implementations can follow the
10
12
same convention.
11
13
12
-
<(template) Title>
13
-
==================
14
+
.. toctree::
15
+
:maxdepth: 1
16
+
:caption: Contents
14
17
15
-
Document
16
-
--------
17
-
CXL Revision <rev>, Version <ver>
18
-
19
-
License
20
-
-------
21
-
SPDX-License Identifier: CC-BY-4.0
22
-
23
-
Creator/Contributors
24
-
--------------------
25
-
26
-
Summary of the Change
27
-
---------------------
28
-
29
-
<Detail the conflict with the specification and where available the
30
-
assumptions and tradeoffs taken by the hardware platform.>
31
-
32
-
33
-
Benefits of the Change
34
-
----------------------
35
-
36
-
<Detail what happens if platforms and Linux do not adopt this
37
-
convention.>
38
-
39
-
References
40
-
----------
41
-
42
-
Detailed Description of the Change
43
-
----------------------------------
44
-
45
-
<Propose spec language that corrects the conflict.>
46
-
47
-
48
-
Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
49
-
============================================================================
50
-
51
-
Document
52
-
--------
53
-
54
-
CXL Revision 3.2, Version 1.0
55
-
56
-
License
57
-
-------
58
-
59
-
SPDX-License Identifier: CC-BY-4.0
60
-
61
-
Creator/Contributors
62
-
--------------------
63
-
64
-
- Fabio M. De Francesco, Intel
65
-
- Dan J. Williams, Intel
66
-
- Mahesh Natu, Intel
67
-
68
-
Summary of the Change
69
-
---------------------
70
-
71
-
According to the current Compute Express Link (CXL) Specifications (Revision
72
-
3.2, Version 1.0), the CXL Fixed Memory Window Structure (CFMWS) describes zero
73
-
or more Host Physical Address (HPA) windows associated with each CXL Host
74
-
Bridge. Each window represents a contiguous HPA range that may be interleaved
75
-
across one or more targets, including CXL Host Bridges. Each window has a set
76
-
of restrictions that govern its usage. It is the Operating System-directed
77
-
configuration and Power Management (OSPM) responsibility to utilize each window
78
-
for the specified use.
79
-
80
-
Table 9-22 of the current CXL Specifications states that the Window Size field
81
-
contains the total number of consecutive bytes of HPA this window describes.
82
-
This value must be a multiple of the Number of Interleave Ways (NIW) * 256 MB.
83
-
84
-
Platform Firmware (BIOS) might reserve physical addresses below 4 GB where a
85
-
memory gap such as the Low Memory Hole for PCIe MMIO may exist. In such cases,
86
-
the CFMWS Range Size may not adhere to the NIW * 256 MB rule.
87
-
88
-
The HPA represents the actual physical memory address space that the CXL devices
89
-
can decode and respond to, while the System Physical Address (SPA), a related
90
-
but distinct concept, represents the system-visible address space that users can
91
-
direct transaction to and so it excludes reserved regions.
92
-
93
-
BIOS publishes CFMWS to communicate the active SPA ranges that, on platforms
94
-
with LMH's, map to a strict subset of the HPA. The SPA range trims out the hole,
95
-
resulting in lost capacity in the Endpoints with no SPA to map to that part of
96
-
the HPA range that intersects the hole.
97
-
98
-
E.g, an x86 platform with two CFMWS and an LMH starting at 2 GB:
99
-
100
-
+--------+------------+-------------------+------------------+-------------------+------+
101
-
| Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways |
102
-
+========+============+===================+==================+===================+======+
103
-
| 0 | 0 GB | 2 GB | 0 GB | 3 GB | 12 |
104
-
+--------+------------+-------------------+------------------+-------------------+------+
105
-
| 1 | 4 GB | NIW*256MB Aligned | 4 GB | NIW*256MB Aligned | 12 |
106
-
+--------+------------+-------------------+------------------+-------------------+------+
107
-
108
-
HDM decoder base and HDM decoder size represent all the 12 Endpoint Decoders of
109
-
a 12 ways region and all the intermediate Switch Decoders. They are configured
110
-
by the BIOS according to the NIW * 256MB rule, resulting in a HPA range size of
111
-
3GB. Instead, the CFMWS Base and CFMWS Size are used to configure the Root
112
-
Decoder HPA range that results smaller (2GB) than that of the Switch and
113
-
Endpoint Decoders in the hierarchy (3GB).
114
-
115
-
This creates 2 issues which lead to a failure to construct a region:
116
-
117
-
1) A mismatch in region size between root and any HDM decoder. The root decoders
118
-
will always be smaller due to the trim.
119
-
120
-
2) The trim causes the root decoder to violate the (NIW * 256MB) rule.
121
-
122
-
This change allows a region with a base address of 0GB to bypass these checks to
123
-
allow for region creation with the trimmed root decoder address range.
124
-
125
-
This change does not allow for any other arbitrary region to violate these
126
-
checks - it is intended exclusively to enable x86 platforms which map CXL memory
127
-
under 4GB.
128
-
129
-
Despite the HDM decoders covering the PCIE hole HPA region, it is expected that
130
-
the platform will never route address accesses to the CXL complex because the
131
-
root decoder only covers the trimmed region (which excludes this). This is
132
-
outside the ability of Linux to enforce.
133
-
134
-
On the example platform, only the first 2GB will be potentially usable, but
135
-
Linux, aiming to adhere to the current specifications, fails to construct
136
-
Regions and attach Endpoint and intermediate Switch Decoders to them.
137
-
138
-
There are several points of failure that due to the expectation that the Root
139
-
Decoder HPA size, that is equal to the CFMWS from which it is configured, has
140
-
to be greater or equal to the matching Switch and Endpoint HDM Decoders.
141
-
142
-
In order to succeed with construction and attachment, Linux must construct a
143
-
Region with Root Decoder HPA range size, and then attach to that all the
144
-
intermediate Switch Decoders and Endpoint Decoders that belong to the hierarchy
145
-
regardless of their range sizes.
146
-
147
-
Benefits of the Change
148
-
----------------------
149
-
150
-
Without the change, the OSPM wouldn't match intermediate Switch and Endpoint
151
-
Decoders with Root Decoders configured with CFMWS HPA sizes that don't align
152
-
with the NIW * 256MB constraint, and so it leads to lost memdev capacity.
153
-
154
-
This change allows the OSPM to construct Regions and attach intermediate Switch
155
-
and Endpoint Decoders to them, so that the addressable part of the memory
156
-
devices total capacity is made available to the users.
157
-
158
-
References
159
-
----------
160
-
161
-
Compute Express Link Specification Revision 3.2, Version 1.0
162
-
<https://www.computeexpresslink.org/>
163
-
164
-
Detailed Description of the Change
165
-
----------------------------------
166
-
167
-
The description of the Window Size field in table 9-22 needs to account for
168
-
platforms with Low Memory Holes, where SPA ranges might be subsets of the
169
-
endpoints HPA. Therefore, it has to be changed to the following:
170
-
171
-
"The total number of consecutive bytes of HPA this window represents. This value
172
-
shall be a multiple of NIW * 256 MB.
173
-
174
-
On platforms that reserve physical addresses below 4 GB, such as the Low Memory
175
-
Hole for PCIe MMIO on x86, an instance of CFMWS whose Base HPA range is 0 might
176
-
have a size that doesn't align with the NIW * 256 MB constraint.
177
-
178
-
Note that the matching intermediate Switch Decoders and the Endpoint Decoders
179
-
HPA range sizes must still align to the above-mentioned rule, but the memory
180
-
capacity that exceeds the CFMWS window size won't be accessible.".
18
+
conventions/cxl-lmh.rst
19
+
conventions/cxl-atl.rst
20
+
conventions/template.rst
+304
Documentation/driver-api/cxl/conventions/cxl-atl.rst
+304
Documentation/driver-api/cxl/conventions/cxl-atl.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
ACPI PRM CXL Address Translation
4
+
================================
5
+
6
+
Document
7
+
--------
8
+
9
+
CXL Revision 3.2, Version 1.0
10
+
11
+
License
12
+
-------
13
+
14
+
SPDX-License Identifier: CC-BY-4.0
15
+
16
+
Creator/Contributors
17
+
--------------------
18
+
19
+
- Robert Richter, AMD et al.
20
+
21
+
Summary of the Change
22
+
---------------------
23
+
24
+
The CXL Fixed Memory Window Structures (CFMWS) describe zero or more Host
25
+
Physical Address (HPA) windows associated with one or more CXL Host Bridges.
26
+
Each HPA range of a CXL Host Bridge is represented by a CFMWS entry. An HPA
27
+
range may include addresses currently assigned to CXL.mem devices, or an OS may
28
+
assign ranges from an address window to a device.
29
+
30
+
Host-managed Device Memory is Device-attached memory that is mapped to system
31
+
coherent address space and accessible to the Host using standard write-back
32
+
semantics. The managed address range is configured in the CXL HDM Decoder
33
+
registers of the device. An HDM Decoder in a device is responsible for
34
+
converting HPA into DPA by stripping off specific address bits.
35
+
36
+
CXL devices and CXL bridges use the same HPA space. It is common across all
37
+
components that belong to the same host domain. The view of the address region
38
+
must be consistent on the CXL.mem path between the Host and the Device.
39
+
40
+
This is described in the *CXL 3.2 specification* (Table 1-1, 3.3.1,
41
+
8.2.4.20, 9.13.1, 9.18.1.3). [#cxl-spec-3.2]_
42
+
43
+
Depending on the interconnect architecture of the platform, components attached
44
+
to a host may not share the same host physical address space. Those platforms
45
+
need address translation to convert an HPA between the host and the attached
46
+
component, such as a CXL device. The translation mechanism is host-specific and
47
+
implementation dependent.
48
+
49
+
For example, x86 AMD platforms use a Data Fabric that manages access to physical
50
+
memory. Devices have their own memory space and can be configured to use
51
+
'Normalized addresses' different from System Physical Addresses (SPA). Address
52
+
translation is then needed. For details, see
53
+
:doc:`x86 AMD Address Translation </admin-guide/RAS/address-translation>`.
54
+
55
+
Those AMD platforms provide PRM [#prm-spec]_ handlers in firmware to perform
56
+
various types of address translation, including for CXL endpoints. AMD Zen5
57
+
systems implement the ACPI PRM CXL Address Translation firmware call. The ACPI
58
+
PRM handler has a specific GUID to uniquely identify platforms with support for
59
+
Normalized addressing. This is documented in the *ACPI v6.5 Porting Guide*
60
+
(Address Translation - CXL DPA to System Physical Address). [#amd-ppr-58088]_
61
+
62
+
When in Normalized address mode, HDM decoder address ranges must be configured
63
+
and handled differently. Hardware addresses used in the HDM decoder
64
+
configurations of an endpoint are not SPA and need to be translated from the
65
+
address range of the endpoint to that of the CXL host bridge. This is especially
66
+
important for finding an endpoint's associated CXL Host Bridge and HPA window
67
+
described in the CFMWS. Additionally, the interleave decoding is done by the
68
+
Data Fabric and the endpoint does not perform decoding when converting HPA to
69
+
DPA. Instead, interleaving is switched off for the endpoint (1-way). Finally,
70
+
address translation might also be needed to inspect the endpoint's hardware
71
+
addresses, such as during profiling, tracing, or error handling.
72
+
73
+
For example, with Normalized addressing the HDM decoders could look as follows::
74
+
75
+
-------------------------------
76
+
| Root Decoder (CFMWS) |
77
+
| SPA Range: 0x850000000 |
78
+
| Size: 0x8000000000 (512 GB) |
79
+
| Interleave Ways: 1 |
80
+
-------------------------------
81
+
|
82
+
v
83
+
-------------------------------
84
+
| Host Bridge Decoder (HDM) |
85
+
| SPA Range: 0x850000000 |
86
+
| Size: 0x8000000000 (512 GB) |
87
+
| Interleave Ways: 4 |
88
+
| Targets: endpoint5,8,11,13 |
89
+
| Granularity: 256 |
90
+
-------------------------------
91
+
|
92
+
-----------------------------+------------------------------
93
+
| | | |
94
+
v v v v
95
+
------------------- ------------------- ------------------- -------------------
96
+
| endpoint5 | | endpoint8 | | endpoint11 | | endpoint13 |
97
+
| decoder5.0 | | decoder8.0 | | decoder11.0 | | decoder13.0 |
98
+
| PCIe: | | PCIe: | | PCIe: | | PCIe: |
99
+
| 0000:e2:00.0 | | 0000:e3:00.0 | | 0000:e4:00.0 | | 0000:e1:00.0 |
100
+
| DPA: | | DPA: | | DPA: | | DPA: |
101
+
| Start: 0x0 | | Start: 0x0 | | Start: 0x0 | | Start: 0x0 |
102
+
| Size: | | Size: | | Size: | | Size: |
103
+
| 0x2000000000 | | 0x2000000000 | | 0x2000000000 | | 0x2000000000 |
104
+
| (128 GB) | | (128 GB) | | (128 GB) | | (128 GB) |
105
+
| Interleaving: | | Interleaving: | | Interleaving: | | Interleaving: |
106
+
| Ways: 1 | | Ways: 1 | | Ways: 1 | | Ways: 1 |
107
+
| Gran: 256 | | Gran: 256 | | Gran: 256 | | Gran: 256 |
108
+
------------------- ------------------- ------------------- -------------------
109
+
| | | |
110
+
v v v v
111
+
DPA DPA DPA DPA
112
+
113
+
This shows the representation in sysfs:
114
+
115
+
.. code-block:: none
116
+
117
+
/sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_granularity:256
118
+
/sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_ways:1
119
+
/sys/bus/cxl/devices/endpoint5/decoder5.0/size:0x2000000000
120
+
/sys/bus/cxl/devices/endpoint5/decoder5.0/start:0x0
121
+
/sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_granularity:256
122
+
/sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_ways:1
123
+
/sys/bus/cxl/devices/endpoint8/decoder8.0/size:0x2000000000
124
+
/sys/bus/cxl/devices/endpoint8/decoder8.0/start:0x0
125
+
/sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_granularity:256
126
+
/sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_ways:1
127
+
/sys/bus/cxl/devices/endpoint11/decoder11.0/size:0x2000000000
128
+
/sys/bus/cxl/devices/endpoint11/decoder11.0/start:0x0
129
+
/sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_granularity:256
130
+
/sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_ways:1
131
+
/sys/bus/cxl/devices/endpoint13/decoder13.0/size:0x2000000000
132
+
/sys/bus/cxl/devices/endpoint13/decoder13.0/start:0x0
133
+
134
+
Note that the endpoint interleaving configurations use direct mapping (1-way).
135
+
136
+
With PRM calls, the kernel can determine the following mappings:
137
+
138
+
.. code-block:: none
139
+
140
+
cxl decoder5.0: address mapping found for 0000:e2:00.0 (hpa -> spa):
141
+
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
142
+
cxl decoder8.0: address mapping found for 0000:e3:00.0 (hpa -> spa):
143
+
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
144
+
cxl decoder11.0: address mapping found for 0000:e4:00.0 (hpa -> spa):
145
+
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
146
+
cxl decoder13.0: address mapping found for 0000:e1:00.0 (hpa -> spa):
147
+
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
148
+
149
+
The corresponding CXL host bridge (HDM) decoders and root decoder (CFMWS) match
150
+
the calculated endpoint mappings shown:
151
+
152
+
.. code-block:: none
153
+
154
+
/sys/bus/cxl/devices/port1/decoder1.0/interleave_granularity:256
155
+
/sys/bus/cxl/devices/port1/decoder1.0/interleave_ways:4
156
+
/sys/bus/cxl/devices/port1/decoder1.0/size:0x8000000000
157
+
/sys/bus/cxl/devices/port1/decoder1.0/start:0x850000000
158
+
/sys/bus/cxl/devices/port1/decoder1.0/target_list:0,1,2,3
159
+
/sys/bus/cxl/devices/port1/decoder1.0/target_type:expander
160
+
/sys/bus/cxl/devices/root0/decoder0.0/interleave_granularity:256
161
+
/sys/bus/cxl/devices/root0/decoder0.0/interleave_ways:1
162
+
/sys/bus/cxl/devices/root0/decoder0.0/size:0x8000000000
163
+
/sys/bus/cxl/devices/root0/decoder0.0/start:0x850000000
164
+
/sys/bus/cxl/devices/root0/decoder0.0/target_list:7
165
+
166
+
The following changes to the specification are needed:
167
+
168
+
* Allow a CXL device to be in an HPA space other than the host's address space.
169
+
170
+
* Allow the platform to use implementation-specific address translation when
171
+
crossing memory domains on the CXL.mem path between the host and the device.
172
+
173
+
* Define a PRM handler method for converting device addresses to SPAs.
174
+
175
+
* Specify that the platform shall provide the PRM handler method to the
176
+
Operating System to detect Normalized addressing and for determining Endpoint
177
+
SPA ranges and interleaving configurations.
178
+
179
+
* Add reference to:
180
+
181
+
| Platform Runtime Mechanism Specification, Version 1.1 – November 2020
182
+
| https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf
183
+
184
+
Benefits of the Change
185
+
----------------------
186
+
187
+
Without the change, the Operating System may be unable to determine the memory
188
+
region and Root Decoder for an Endpoint and its corresponding HDM decoder.
189
+
Region creation would fail. Platforms with a different interconnect architecture
190
+
would fail to set up and use CXL.
191
+
192
+
References
193
+
----------
194
+
195
+
.. [#cxl-spec-3.2] Compute Express Link Specification, Revision 3.2, Version 1.0,
196
+
https://www.computeexpresslink.org/
197
+
198
+
.. [#amd-ppr-58088] AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh,
199
+
ACPI v6.5 Porting Guide, Publication # 58088,
200
+
https://www.amd.com/en/search/documentation/hub.html
201
+
202
+
.. [#prm-spec] Platform Runtime Mechanism, Version: 1.1,
203
+
https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf
204
+
205
+
Detailed Description of the Change
206
+
----------------------------------
207
+
208
+
The following describes the necessary changes to the *CXL 3.2 specification*
209
+
[#cxl-spec-3.2]_:
210
+
211
+
Add the following reference to the table:
212
+
213
+
Table 1-2. Reference Documents
214
+
215
+
+----------------------------+-------------------+---------------------------+
216
+
| Document | Chapter Reference | Document No./Location |
217
+
+============================+===================+===========================+
218
+
| Platform Runtime Mechanism | Chapter 8, 9 | https://www.uefi.org/acpi |
219
+
| Version: 1.1 | | |
220
+
+----------------------------+-------------------+---------------------------+
221
+
222
+
Add the following paragraphs to the end of the section:
223
+
224
+
**8.2.4.20 CXL HDM Decoder Capability Structure**
225
+
226
+
"A device may use an HPA space that is not common to other components of the
227
+
host domain. The platform is responsible for address translation when crossing
228
+
HPA spaces. The Operating System must determine the interleaving configuration
229
+
and perform address translation to the HPA ranges of the HDM decoders as needed.
230
+
The translation mechanism is host-specific and implementation dependent.
231
+
232
+
The platform indicates support of independent HPA spaces and the need for
233
+
address translation by providing a Platform Runtime Mechanism (PRM) handler. The
234
+
OS shall use that handler to perform the necessary translations from the DPA
235
+
space to the HPA space. The handler is defined in Section 9.18.4 *PRM Handler
236
+
for CXL DPA to System Physical Address Translation*."
237
+
238
+
Add the following section and sub-section including tables:
239
+
240
+
**9.18.4 PRM Handler for CXL DPA to System Physical Address Translation**
241
+
242
+
"A platform may be configured to use 'Normalized addresses'. Host physical
243
+
address (HPA) spaces are component-specific and differ from system physical
244
+
addresses (SPAs). The endpoint has its own physical address space. All requests
245
+
presented to the device already use Device Physical Addresses (DPAs). The CXL
246
+
endpoint decoders have interleaving disabled (1-way interleaving) and the device
247
+
does not perform HPA decoding to determine a DPA.
248
+
249
+
The platform provides a PRM handler for CXL DPA to System Physical Address
250
+
Translation. The PRM handler translates a Device Physical Address (DPA) to a
251
+
System Physical Address (SPA) for a specified CXL endpoint. In the address space
252
+
of the host, SPA and HPA are equivalent, and the OS shall use this handler to
253
+
determine the HPA that corresponds to a device address, for example when
254
+
configuring HDM decoders on platforms with Normalized addressing. The GUID and
255
+
the parameter buffer format of the handler are specified in section 9.18.4.1. If
256
+
the OS identifies the PRM handler, the platform supports Normalized addressing
257
+
and the OS must perform DPA address translation as needed."
258
+
259
+
**9.18.4.1 PRM Handler Invocation**
260
+
261
+
"The OS calls the PRM handler for CXL DPA to System Physical Address Translation
262
+
using the direct invocation mechanism. Details of calling a PRM handler are
263
+
described in the Platform Runtime Mechanism (PRM) specification.
264
+
265
+
The PRM handler is identified by the following GUID:
266
+
267
+
EE41B397-25D4-452C-AD54-48C6E3480B94
268
+
269
+
The caller allocates and prepares a Parameter Buffer, then passes the PRM
270
+
handler GUID and a pointer to the Parameter Buffer to invoke the handler. The
271
+
Parameter Buffer is described in Table 9-32."
272
+
273
+
**Table 9-32. PRM Parameter Buffer used for CXL DPA to System Physical Address Translation**
274
+
275
+
+-------------+-----------+------------------------------------------------------------------------+
276
+
| Byte Offset | Length in | Description |
277
+
| | Bytes | |
278
+
+=============+===========+========================================================================+
279
+
| 00h | 8 | **CXL Device Physical Address (DPA)**: CXL DPA (e.g., from |
280
+
| | | CXL Component Event Log) |
281
+
+-------------+-----------+------------------------------------------------------------------------+
282
+
| 08h | 4 | **CXL Endpoint SBDF**: |
283
+
| | | |
284
+
| | | - Byte 3 - PCIe Segment |
285
+
| | | - Byte 2 - Bus Number |
286
+
| | | - Byte 1: |
287
+
| | | - Device Number Bits[7:3] |
288
+
| | | - Function Number Bits[2:0] |
289
+
| | | - Byte 0 - RESERVED (MBZ) |
290
+
| | | |
291
+
+-------------+-----------+------------------------------------------------------------------------+
292
+
| 0Ch | 8 | **Output Buffer**: Virtual Address Pointer to the buffer, |
293
+
| | | as defined in Table 9-33. |
294
+
+-------------+-----------+------------------------------------------------------------------------+
295
+
296
+
**Table 9-33. PRM Output Buffer used for CXL DPA to System Physical Address Translation**
297
+
298
+
+-------------+-----------+------------------------------------------------------------------------+
299
+
| Byte Offset | Length in | Description |
300
+
| | Bytes | |
301
+
+=============+===========+========================================================================+
302
+
| 00h | 8 | **System Physical Address (SPA)**: The SPA converted |
303
+
| | | from the CXL DPA. |
304
+
+-------------+-----------+------------------------------------------------------------------------+
+135
Documentation/driver-api/cxl/conventions/cxl-lmh.rst
+135
Documentation/driver-api/cxl/conventions/cxl-lmh.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
4
+
============================================================================
5
+
6
+
Document
7
+
--------
8
+
9
+
CXL Revision 3.2, Version 1.0
10
+
11
+
License
12
+
-------
13
+
14
+
SPDX-License Identifier: CC-BY-4.0
15
+
16
+
Creator/Contributors
17
+
--------------------
18
+
19
+
- Fabio M. De Francesco, Intel
20
+
- Dan J. Williams, Intel
21
+
- Mahesh Natu, Intel
22
+
23
+
Summary of the Change
24
+
---------------------
25
+
26
+
According to the current Compute Express Link (CXL) Specifications (Revision
27
+
3.2, Version 1.0), the CXL Fixed Memory Window Structure (CFMWS) describes zero
28
+
or more Host Physical Address (HPA) windows associated with each CXL Host
29
+
Bridge. Each window represents a contiguous HPA range that may be interleaved
30
+
across one or more targets, including CXL Host Bridges. Each window has a set
31
+
of restrictions that govern its usage. It is the Operating System-directed
32
+
configuration and Power Management (OSPM) responsibility to utilize each window
33
+
for the specified use.
34
+
35
+
Table 9-22 of the current CXL Specifications states that the Window Size field
36
+
contains the total number of consecutive bytes of HPA this window describes.
37
+
This value must be a multiple of the Number of Interleave Ways (NIW) * 256 MB.
38
+
39
+
Platform Firmware (BIOS) might reserve physical addresses below 4 GB where a
40
+
memory gap such as the Low Memory Hole for PCIe MMIO may exist. In such cases,
41
+
the CFMWS Range Size may not adhere to the NIW * 256 MB rule.
42
+
43
+
The HPA represents the actual physical memory address space that the CXL devices
44
+
can decode and respond to, while the System Physical Address (SPA), a related
45
+
but distinct concept, represents the system-visible address space that users can
46
+
direct transaction to and so it excludes reserved regions.
47
+
48
+
BIOS publishes CFMWS to communicate the active SPA ranges that, on platforms
49
+
with LMH's, map to a strict subset of the HPA. The SPA range trims out the hole,
50
+
resulting in lost capacity in the Endpoints with no SPA to map to that part of
51
+
the HPA range that intersects the hole.
52
+
53
+
E.g, an x86 platform with two CFMWS and an LMH starting at 2 GB:
54
+
55
+
+--------+------------+-------------------+------------------+-------------------+------+
56
+
| Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways |
57
+
+========+============+===================+==================+===================+======+
58
+
| 0 | 0 GB | 2 GB | 0 GB | 3 GB | 12 |
59
+
+--------+------------+-------------------+------------------+-------------------+------+
60
+
| 1 | 4 GB | NIW*256MB Aligned | 4 GB | NIW*256MB Aligned | 12 |
61
+
+--------+------------+-------------------+------------------+-------------------+------+
62
+
63
+
HDM decoder base and HDM decoder size represent all the 12 Endpoint Decoders of
64
+
a 12 ways region and all the intermediate Switch Decoders. They are configured
65
+
by the BIOS according to the NIW * 256MB rule, resulting in a HPA range size of
66
+
3GB. Instead, the CFMWS Base and CFMWS Size are used to configure the Root
67
+
Decoder HPA range that results smaller (2GB) than that of the Switch and
68
+
Endpoint Decoders in the hierarchy (3GB).
69
+
70
+
This creates 2 issues which lead to a failure to construct a region:
71
+
72
+
1) A mismatch in region size between root and any HDM decoder. The root decoders
73
+
will always be smaller due to the trim.
74
+
75
+
2) The trim causes the root decoder to violate the (NIW * 256MB) rule.
76
+
77
+
This change allows a region with a base address of 0GB to bypass these checks to
78
+
allow for region creation with the trimmed root decoder address range.
79
+
80
+
This change does not allow for any other arbitrary region to violate these
81
+
checks - it is intended exclusively to enable x86 platforms which map CXL memory
82
+
under 4GB.
83
+
84
+
Despite the HDM decoders covering the PCIE hole HPA region, it is expected that
85
+
the platform will never route address accesses to the CXL complex because the
86
+
root decoder only covers the trimmed region (which excludes this). This is
87
+
outside the ability of Linux to enforce.
88
+
89
+
On the example platform, only the first 2GB will be potentially usable, but
90
+
Linux, aiming to adhere to the current specifications, fails to construct
91
+
Regions and attach Endpoint and intermediate Switch Decoders to them.
92
+
93
+
There are several points of failure that due to the expectation that the Root
94
+
Decoder HPA size, that is equal to the CFMWS from which it is configured, has
95
+
to be greater or equal to the matching Switch and Endpoint HDM Decoders.
96
+
97
+
In order to succeed with construction and attachment, Linux must construct a
98
+
Region with Root Decoder HPA range size, and then attach to that all the
99
+
intermediate Switch Decoders and Endpoint Decoders that belong to the hierarchy
100
+
regardless of their range sizes.
101
+
102
+
Benefits of the Change
103
+
----------------------
104
+
105
+
Without the change, the OSPM wouldn't match intermediate Switch and Endpoint
106
+
Decoders with Root Decoders configured with CFMWS HPA sizes that don't align
107
+
with the NIW * 256MB constraint, and so it leads to lost memdev capacity.
108
+
109
+
This change allows the OSPM to construct Regions and attach intermediate Switch
110
+
and Endpoint Decoders to them, so that the addressable part of the memory
111
+
devices total capacity is made available to the users.
112
+
113
+
References
114
+
----------
115
+
116
+
Compute Express Link Specification Revision 3.2, Version 1.0
117
+
<https://www.computeexpresslink.org/>
118
+
119
+
Detailed Description of the Change
120
+
----------------------------------
121
+
122
+
The description of the Window Size field in table 9-22 needs to account for
123
+
platforms with Low Memory Holes, where SPA ranges might be subsets of the
124
+
endpoints HPA. Therefore, it has to be changed to the following:
125
+
126
+
"The total number of consecutive bytes of HPA this window represents. This value
127
+
shall be a multiple of NIW * 256 MB.
128
+
129
+
On platforms that reserve physical addresses below 4 GB, such as the Low Memory
130
+
Hole for PCIe MMIO on x86, an instance of CFMWS whose Base HPA range is 0 might
131
+
have a size that doesn't align with the NIW * 256 MB constraint.
132
+
133
+
Note that the matching intermediate Switch Decoders and the Endpoint Decoders
134
+
HPA range sizes must still align to the above-mentioned rule, but the memory
135
+
capacity that exceeds the CFMWS window size won't be accessible.".
+37
Documentation/driver-api/cxl/conventions/template.rst
+37
Documentation/driver-api/cxl/conventions/template.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
.. :: Template Title here:
4
+
5
+
Template File
6
+
=============
7
+
8
+
Document
9
+
--------
10
+
CXL Revision <rev>, Version <ver>
11
+
12
+
License
13
+
-------
14
+
SPDX-License Identifier: CC-BY-4.0
15
+
16
+
Creator/Contributors
17
+
--------------------
18
+
19
+
Summary of the Change
20
+
---------------------
21
+
22
+
<Detail the conflict with the specification and where available the
23
+
assumptions and tradeoffs taken by the hardware platform.>
24
+
25
+
Benefits of the Change
26
+
----------------------
27
+
28
+
<Detail what happens if platforms and Linux do not adopt this
29
+
convention.>
30
+
31
+
References
32
+
----------
33
+
34
+
Detailed Description of the Change
35
+
----------------------------------
36
+
37
+
<Propose spec language that corrects the conflict.>
+1
Documentation/driver-api/cxl/index.rst
+1
Documentation/driver-api/cxl/index.rst
+23
Documentation/driver-api/cxl/platform/bios-and-efi.rst
+23
Documentation/driver-api/cxl/platform/bios-and-efi.rst
···
29
29
on physical memory region size and alignment, memory holes, HDM interleave,
30
30
and what linux expects of HDM decoders trying to work with these features.
31
31
32
+
33
+
Linux Expectations of BIOS/EFI Software
34
+
=======================================
35
+
Linux expects BIOS/EFI software to construct sufficient ACPI tables (such as
36
+
CEDT, SRAT, HMAT, etc) and platform-specific configurations (such as HPA spaces
37
+
and host-bridge interleave configurations) to allow the Linux driver to
38
+
subsequently configure the devices in the CXL fabric at runtime.
39
+
40
+
Programming of HDM decoders and switch ports is not required, and may be
41
+
deferred to the CXL driver based on admin policy (e.g. udev rules).
42
+
43
+
Some platforms may require pre-programming HDM decoders and locking them
44
+
due to quirks (see: Zen5 address translation), but this is not the normal,
45
+
"expected" configuration path. This should be avoided if possible.
46
+
47
+
Some platforms may wish to pre-configure these resources to bring memory
48
+
up without requiring CXL driver support. These platform vendors should
49
+
test their configurations with the existing CXL driver and provide driver
50
+
support for their auto-configurations if features like RAS are required.
51
+
52
+
Platforms requiring boot-time programming and/or locking of CXL fabric
53
+
components may prevent features, such as device hot-plug, from working.
54
+
32
55
UEFI Settings
33
56
=============
34
57
If your platform supports it, the :code:`uefisettings` command can be used to
+130
Documentation/driver-api/cxl/platform/device-hotplug.rst
+130
Documentation/driver-api/cxl/platform/device-hotplug.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
==================
4
+
CXL Device Hotplug
5
+
==================
6
+
7
+
Device hotplug refers to *physical* hotplug of a device (addition or removal
8
+
of a physical device from the machine).
9
+
10
+
BIOS/EFI software is expected to configure sufficient resources **at boot
11
+
time** to allow hotplugged devices to be configured by software (such as
12
+
proximity domains, HPA regions, and host-bridge configurations).
13
+
14
+
BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
15
+
devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
16
+
17
+
This document covers some examples of those resources, but should not
18
+
be considered exhaustive.
19
+
20
+
Hot-Remove
21
+
==========
22
+
Hot removal of a device typically requires careful removal of software
23
+
constructs (memory regions, associated drivers) which manage these devices.
24
+
25
+
Hard-removing a CXL.mem device without carefully tearing down driver stacks
26
+
is likely to cause the system to machine-check (or at least SIGBUS if memory
27
+
access is limited to user space).
28
+
29
+
Memory Device Hot-Add
30
+
=====================
31
+
A device present at boot may be associated with a CXL Fixed Memory Window
32
+
reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
33
+
device, but the construction of the CEDT CFMWS is platform-defined.
34
+
35
+
Hot-adding a memory device requires this pre-defined, **static** CFMWS to
36
+
have sufficient HPA space to describe that device.
37
+
38
+
There are a few common scenarios to consider.
39
+
40
+
Single-Endpoint Memory Device Present at Boot
41
+
---------------------------------------------
42
+
A device present at boot likely had its capacity reported in the
43
+
:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
44
+
the capacity of the new device will be limited to the original CFMWS capacity.
45
+
46
+
Adding capacity larger than the original device will cause memory region
47
+
creation to fail if the region size is greater than the CFMWS size.
48
+
49
+
The CFMWS is **static** and cannot be adjusted. Platforms which may expect
50
+
different sized devices to be hotplugged must allocate sufficient CFMWS space
51
+
**at boot time** to cover all future expected devices.
52
+
53
+
Multi-Endpoint Memory Device Present at Boot
54
+
--------------------------------------------
55
+
Non-switch-based Multi-Endpoint devices are outside the scope of what the
56
+
CXL specification describes, but they are technically possible. We describe
57
+
them here for instructive reasons only - this does not imply Linux support.
58
+
59
+
A hot-plug capable CXL memory device, such as one which presents multiple
60
+
expanders as a single large-capacity device, should report the **maximum
61
+
possible capacity** for the device at boot. ::
62
+
63
+
HB0
64
+
RP0
65
+
|
66
+
[Multi-Endpoint Memory Device]
67
+
_____|_____
68
+
| |
69
+
[Endpoint0] [Empty]
70
+
71
+
72
+
Limiting the size to the capacity preset at boot will limit hot-add support
73
+
to replacing capacity that was present at boot.
74
+
75
+
No CXL Device Present at Boot
76
+
-----------------------------
77
+
When no CXL memory device is present on boot, some platforms omit the CFMWS
78
+
in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
79
+
80
+
This describes the base case for any given device not being present at boot.
81
+
If a future possible device is not described in the CEDT at boot, hot-add
82
+
of that device is either limited or not possible.
83
+
84
+
For a platform to support hot-add of a full memory device, it must allocate
85
+
a CEDT CFMWS region with sufficient memory capacity to cover all future
86
+
potentially added capacity (along with any relevant CEDT CHBS entry).
87
+
88
+
To support memory hotplug directly on the host bridge/root port, or on a switch
89
+
downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
90
+
with sufficient resources to support the max possible (or expected) hotplug
91
+
memory capacity. ::
92
+
93
+
HB0 HB1
94
+
RP0 RP1 RP2
95
+
| | |
96
+
Empty Empty USP
97
+
________|________
98
+
| | | |
99
+
DSP DSP DSP DSP
100
+
| | | |
101
+
All Empty
102
+
103
+
For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
104
+
a pre-configured amount of memory capacity (per host bridge, or host bridge
105
+
interleave set), even if no device is attached to Root Ports or Downstream
106
+
Ports at boot (as depicted in the figure above).
107
+
108
+
109
+
Interleave Sets
110
+
===============
111
+
112
+
Host Bridge Interleave
113
+
----------------------
114
+
Host-bridge interleaved memory regions are defined **statically** in the
115
+
:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
116
+
describing that interleave must have been provided **at boot**. Hotplugged
117
+
devices cannot add host-bridge interleave capabilities at hotplug time.
118
+
119
+
See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
120
+
example to see how a platform can provide this kind of flexibility regarding
121
+
hotplugged memory devices. BIOS/EFI software should consider options to
122
+
present flexible CEDT configurations with hotplug support.
123
+
124
+
HDM Interleave
125
+
--------------
126
+
Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
127
+
can be re-programmed after hotplug.
128
+
129
+
To add or remove a device to/from an existing HDM-applied interleaved region,
130
+
that region must be torn down an re-created.
+14
-8
arch/x86/kernel/e820.c
+14
-8
arch/x86/kernel/e820.c
···
1173
1173
1174
1174
__init void e820__reserve_resources_late(void)
1175
1175
{
1176
-
u32 idx;
1177
-
struct resource *res;
1178
-
1179
1176
/*
1180
1177
* Register device address regions listed in the E820 map,
1181
1178
* these can be claimed by device drivers later on:
1182
1179
*/
1183
-
res = e820_res;
1184
-
for (idx = 0; idx < e820_table->nr_entries; idx++) {
1185
-
if (!res->parent && res->end)
1180
+
for (u32 idx = 0; idx < e820_table->nr_entries; idx++) {
1181
+
struct resource *res = e820_res + idx;
1182
+
1183
+
/* skip added or uninitialized resources */
1184
+
if (res->parent || !res->end)
1185
+
continue;
1186
+
1187
+
/* set aside soft-reserved resources for driver consideration */
1188
+
if (res->desc == IORES_DESC_SOFT_RESERVED) {
1189
+
insert_resource_expand_to_fit(&soft_reserve_resource, res);
1190
+
} else {
1191
+
/* publish the rest immediately */
1186
1192
insert_resource_expand_to_fit(&iomem_resource, res);
1187
-
res++;
1193
+
}
1188
1194
}
1189
1195
1190
1196
/*
···
1205
1199
* doesn't properly list 'stolen RAM' as a system region
1206
1200
* in the E820 map.
1207
1201
*/
1208
-
for (idx = 0; idx < e820_table->nr_entries; idx++) {
1202
+
for (u32 idx = 0; idx < e820_table->nr_entries; idx++) {
1209
1203
struct e820_entry *entry = &e820_table->entries[idx];
1210
1204
u64 start, end;
1211
1205
+10
-1
drivers/cxl/Kconfig
+10
-1
drivers/cxl/Kconfig
···
22
22
config CXL_PCI
23
23
tristate "PCI manageability"
24
24
default CXL_BUS
25
+
select CXL_MEM
25
26
help
26
27
The CXL specification defines a "CXL memory device" sub-class in the
27
28
PCI "memory controller" base class of devices. Device's identified by
···
90
89
91
90
config CXL_MEM
92
91
tristate "CXL: Memory Expansion"
93
-
depends on CXL_PCI
94
92
default CXL_BUS
95
93
help
96
94
The CXL.mem protocol allows a device to act as a provider of "System
···
232
232
config CXL_MCE
233
233
def_bool y
234
234
depends on X86_MCE && MEMORY_FAILURE
235
+
236
+
config CXL_RAS
237
+
def_bool y
238
+
depends on ACPI_APEI_GHES && PCIEAER && CXL_BUS
239
+
240
+
config CXL_ATL
241
+
def_bool y
242
+
depends on CXL_REGION
243
+
depends on ACPI_PRMT && AMD_NB
235
244
236
245
endif
+15
-31
drivers/cxl/acpi.c
+15
-31
drivers/cxl/acpi.c
···
325
325
return cxl_acpi_evaluate_qtg_dsm(handle, coord, entries, qos_class);
326
326
}
327
327
328
-
static const struct cxl_root_ops acpi_root_ops = {
329
-
.qos_class = cxl_acpi_qos_class,
330
-
};
331
-
332
328
static void del_cxl_resource(struct resource *res)
333
329
{
334
330
if (!res)
···
360
364
return rc;
361
365
}
362
366
363
-
static int cxl_acpi_set_cache_size(struct cxl_root_decoder *cxlrd)
367
+
static void cxl_setup_extended_linear_cache(struct cxl_root_decoder *cxlrd)
364
368
{
365
369
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
366
370
struct range *hpa = &cxld->hpa_range;
···
370
374
struct resource res;
371
375
int nid, rc;
372
376
377
+
/* Explicitly initialize cache size to 0 at the beginning */
378
+
cxlrd->cache_size = 0;
373
379
res = DEFINE_RES_MEM(start, size);
374
380
nid = phys_to_target_node(start);
375
381
376
382
rc = hmat_get_extended_linear_cache_size(&res, nid, &cache_size);
377
383
if (rc)
378
-
return 0;
384
+
return;
379
385
380
386
/*
381
387
* The cache range is expected to be within the CFMWS.
···
389
391
dev_warn(&cxld->dev,
390
392
"Extended Linear Cache size %pa != CXL size %pa. No Support!",
391
393
&cache_size, &size);
392
-
return -ENXIO;
394
+
return;
393
395
}
394
396
395
397
cxlrd->cache_size = cache_size;
396
-
397
-
return 0;
398
-
}
399
-
400
-
static void cxl_setup_extended_linear_cache(struct cxl_root_decoder *cxlrd)
401
-
{
402
-
int rc;
403
-
404
-
rc = cxl_acpi_set_cache_size(cxlrd);
405
-
if (rc) {
406
-
/*
407
-
* Failing to retrieve extended linear cache region resize does not
408
-
* prevent the region from functioning. Only causes cxl list showing
409
-
* incorrect region size.
410
-
*/
411
-
dev_warn(cxlrd->cxlsd.cxld.dev.parent,
412
-
"Extended linear cache retrieval failed rc:%d\n", rc);
413
-
414
-
/* Ignoring return code */
415
-
cxlrd->cache_size = 0;
416
-
}
417
398
}
418
399
419
400
DEFINE_FREE(put_cxlrd, struct cxl_root_decoder *,
···
907
930
cxl_res->end = -1;
908
931
cxl_res->flags = IORESOURCE_MEM;
909
932
910
-
cxl_root = devm_cxl_add_root(host, &acpi_root_ops);
933
+
cxl_root = devm_cxl_add_root(host);
911
934
if (IS_ERR(cxl_root))
912
935
return PTR_ERR(cxl_root);
936
+
cxl_root->ops.qos_class = cxl_acpi_qos_class;
913
937
root_port = &cxl_root->port;
938
+
939
+
cxl_setup_prm_address_translation(cxl_root);
914
940
915
941
rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
916
942
add_host_bridge_dport);
···
995
1015
cxl_bus_drain();
996
1016
}
997
1017
998
-
/* load before dax_hmem sees 'Soft Reserved' CXL ranges */
999
-
subsys_initcall(cxl_acpi_init);
1018
+
/*
1019
+
* Load before dax_hmem sees 'Soft Reserved' CXL ranges. Use
1020
+
* subsys_initcall_sync() since there is an order dependency with
1021
+
* subsys_initcall(efisubsys_init), which must run first.
1022
+
*/
1023
+
subsys_initcall_sync(cxl_acpi_init);
1000
1024
1001
1025
/*
1002
1026
* Arrange for host-bridge ports to be active synchronous with
+3
-1
drivers/cxl/core/Makefile
+3
-1
drivers/cxl/core/Makefile
···
14
14
cxl_core-y += hdm.o
15
15
cxl_core-y += pmu.o
16
16
cxl_core-y += cdat.o
17
-
cxl_core-y += ras.o
18
17
cxl_core-$(CONFIG_TRACING) += trace.o
19
18
cxl_core-$(CONFIG_CXL_REGION) += region.o
20
19
cxl_core-$(CONFIG_CXL_MCE) += mce.o
21
20
cxl_core-$(CONFIG_CXL_FEATURES) += features.o
22
21
cxl_core-$(CONFIG_CXL_EDAC_MEM_FEATURES) += edac.o
22
+
cxl_core-$(CONFIG_CXL_RAS) += ras.o
23
+
cxl_core-$(CONFIG_CXL_RAS) += ras_rch.o
24
+
cxl_core-$(CONFIG_CXL_ATL) += atl.o
+211
drivers/cxl/core/atl.c
+211
drivers/cxl/core/atl.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/*
3
+
* Copyright (C) 2025 Advanced Micro Devices, Inc.
4
+
*/
5
+
6
+
#include <linux/prmt.h>
7
+
#include <linux/pci.h>
8
+
#include <linux/acpi.h>
9
+
10
+
#include <cxlmem.h>
11
+
#include "core.h"
12
+
13
+
/*
14
+
* PRM Address Translation - CXL DPA to System Physical Address
15
+
*
16
+
* Reference:
17
+
*
18
+
* AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh
19
+
* ACPI v6.5 Porting Guide, Publication # 58088
20
+
*/
21
+
22
+
static const guid_t prm_cxl_dpa_spa_guid =
23
+
GUID_INIT(0xee41b397, 0x25d4, 0x452c, 0xad, 0x54, 0x48, 0xc6, 0xe3,
24
+
0x48, 0x0b, 0x94);
25
+
26
+
struct prm_cxl_dpa_spa_data {
27
+
u64 dpa;
28
+
u8 reserved;
29
+
u8 devfn;
30
+
u8 bus;
31
+
u8 segment;
32
+
u64 *spa;
33
+
} __packed;
34
+
35
+
static u64 prm_cxl_dpa_spa(struct pci_dev *pci_dev, u64 dpa)
36
+
{
37
+
struct prm_cxl_dpa_spa_data data;
38
+
u64 spa;
39
+
int rc;
40
+
41
+
data = (struct prm_cxl_dpa_spa_data) {
42
+
.dpa = dpa,
43
+
.devfn = pci_dev->devfn,
44
+
.bus = pci_dev->bus->number,
45
+
.segment = pci_domain_nr(pci_dev->bus),
46
+
.spa = &spa,
47
+
};
48
+
49
+
rc = acpi_call_prm_handler(prm_cxl_dpa_spa_guid, &data);
50
+
if (rc) {
51
+
pci_dbg(pci_dev, "failed to get SPA for %#llx: %d\n", dpa, rc);
52
+
return ULLONG_MAX;
53
+
}
54
+
55
+
pci_dbg(pci_dev, "PRM address translation: DPA -> SPA: %#llx -> %#llx\n", dpa, spa);
56
+
57
+
return spa;
58
+
}
59
+
60
+
static int cxl_prm_setup_root(struct cxl_root *cxl_root, void *data)
61
+
{
62
+
struct cxl_region_context *ctx = data;
63
+
struct cxl_endpoint_decoder *cxled = ctx->cxled;
64
+
struct cxl_decoder *cxld = &cxled->cxld;
65
+
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
66
+
struct range hpa_range = ctx->hpa_range;
67
+
struct pci_dev *pci_dev;
68
+
u64 spa_len, len;
69
+
u64 addr, base_spa, base;
70
+
int ways, gran;
71
+
72
+
/*
73
+
* When Normalized Addressing is enabled, the endpoint maintains a 1:1
74
+
* mapping between HPA and DPA. If disabled, skip address translation
75
+
* and perform only a range check.
76
+
*/
77
+
if (hpa_range.start != cxled->dpa_res->start)
78
+
return 0;
79
+
80
+
/*
81
+
* Endpoints are programmed passthrough in Normalized Addressing mode.
82
+
*/
83
+
if (ctx->interleave_ways != 1) {
84
+
dev_dbg(&cxld->dev, "unexpected interleaving config: ways: %d granularity: %d\n",
85
+
ctx->interleave_ways, ctx->interleave_granularity);
86
+
return -ENXIO;
87
+
}
88
+
89
+
if (!cxlmd || !dev_is_pci(cxlmd->dev.parent)) {
90
+
dev_dbg(&cxld->dev, "No endpoint found: %s, range %#llx-%#llx\n",
91
+
dev_name(cxld->dev.parent), hpa_range.start,
92
+
hpa_range.end);
93
+
return -ENXIO;
94
+
}
95
+
96
+
pci_dev = to_pci_dev(cxlmd->dev.parent);
97
+
98
+
/* Translate HPA range to SPA. */
99
+
base = hpa_range.start;
100
+
hpa_range.start = prm_cxl_dpa_spa(pci_dev, hpa_range.start);
101
+
hpa_range.end = prm_cxl_dpa_spa(pci_dev, hpa_range.end);
102
+
base_spa = hpa_range.start;
103
+
104
+
if (hpa_range.start == ULLONG_MAX || hpa_range.end == ULLONG_MAX) {
105
+
dev_dbg(cxld->dev.parent,
106
+
"CXL address translation: Failed to translate HPA range: %#llx-%#llx:%#llx-%#llx(%s)\n",
107
+
hpa_range.start, hpa_range.end, ctx->hpa_range.start,
108
+
ctx->hpa_range.end, dev_name(&cxld->dev));
109
+
return -ENXIO;
110
+
}
111
+
112
+
/*
113
+
* Since translated addresses include the interleaving offsets, align
114
+
* the range to 256 MB.
115
+
*/
116
+
hpa_range.start = ALIGN_DOWN(hpa_range.start, SZ_256M);
117
+
hpa_range.end = ALIGN(hpa_range.end, SZ_256M) - 1;
118
+
119
+
len = range_len(&ctx->hpa_range);
120
+
spa_len = range_len(&hpa_range);
121
+
if (!len || !spa_len || spa_len % len) {
122
+
dev_dbg(cxld->dev.parent,
123
+
"CXL address translation: HPA range not contiguous: %#llx-%#llx:%#llx-%#llx(%s)\n",
124
+
hpa_range.start, hpa_range.end, ctx->hpa_range.start,
125
+
ctx->hpa_range.end, dev_name(&cxld->dev));
126
+
return -ENXIO;
127
+
}
128
+
129
+
ways = spa_len / len;
130
+
gran = SZ_256;
131
+
132
+
/*
133
+
* Determine interleave granularity
134
+
*
135
+
* Note: The position of the chunk from one interleaving block to the
136
+
* next may vary and thus cannot be considered constant. Address offsets
137
+
* larger than the interleaving block size cannot be used to calculate
138
+
* the granularity.
139
+
*/
140
+
if (ways > 1) {
141
+
while (gran <= SZ_16M) {
142
+
addr = prm_cxl_dpa_spa(pci_dev, base + gran);
143
+
if (addr != base_spa + gran)
144
+
break;
145
+
gran <<= 1;
146
+
}
147
+
}
148
+
149
+
if (gran > SZ_16M) {
150
+
dev_dbg(cxld->dev.parent,
151
+
"CXL address translation: Cannot determine granularity: %#llx-%#llx:%#llx-%#llx(%s)\n",
152
+
hpa_range.start, hpa_range.end, ctx->hpa_range.start,
153
+
ctx->hpa_range.end, dev_name(&cxld->dev));
154
+
return -ENXIO;
155
+
}
156
+
157
+
/*
158
+
* The current kernel implementation does not support endpoint
159
+
* setup with Normalized Addressing. It only translates an
160
+
* endpoint's DPA to the SPA range of the host bridge.
161
+
* Therefore, the endpoint address range cannot be determined,
162
+
* making a non-auto setup impossible. If a decoder requires
163
+
* address translation, reprogramming should be disabled and
164
+
* the decoder locked.
165
+
*
166
+
* The BIOS, however, provides all the necessary address
167
+
* translation data, which the kernel can use to reconfigure
168
+
* endpoint decoders with normalized addresses. Locking the
169
+
* decoders in the BIOS would prevent a capable kernel (or
170
+
* other operating systems) from shutting down auto-generated
171
+
* regions and managing resources dynamically.
172
+
*
173
+
* Indicate that Normalized Addressing is enabled.
174
+
*/
175
+
cxld->flags |= CXL_DECODER_F_LOCK;
176
+
cxld->flags |= CXL_DECODER_F_NORMALIZED_ADDRESSING;
177
+
178
+
ctx->hpa_range = hpa_range;
179
+
ctx->interleave_ways = ways;
180
+
ctx->interleave_granularity = gran;
181
+
182
+
dev_dbg(&cxld->dev,
183
+
"address mapping found for %s (hpa -> spa): %#llx+%#llx -> %#llx+%#llx ways:%d granularity:%d\n",
184
+
dev_name(cxlmd->dev.parent), base, len, hpa_range.start,
185
+
spa_len, ways, gran);
186
+
187
+
return 0;
188
+
}
189
+
190
+
void cxl_setup_prm_address_translation(struct cxl_root *cxl_root)
191
+
{
192
+
struct device *host = cxl_root->port.uport_dev;
193
+
u64 spa;
194
+
struct prm_cxl_dpa_spa_data data = { .spa = &spa };
195
+
int rc;
196
+
197
+
/*
198
+
* Applies only to PCIe Host Bridges which are children of the CXL Root
199
+
* Device (HID=“ACPI0017”). Check this and drop cxl_test instances.
200
+
*/
201
+
if (!acpi_match_device(host->driver->acpi_match_table, host))
202
+
return;
203
+
204
+
/* Check kernel (-EOPNOTSUPP) and firmware support (-ENODEV) */
205
+
rc = acpi_call_prm_handler(prm_cxl_dpa_spa_guid, &data);
206
+
if (rc == -EOPNOTSUPP || rc == -ENODEV)
207
+
return;
208
+
209
+
cxl_root->ops.translation_setup_root = cxl_prm_setup_root;
210
+
}
211
+
EXPORT_SYMBOL_NS_GPL(cxl_setup_prm_address_translation, "CXL");
+4
-4
drivers/cxl/core/cdat.c
+4
-4
drivers/cxl/core/cdat.c
···
213
213
if (!cxl_root)
214
214
return -ENODEV;
215
215
216
-
if (!cxl_root->ops || !cxl_root->ops->qos_class)
216
+
if (!cxl_root->ops.qos_class)
217
217
return -EOPNOTSUPP;
218
218
219
219
xa_for_each(dsmas_xa, index, dent) {
···
221
221
222
222
cxl_coordinates_combine(dent->coord, dent->cdat_coord, ep_c);
223
223
dent->entries = 1;
224
-
rc = cxl_root->ops->qos_class(cxl_root,
225
-
&dent->coord[ACCESS_COORDINATE_CPU],
226
-
1, &qos_class);
224
+
rc = cxl_root->ops.qos_class(cxl_root,
225
+
&dent->coord[ACCESS_COORDINATE_CPU],
226
+
1, &qos_class);
227
227
if (rc != 1)
228
228
continue;
229
229
+40
drivers/cxl/core/core.h
+40
drivers/cxl/core/core.h
···
19
19
};
20
20
21
21
#ifdef CONFIG_CXL_REGION
22
+
23
+
struct cxl_region_context {
24
+
struct cxl_endpoint_decoder *cxled;
25
+
struct range hpa_range;
26
+
int interleave_ways;
27
+
int interleave_granularity;
28
+
};
29
+
22
30
extern struct device_attribute dev_attr_create_pmem_region;
23
31
extern struct device_attribute dev_attr_create_ram_region;
24
32
extern struct device_attribute dev_attr_delete_region;
···
152
144
int cxl_port_get_switch_dport_bandwidth(struct cxl_port *port,
153
145
struct access_coordinate *c);
154
146
147
+
static inline struct device *dport_to_host(struct cxl_dport *dport)
148
+
{
149
+
struct cxl_port *port = dport->port;
150
+
151
+
if (is_cxl_root(port))
152
+
return port->uport_dev;
153
+
return &port->dev;
154
+
}
155
+
#ifdef CONFIG_CXL_RAS
155
156
int cxl_ras_init(void);
156
157
void cxl_ras_exit(void);
158
+
bool cxl_handle_ras(struct device *dev, void __iomem *ras_base);
159
+
void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base);
160
+
void cxl_dport_map_rch_aer(struct cxl_dport *dport);
161
+
void cxl_disable_rch_root_ints(struct cxl_dport *dport);
162
+
void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds);
163
+
void devm_cxl_dport_ras_setup(struct cxl_dport *dport);
164
+
#else
165
+
static inline int cxl_ras_init(void)
166
+
{
167
+
return 0;
168
+
}
169
+
static inline void cxl_ras_exit(void) { }
170
+
static inline bool cxl_handle_ras(struct device *dev, void __iomem *ras_base)
171
+
{
172
+
return false;
173
+
}
174
+
static inline void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base) { }
175
+
static inline void cxl_dport_map_rch_aer(struct cxl_dport *dport) { }
176
+
static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { }
177
+
static inline void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { }
178
+
static inline void devm_cxl_dport_ras_setup(struct cxl_dport *dport) { }
179
+
#endif /* CONFIG_CXL_RAS */
180
+
157
181
int cxl_gpf_port_setup(struct cxl_dport *dport);
158
182
159
183
struct cxl_hdm;
+37
-27
drivers/cxl/core/edac.c
+37
-27
drivers/cxl/core/edac.c
···
1988
1988
return 0;
1989
1989
}
1990
1990
1991
+
static void err_rec_free(void *_cxlmd)
1992
+
{
1993
+
struct cxl_memdev *cxlmd = _cxlmd;
1994
+
struct cxl_mem_err_rec *array_rec = cxlmd->err_rec_array;
1995
+
struct cxl_event_gen_media *rec_gen_media;
1996
+
struct cxl_event_dram *rec_dram;
1997
+
unsigned long index;
1998
+
1999
+
cxlmd->err_rec_array = NULL;
2000
+
xa_for_each(&array_rec->rec_dram, index, rec_dram)
2001
+
kfree(rec_dram);
2002
+
xa_destroy(&array_rec->rec_dram);
2003
+
2004
+
xa_for_each(&array_rec->rec_gen_media, index, rec_gen_media)
2005
+
kfree(rec_gen_media);
2006
+
xa_destroy(&array_rec->rec_gen_media);
2007
+
kfree(array_rec);
2008
+
}
2009
+
2010
+
static int devm_cxl_memdev_setup_err_rec(struct cxl_memdev *cxlmd)
2011
+
{
2012
+
struct cxl_mem_err_rec *array_rec =
2013
+
kzalloc(sizeof(*array_rec), GFP_KERNEL);
2014
+
2015
+
if (!array_rec)
2016
+
return -ENOMEM;
2017
+
2018
+
xa_init(&array_rec->rec_gen_media);
2019
+
xa_init(&array_rec->rec_dram);
2020
+
cxlmd->err_rec_array = array_rec;
2021
+
2022
+
return devm_add_action_or_reset(&cxlmd->dev, err_rec_free, cxlmd);
2023
+
}
2024
+
1991
2025
int devm_cxl_memdev_edac_register(struct cxl_memdev *cxlmd)
1992
2026
{
1993
2027
struct edac_dev_feature ras_features[CXL_NR_EDAC_DEV_FEATURES];
···
2072
2038
}
2073
2039
2074
2040
if (repair_inst) {
2075
-
struct cxl_mem_err_rec *array_rec =
2076
-
devm_kzalloc(&cxlmd->dev, sizeof(*array_rec),
2077
-
GFP_KERNEL);
2078
-
if (!array_rec)
2079
-
return -ENOMEM;
2080
-
2081
-
xa_init(&array_rec->rec_gen_media);
2082
-
xa_init(&array_rec->rec_dram);
2083
-
cxlmd->err_rec_array = array_rec;
2041
+
rc = devm_cxl_memdev_setup_err_rec(cxlmd);
2042
+
if (rc)
2043
+
return rc;
2084
2044
}
2085
2045
}
2086
2046
···
2116
2088
}
2117
2089
EXPORT_SYMBOL_NS_GPL(devm_cxl_region_edac_register, "CXL");
2118
2090
2119
-
void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd)
2120
-
{
2121
-
struct cxl_mem_err_rec *array_rec = cxlmd->err_rec_array;
2122
-
struct cxl_event_gen_media *rec_gen_media;
2123
-
struct cxl_event_dram *rec_dram;
2124
-
unsigned long index;
2125
2091
2126
-
if (!IS_ENABLED(CONFIG_CXL_EDAC_MEM_REPAIR) || !array_rec)
2127
-
return;
2128
-
2129
-
xa_for_each(&array_rec->rec_dram, index, rec_dram)
2130
-
kfree(rec_dram);
2131
-
xa_destroy(&array_rec->rec_dram);
2132
-
2133
-
xa_for_each(&array_rec->rec_gen_media, index, rec_gen_media)
2134
-
kfree(rec_gen_media);
2135
-
xa_destroy(&array_rec->rec_gen_media);
2136
-
}
2137
-
EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_edac_release, "CXL");
+6
-7
drivers/cxl/core/hdm.c
+6
-7
drivers/cxl/core/hdm.c
···
844
844
scoped_guard(rwsem_read, &cxl_rwsem.dpa)
845
845
setup_hw_decoder(cxld, hdm);
846
846
847
-
port->commit_end++;
848
847
rc = cxld_await_commit(hdm, cxld->id);
849
848
if (rc) {
850
849
dev_dbg(&port->dev, "%s: error %d committing decoder\n",
851
850
dev_name(&cxld->dev), rc);
852
-
cxld->reset(cxld);
853
851
return rc;
854
852
}
853
+
port->commit_end++;
855
854
cxld->flags |= CXL_DECODER_F_ENABLE;
856
855
857
856
return 0;
···
965
966
rc = devm_cxl_dpa_reserve(cxled, *dpa_base, len, 0);
966
967
if (rc) {
967
968
dev_err(&port->dev,
968
-
"decoder%d.%d: Failed to reserve DPA range %#llx - %#llx\n (%d)",
969
+
"decoder%d.%d: Failed to reserve DPA range %#llx - %#llx: %d\n",
969
970
port->id, cxld->id, *dpa_base, *dpa_base + len - 1, rc);
970
971
return rc;
971
972
}
···
1116
1117
rc = devm_cxl_dpa_reserve(cxled, *dpa_base + skip, dpa_size, skip);
1117
1118
if (rc) {
1118
1119
dev_err(&port->dev,
1119
-
"decoder%d.%d: Failed to reserve DPA range %#llx - %#llx\n (%d)",
1120
+
"decoder%d.%d: Failed to reserve DPA range %#llx - %#llx: %d\n",
1120
1121
port->id, cxld->id, *dpa_base,
1121
1122
*dpa_base + dpa_size + skip - 1, rc);
1122
1123
return rc;
···
1218
1219
}
1219
1220
1220
1221
/**
1221
-
* __devm_cxl_switch_port_decoders_setup - allocate and setup switch decoders
1222
+
* devm_cxl_switch_port_decoders_setup - allocate and setup switch decoders
1222
1223
* @port: CXL port context
1223
1224
*
1224
1225
* Return 0 or -errno on error
1225
1226
*/
1226
-
int __devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
1227
+
int devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
1227
1228
{
1228
1229
struct cxl_hdm *cxlhdm;
1229
1230
···
1247
1248
dev_err(&port->dev, "HDM decoder capability not found\n");
1248
1249
return -ENXIO;
1249
1250
}
1250
-
EXPORT_SYMBOL_NS_GPL(__devm_cxl_switch_port_decoders_setup, "CXL");
1251
+
EXPORT_SYMBOL_NS_GPL(devm_cxl_switch_port_decoders_setup, "CXL");
1251
1252
1252
1253
/**
1253
1254
* devm_cxl_endpoint_decoders_setup - allocate and setup endpoint decoders
+78
-33
drivers/cxl/core/memdev.c
+78
-33
drivers/cxl/core/memdev.c
···
27
27
struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
28
28
29
29
ida_free(&cxl_memdev_ida, cxlmd->id);
30
-
devm_cxl_memdev_edac_release(cxlmd);
31
30
kfree(cxlmd);
32
31
}
33
32
···
641
642
struct cxl_memdev *cxlmd;
642
643
643
644
cxlmd = container_of(work, typeof(*cxlmd), detach_work);
644
-
device_release_driver(&cxlmd->dev);
645
+
646
+
/*
647
+
* When the creator of @cxlmd sets ->attach it indicates CXL operation
648
+
* is required. In that case, @cxlmd detach escalates to parent device
649
+
* detach.
650
+
*/
651
+
if (cxlmd->attach)
652
+
device_release_driver(cxlmd->dev.parent);
653
+
else
654
+
device_release_driver(&cxlmd->dev);
645
655
put_device(&cxlmd->dev);
646
656
}
647
657
648
658
static struct lock_class_key cxl_memdev_key;
649
659
650
660
static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
651
-
const struct file_operations *fops)
661
+
const struct file_operations *fops,
662
+
const struct cxl_memdev_attach *attach)
652
663
{
653
664
struct cxl_memdev *cxlmd;
654
665
struct device *dev;
···
674
665
goto err;
675
666
cxlmd->id = rc;
676
667
cxlmd->depth = -1;
668
+
cxlmd->attach = attach;
669
+
cxlmd->endpoint = ERR_PTR(-ENXIO);
677
670
678
671
dev = &cxlmd->dev;
679
672
device_initialize(dev);
···
1062
1051
.llseek = noop_llseek,
1063
1052
};
1064
1053
1065
-
struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
1066
-
struct cxl_dev_state *cxlds)
1054
+
/*
1055
+
* Activate ioctl operations, no cxl_memdev_rwsem manipulation needed as this is
1056
+
* ordered with cdev_add() publishing the device.
1057
+
*/
1058
+
static int cxlmd_add(struct cxl_memdev *cxlmd, struct cxl_dev_state *cxlds)
1067
1059
{
1068
-
struct cxl_memdev *cxlmd;
1069
-
struct device *dev;
1070
-
struct cdev *cdev;
1071
1060
int rc;
1072
1061
1073
-
cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
1062
+
cxlmd->cxlds = cxlds;
1063
+
cxlds->cxlmd = cxlmd;
1064
+
1065
+
rc = cdev_device_add(&cxlmd->cdev, &cxlmd->dev);
1066
+
if (rc) {
1067
+
/*
1068
+
* The cdev was briefly live, shutdown any ioctl operations that
1069
+
* saw that state.
1070
+
*/
1071
+
cxl_memdev_shutdown(&cxlmd->dev);
1072
+
return rc;
1073
+
}
1074
+
1075
+
return 0;
1076
+
}
1077
+
1078
+
DEFINE_FREE(put_cxlmd, struct cxl_memdev *,
1079
+
if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
1080
+
1081
+
static struct cxl_memdev *cxl_memdev_autoremove(struct cxl_memdev *cxlmd)
1082
+
{
1083
+
int rc;
1084
+
1085
+
/*
1086
+
* If @attach is provided fail if the driver is not attached upon
1087
+
* return. Note that failure here could be the result of a race to
1088
+
* teardown the CXL port topology. I.e. cxl_mem_probe() could have
1089
+
* succeeded and then cxl_mem unbound before the lock is acquired.
1090
+
*/
1091
+
guard(device)(&cxlmd->dev);
1092
+
if (cxlmd->attach && !cxlmd->dev.driver) {
1093
+
cxl_memdev_unregister(cxlmd);
1094
+
return ERR_PTR(-ENXIO);
1095
+
}
1096
+
1097
+
rc = devm_add_action_or_reset(cxlmd->cxlds->dev, cxl_memdev_unregister,
1098
+
cxlmd);
1099
+
if (rc)
1100
+
return ERR_PTR(rc);
1101
+
1102
+
return cxlmd;
1103
+
}
1104
+
1105
+
/*
1106
+
* Core helper for devm_cxl_add_memdev() that wants to both create a device and
1107
+
* assert to the caller that upon return cxl_mem::probe() has been invoked.
1108
+
*/
1109
+
struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds,
1110
+
const struct cxl_memdev_attach *attach)
1111
+
{
1112
+
struct device *dev;
1113
+
int rc;
1114
+
1115
+
struct cxl_memdev *cxlmd __free(put_cxlmd) =
1116
+
cxl_memdev_alloc(cxlds, &cxl_memdev_fops, attach);
1074
1117
if (IS_ERR(cxlmd))
1075
1118
return cxlmd;
1076
1119
1077
1120
dev = &cxlmd->dev;
1078
1121
rc = dev_set_name(dev, "mem%d", cxlmd->id);
1079
1122
if (rc)
1080
-
goto err;
1123
+
return ERR_PTR(rc);
1081
1124
1082
-
/*
1083
-
* Activate ioctl operations, no cxl_memdev_rwsem manipulation
1084
-
* needed as this is ordered with cdev_add() publishing the device.
1085
-
*/
1086
-
cxlmd->cxlds = cxlds;
1087
-
cxlds->cxlmd = cxlmd;
1088
-
1089
-
cdev = &cxlmd->cdev;
1090
-
rc = cdev_device_add(cdev, dev);
1091
-
if (rc)
1092
-
goto err;
1093
-
1094
-
rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
1125
+
rc = cxlmd_add(cxlmd, cxlds);
1095
1126
if (rc)
1096
1127
return ERR_PTR(rc);
1097
-
return cxlmd;
1098
1128
1099
-
err:
1100
-
/*
1101
-
* The cdev was briefly live, shutdown any ioctl operations that
1102
-
* saw that state.
1103
-
*/
1104
-
cxl_memdev_shutdown(dev);
1105
-
put_device(dev);
1106
-
return ERR_PTR(rc);
1129
+
return cxl_memdev_autoremove(no_free_ptr(cxlmd));
1107
1130
}
1108
-
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
1131
+
EXPORT_SYMBOL_FOR_MODULES(__devm_cxl_add_memdev, "cxl_mem");
1109
1132
1110
1133
static void sanitize_teardown_notifier(void *data)
1111
1134
{
+33
-351
drivers/cxl/core/pci.c
+33
-351
drivers/cxl/core/pci.c
···
41
41
}
42
42
43
43
/**
44
-
* __devm_cxl_add_dport_by_dev - allocate a dport by dport device
44
+
* devm_cxl_add_dport_by_dev - allocate a dport by dport device
45
45
* @port: cxl_port that hosts the dport
46
46
* @dport_dev: 'struct device' of the dport
47
47
*
48
48
* Returns the allocated dport on success or ERR_PTR() of -errno on error
49
49
*/
50
-
struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
51
-
struct device *dport_dev)
50
+
struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
51
+
struct device *dport_dev)
52
52
{
53
53
struct cxl_register_map map;
54
54
struct pci_dev *pdev;
···
69
69
device_lock_assert(&port->dev);
70
70
return devm_cxl_add_dport(port, dport_dev, port_num, map.resource);
71
71
}
72
-
EXPORT_SYMBOL_NS_GPL(__devm_cxl_add_dport_by_dev, "CXL");
72
+
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport_by_dev, "CXL");
73
73
74
74
static int cxl_dvsec_mem_range_valid(struct cxl_dev_state *cxlds, int id)
75
75
{
···
86
86
i = 1;
87
87
do {
88
88
rc = pci_read_config_dword(pdev,
89
-
d + CXL_DVSEC_RANGE_SIZE_LOW(id),
89
+
d + PCI_DVSEC_CXL_RANGE_SIZE_LOW(id),
90
90
&temp);
91
91
if (rc)
92
92
return rc;
93
93
94
-
valid = FIELD_GET(CXL_DVSEC_MEM_INFO_VALID, temp);
94
+
valid = FIELD_GET(PCI_DVSEC_CXL_MEM_INFO_VALID, temp);
95
95
if (valid)
96
96
break;
97
97
msleep(1000);
···
121
121
/* Check MEM ACTIVE bit, up to 60s timeout by default */
122
122
for (i = media_ready_timeout; i; i--) {
123
123
rc = pci_read_config_dword(
124
-
pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(id), &temp);
124
+
pdev, d + PCI_DVSEC_CXL_RANGE_SIZE_LOW(id), &temp);
125
125
if (rc)
126
126
return rc;
127
127
128
-
active = FIELD_GET(CXL_DVSEC_MEM_ACTIVE, temp);
128
+
active = FIELD_GET(PCI_DVSEC_CXL_MEM_ACTIVE, temp);
129
129
if (active)
130
130
break;
131
131
msleep(1000);
···
154
154
u16 cap;
155
155
156
156
rc = pci_read_config_word(pdev,
157
-
d + CXL_DVSEC_CAP_OFFSET, &cap);
157
+
d + PCI_DVSEC_CXL_CAP, &cap);
158
158
if (rc)
159
159
return rc;
160
160
161
-
hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap);
161
+
hdm_count = FIELD_GET(PCI_DVSEC_CXL_HDM_COUNT, cap);
162
162
for (i = 0; i < hdm_count; i++) {
163
163
rc = cxl_dvsec_mem_range_valid(cxlds, i);
164
164
if (rc)
···
186
186
u16 ctrl;
187
187
int rc;
188
188
189
-
rc = pci_read_config_word(pdev, d + CXL_DVSEC_CTRL_OFFSET, &ctrl);
189
+
rc = pci_read_config_word(pdev, d + PCI_DVSEC_CXL_CTRL, &ctrl);
190
190
if (rc < 0)
191
191
return rc;
192
192
193
-
if ((ctrl & CXL_DVSEC_MEM_ENABLE) == val)
193
+
if ((ctrl & PCI_DVSEC_CXL_MEM_ENABLE) == val)
194
194
return 1;
195
-
ctrl &= ~CXL_DVSEC_MEM_ENABLE;
195
+
ctrl &= ~PCI_DVSEC_CXL_MEM_ENABLE;
196
196
ctrl |= val;
197
197
198
-
rc = pci_write_config_word(pdev, d + CXL_DVSEC_CTRL_OFFSET, ctrl);
198
+
rc = pci_write_config_word(pdev, d + PCI_DVSEC_CXL_CTRL, ctrl);
199
199
if (rc < 0)
200
200
return rc;
201
201
···
211
211
{
212
212
int rc;
213
213
214
-
rc = cxl_set_mem_enable(cxlds, CXL_DVSEC_MEM_ENABLE);
214
+
rc = cxl_set_mem_enable(cxlds, PCI_DVSEC_CXL_MEM_ENABLE);
215
215
if (rc < 0)
216
216
return rc;
217
217
if (rc > 0)
···
273
273
return -ENXIO;
274
274
}
275
275
276
-
rc = pci_read_config_word(pdev, d + CXL_DVSEC_CAP_OFFSET, &cap);
276
+
rc = pci_read_config_word(pdev, d + PCI_DVSEC_CXL_CAP, &cap);
277
277
if (rc)
278
278
return rc;
279
279
280
-
if (!(cap & CXL_DVSEC_MEM_CAPABLE)) {
280
+
if (!(cap & PCI_DVSEC_CXL_MEM_CAPABLE)) {
281
281
dev_dbg(dev, "Not MEM Capable\n");
282
282
return -ENXIO;
283
283
}
···
288
288
* driver is for a spec defined class code which must be CXL.mem
289
289
* capable, there is no point in continuing to enable CXL.mem.
290
290
*/
291
-
hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap);
291
+
hdm_count = FIELD_GET(PCI_DVSEC_CXL_HDM_COUNT, cap);
292
292
if (!hdm_count || hdm_count > 2)
293
293
return -EINVAL;
294
294
···
297
297
* disabled, and they will remain moot after the HDM Decoder
298
298
* capability is enabled.
299
299
*/
300
-
rc = pci_read_config_word(pdev, d + CXL_DVSEC_CTRL_OFFSET, &ctrl);
300
+
rc = pci_read_config_word(pdev, d + PCI_DVSEC_CXL_CTRL, &ctrl);
301
301
if (rc)
302
302
return rc;
303
303
304
-
info->mem_enabled = FIELD_GET(CXL_DVSEC_MEM_ENABLE, ctrl);
304
+
info->mem_enabled = FIELD_GET(PCI_DVSEC_CXL_MEM_ENABLE, ctrl);
305
305
if (!info->mem_enabled)
306
306
return 0;
307
307
···
314
314
return rc;
315
315
316
316
rc = pci_read_config_dword(
317
-
pdev, d + CXL_DVSEC_RANGE_SIZE_HIGH(i), &temp);
317
+
pdev, d + PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i), &temp);
318
318
if (rc)
319
319
return rc;
320
320
321
321
size = (u64)temp << 32;
322
322
323
323
rc = pci_read_config_dword(
324
-
pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(i), &temp);
324
+
pdev, d + PCI_DVSEC_CXL_RANGE_SIZE_LOW(i), &temp);
325
325
if (rc)
326
326
return rc;
327
327
328
-
size |= temp & CXL_DVSEC_MEM_SIZE_LOW_MASK;
328
+
size |= temp & PCI_DVSEC_CXL_MEM_SIZE_LOW;
329
329
if (!size) {
330
330
continue;
331
331
}
332
332
333
333
rc = pci_read_config_dword(
334
-
pdev, d + CXL_DVSEC_RANGE_BASE_HIGH(i), &temp);
334
+
pdev, d + PCI_DVSEC_CXL_RANGE_BASE_HIGH(i), &temp);
335
335
if (rc)
336
336
return rc;
337
337
338
338
base = (u64)temp << 32;
339
339
340
340
rc = pci_read_config_dword(
341
-
pdev, d + CXL_DVSEC_RANGE_BASE_LOW(i), &temp);
341
+
pdev, d + PCI_DVSEC_CXL_RANGE_BASE_LOW(i), &temp);
342
342
if (rc)
343
343
return rc;
344
344
345
-
base |= temp & CXL_DVSEC_MEM_BASE_LOW_MASK;
345
+
base |= temp & PCI_DVSEC_CXL_MEM_BASE_LOW;
346
346
347
347
info->dvsec_range[ranges++] = (struct range) {
348
348
.start = base,
···
632
632
}
633
633
EXPORT_SYMBOL_NS_GPL(read_cdat_data, "CXL");
634
634
635
-
static void __cxl_handle_cor_ras(struct cxl_dev_state *cxlds,
636
-
void __iomem *ras_base)
637
-
{
638
-
void __iomem *addr;
639
-
u32 status;
640
-
641
-
if (!ras_base)
642
-
return;
643
-
644
-
addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
645
-
status = readl(addr);
646
-
if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) {
647
-
writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr);
648
-
trace_cxl_aer_correctable_error(cxlds->cxlmd, status);
649
-
}
650
-
}
651
-
652
-
static void cxl_handle_endpoint_cor_ras(struct cxl_dev_state *cxlds)
653
-
{
654
-
return __cxl_handle_cor_ras(cxlds, cxlds->regs.ras);
655
-
}
656
-
657
-
/* CXL spec rev3.0 8.2.4.16.1 */
658
-
static void header_log_copy(void __iomem *ras_base, u32 *log)
659
-
{
660
-
void __iomem *addr;
661
-
u32 *log_addr;
662
-
int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32);
663
-
664
-
addr = ras_base + CXL_RAS_HEADER_LOG_OFFSET;
665
-
log_addr = log;
666
-
667
-
for (i = 0; i < log_u32_size; i++) {
668
-
*log_addr = readl(addr);
669
-
log_addr++;
670
-
addr += sizeof(u32);
671
-
}
672
-
}
673
-
674
-
/*
675
-
* Log the state of the RAS status registers and prepare them to log the
676
-
* next error status. Return 1 if reset needed.
677
-
*/
678
-
static bool __cxl_handle_ras(struct cxl_dev_state *cxlds,
679
-
void __iomem *ras_base)
680
-
{
681
-
u32 hl[CXL_HEADERLOG_SIZE_U32];
682
-
void __iomem *addr;
683
-
u32 status;
684
-
u32 fe;
685
-
686
-
if (!ras_base)
687
-
return false;
688
-
689
-
addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
690
-
status = readl(addr);
691
-
if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
692
-
return false;
693
-
694
-
/* If multiple errors, log header points to first error from ctrl reg */
695
-
if (hweight32(status) > 1) {
696
-
void __iomem *rcc_addr =
697
-
ras_base + CXL_RAS_CAP_CONTROL_OFFSET;
698
-
699
-
fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK,
700
-
readl(rcc_addr)));
701
-
} else {
702
-
fe = status;
703
-
}
704
-
705
-
header_log_copy(ras_base, hl);
706
-
trace_cxl_aer_uncorrectable_error(cxlds->cxlmd, status, fe, hl);
707
-
writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr);
708
-
709
-
return true;
710
-
}
711
-
712
-
static bool cxl_handle_endpoint_ras(struct cxl_dev_state *cxlds)
713
-
{
714
-
return __cxl_handle_ras(cxlds, cxlds->regs.ras);
715
-
}
716
-
717
-
#ifdef CONFIG_PCIEAER_CXL
718
-
719
-
static void cxl_dport_map_rch_aer(struct cxl_dport *dport)
720
-
{
721
-
resource_size_t aer_phys;
722
-
struct device *host;
723
-
u16 aer_cap;
724
-
725
-
aer_cap = cxl_rcrb_to_aer(dport->dport_dev, dport->rcrb.base);
726
-
if (aer_cap) {
727
-
host = dport->reg_map.host;
728
-
aer_phys = aer_cap + dport->rcrb.base;
729
-
dport->regs.dport_aer = devm_cxl_iomap_block(host, aer_phys,
730
-
sizeof(struct aer_capability_regs));
731
-
}
732
-
}
733
-
734
-
static void cxl_dport_map_ras(struct cxl_dport *dport)
735
-
{
736
-
struct cxl_register_map *map = &dport->reg_map;
737
-
struct device *dev = dport->dport_dev;
738
-
739
-
if (!map->component_map.ras.valid)
740
-
dev_dbg(dev, "RAS registers not found\n");
741
-
else if (cxl_map_component_regs(map, &dport->regs.component,
742
-
BIT(CXL_CM_CAP_CAP_ID_RAS)))
743
-
dev_dbg(dev, "Failed to map RAS capability.\n");
744
-
}
745
-
746
-
static void cxl_disable_rch_root_ints(struct cxl_dport *dport)
747
-
{
748
-
void __iomem *aer_base = dport->regs.dport_aer;
749
-
u32 aer_cmd_mask, aer_cmd;
750
-
751
-
if (!aer_base)
752
-
return;
753
-
754
-
/*
755
-
* Disable RCH root port command interrupts.
756
-
* CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors
757
-
*
758
-
* This sequence may not be necessary. CXL spec states disabling
759
-
* the root cmd register's interrupts is required. But, PCI spec
760
-
* shows these are disabled by default on reset.
761
-
*/
762
-
aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN |
763
-
PCI_ERR_ROOT_CMD_NONFATAL_EN |
764
-
PCI_ERR_ROOT_CMD_FATAL_EN);
765
-
aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND);
766
-
aer_cmd &= ~aer_cmd_mask;
767
-
writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND);
768
-
}
769
-
770
-
/**
771
-
* cxl_dport_init_ras_reporting - Setup CXL RAS report on this dport
772
-
* @dport: the cxl_dport that needs to be initialized
773
-
* @host: host device for devm operations
774
-
*/
775
-
void cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host)
776
-
{
777
-
dport->reg_map.host = host;
778
-
cxl_dport_map_ras(dport);
779
-
780
-
if (dport->rch) {
781
-
struct pci_host_bridge *host_bridge = to_pci_host_bridge(dport->dport_dev);
782
-
783
-
if (!host_bridge->native_aer)
784
-
return;
785
-
786
-
cxl_dport_map_rch_aer(dport);
787
-
cxl_disable_rch_root_ints(dport);
788
-
}
789
-
}
790
-
EXPORT_SYMBOL_NS_GPL(cxl_dport_init_ras_reporting, "CXL");
791
-
792
-
static void cxl_handle_rdport_cor_ras(struct cxl_dev_state *cxlds,
793
-
struct cxl_dport *dport)
794
-
{
795
-
return __cxl_handle_cor_ras(cxlds, dport->regs.ras);
796
-
}
797
-
798
-
static bool cxl_handle_rdport_ras(struct cxl_dev_state *cxlds,
799
-
struct cxl_dport *dport)
800
-
{
801
-
return __cxl_handle_ras(cxlds, dport->regs.ras);
802
-
}
803
-
804
-
/*
805
-
* Copy the AER capability registers using 32 bit read accesses.
806
-
* This is necessary because RCRB AER capability is MMIO mapped. Clear the
807
-
* status after copying.
808
-
*
809
-
* @aer_base: base address of AER capability block in RCRB
810
-
* @aer_regs: destination for copying AER capability
811
-
*/
812
-
static bool cxl_rch_get_aer_info(void __iomem *aer_base,
813
-
struct aer_capability_regs *aer_regs)
814
-
{
815
-
int read_cnt = sizeof(struct aer_capability_regs) / sizeof(u32);
816
-
u32 *aer_regs_buf = (u32 *)aer_regs;
817
-
int n;
818
-
819
-
if (!aer_base)
820
-
return false;
821
-
822
-
/* Use readl() to guarantee 32-bit accesses */
823
-
for (n = 0; n < read_cnt; n++)
824
-
aer_regs_buf[n] = readl(aer_base + n * sizeof(u32));
825
-
826
-
writel(aer_regs->uncor_status, aer_base + PCI_ERR_UNCOR_STATUS);
827
-
writel(aer_regs->cor_status, aer_base + PCI_ERR_COR_STATUS);
828
-
829
-
return true;
830
-
}
831
-
832
-
/* Get AER severity. Return false if there is no error. */
833
-
static bool cxl_rch_get_aer_severity(struct aer_capability_regs *aer_regs,
834
-
int *severity)
835
-
{
836
-
if (aer_regs->uncor_status & ~aer_regs->uncor_mask) {
837
-
if (aer_regs->uncor_status & PCI_ERR_ROOT_FATAL_RCV)
838
-
*severity = AER_FATAL;
839
-
else
840
-
*severity = AER_NONFATAL;
841
-
return true;
842
-
}
843
-
844
-
if (aer_regs->cor_status & ~aer_regs->cor_mask) {
845
-
*severity = AER_CORRECTABLE;
846
-
return true;
847
-
}
848
-
849
-
return false;
850
-
}
851
-
852
-
static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
853
-
{
854
-
struct pci_dev *pdev = to_pci_dev(cxlds->dev);
855
-
struct aer_capability_regs aer_regs;
856
-
struct cxl_dport *dport;
857
-
int severity;
858
-
859
-
struct cxl_port *port __free(put_cxl_port) =
860
-
cxl_pci_find_port(pdev, &dport);
861
-
if (!port)
862
-
return;
863
-
864
-
if (!cxl_rch_get_aer_info(dport->regs.dport_aer, &aer_regs))
865
-
return;
866
-
867
-
if (!cxl_rch_get_aer_severity(&aer_regs, &severity))
868
-
return;
869
-
870
-
pci_print_aer(pdev, severity, &aer_regs);
871
-
872
-
if (severity == AER_CORRECTABLE)
873
-
cxl_handle_rdport_cor_ras(cxlds, dport);
874
-
else
875
-
cxl_handle_rdport_ras(cxlds, dport);
876
-
}
877
-
878
-
#else
879
-
static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { }
880
-
#endif
881
-
882
-
void cxl_cor_error_detected(struct pci_dev *pdev)
883
-
{
884
-
struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
885
-
struct device *dev = &cxlds->cxlmd->dev;
886
-
887
-
scoped_guard(device, dev) {
888
-
if (!dev->driver) {
889
-
dev_warn(&pdev->dev,
890
-
"%s: memdev disabled, abort error handling\n",
891
-
dev_name(dev));
892
-
return;
893
-
}
894
-
895
-
if (cxlds->rcd)
896
-
cxl_handle_rdport_errors(cxlds);
897
-
898
-
cxl_handle_endpoint_cor_ras(cxlds);
899
-
}
900
-
}
901
-
EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, "CXL");
902
-
903
-
pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
904
-
pci_channel_state_t state)
905
-
{
906
-
struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
907
-
struct cxl_memdev *cxlmd = cxlds->cxlmd;
908
-
struct device *dev = &cxlmd->dev;
909
-
bool ue;
910
-
911
-
scoped_guard(device, dev) {
912
-
if (!dev->driver) {
913
-
dev_warn(&pdev->dev,
914
-
"%s: memdev disabled, abort error handling\n",
915
-
dev_name(dev));
916
-
return PCI_ERS_RESULT_DISCONNECT;
917
-
}
918
-
919
-
if (cxlds->rcd)
920
-
cxl_handle_rdport_errors(cxlds);
921
-
/*
922
-
* A frozen channel indicates an impending reset which is fatal to
923
-
* CXL.mem operation, and will likely crash the system. On the off
924
-
* chance the situation is recoverable dump the status of the RAS
925
-
* capability registers and bounce the active state of the memdev.
926
-
*/
927
-
ue = cxl_handle_endpoint_ras(cxlds);
928
-
}
929
-
930
-
931
-
switch (state) {
932
-
case pci_channel_io_normal:
933
-
if (ue) {
934
-
device_release_driver(dev);
935
-
return PCI_ERS_RESULT_NEED_RESET;
936
-
}
937
-
return PCI_ERS_RESULT_CAN_RECOVER;
938
-
case pci_channel_io_frozen:
939
-
dev_warn(&pdev->dev,
940
-
"%s: frozen state error detected, disable CXL.mem\n",
941
-
dev_name(dev));
942
-
device_release_driver(dev);
943
-
return PCI_ERS_RESULT_NEED_RESET;
944
-
case pci_channel_io_perm_failure:
945
-
dev_warn(&pdev->dev,
946
-
"failure state error detected, request disconnect\n");
947
-
return PCI_ERS_RESULT_DISCONNECT;
948
-
}
949
-
return PCI_ERS_RESULT_NEED_RESET;
950
-
}
951
-
EXPORT_SYMBOL_NS_GPL(cxl_error_detected, "CXL");
952
-
953
635
static int cxl_flit_size(struct pci_dev *pdev)
954
636
{
955
637
if (cxl_pci_flit_256(pdev))
···
750
1068
is_port = false;
751
1069
752
1070
dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
753
-
is_port ? CXL_DVSEC_PORT_GPF : CXL_DVSEC_DEVICE_GPF);
1071
+
is_port ? PCI_DVSEC_CXL_PORT_GPF : PCI_DVSEC_CXL_DEVICE_GPF);
754
1072
if (!dvsec)
755
1073
dev_warn(dev, "%s GPF DVSEC not present\n",
756
1074
is_port ? "Port" : "Device");
···
766
1084
767
1085
switch (phase) {
768
1086
case 1:
769
-
offset = CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET;
770
-
base = CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK;
771
-
scale = CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK;
1087
+
offset = PCI_DVSEC_CXL_PORT_GPF_PHASE_1_CONTROL;
1088
+
base = PCI_DVSEC_CXL_PORT_GPF_PHASE_1_TMO_BASE;
1089
+
scale = PCI_DVSEC_CXL_PORT_GPF_PHASE_1_TMO_SCALE;
772
1090
break;
773
1091
case 2:
774
-
offset = CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET;
775
-
base = CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK;
776
-
scale = CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK;
1092
+
offset = PCI_DVSEC_CXL_PORT_GPF_PHASE_2_CONTROL;
1093
+
base = PCI_DVSEC_CXL_PORT_GPF_PHASE_2_TMO_BASE;
1094
+
scale = PCI_DVSEC_CXL_PORT_GPF_PHASE_2_TMO_SCALE;
777
1095
break;
778
1096
default:
779
1097
return -EINVAL;
+7
-6
drivers/cxl/core/pmem.c
+7
-6
drivers/cxl/core/pmem.c
···
237
237
238
238
/**
239
239
* devm_cxl_add_nvdimm() - add a bridge between a cxl_memdev and an nvdimm
240
-
* @parent_port: parent port for the (to be added) @cxlmd endpoint port
241
-
* @cxlmd: cxl_memdev instance that will perform LIBNVDIMM operations
240
+
* @host: host device for devm operations
241
+
* @port: any port in the CXL topology to find the nvdimm-bridge device
242
+
* @cxlmd: parent of the to be created cxl_nvdimm device
242
243
*
243
244
* Return: 0 on success negative error code on failure.
244
245
*/
245
-
int devm_cxl_add_nvdimm(struct cxl_port *parent_port,
246
+
int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
246
247
struct cxl_memdev *cxlmd)
247
248
{
248
249
struct cxl_nvdimm_bridge *cxl_nvb;
···
251
250
struct device *dev;
252
251
int rc;
253
252
254
-
cxl_nvb = cxl_find_nvdimm_bridge(parent_port);
253
+
cxl_nvb = cxl_find_nvdimm_bridge(port);
255
254
if (!cxl_nvb)
256
255
return -ENODEV;
257
256
···
271
270
if (rc)
272
271
goto err;
273
272
274
-
dev_dbg(&cxlmd->dev, "register %s\n", dev_name(dev));
273
+
dev_dbg(host, "register %s\n", dev_name(dev));
275
274
276
275
/* @cxlmd carries a reference on @cxl_nvb until cxlmd_release_nvdimm */
277
-
return devm_add_action_or_reset(&cxlmd->dev, cxlmd_release_nvdimm, cxlmd);
276
+
return devm_add_action_or_reset(host, cxlmd_release_nvdimm, cxlmd);
278
277
279
278
err:
280
279
put_device(dev);
+112
-71
drivers/cxl/core/port.c
+112
-71
drivers/cxl/core/port.c
···
778
778
return cxl_setup_regs(map);
779
779
}
780
780
781
-
static int cxl_port_setup_regs(struct cxl_port *port,
781
+
int cxl_port_setup_regs(struct cxl_port *port,
782
782
resource_size_t component_reg_phys)
783
783
{
784
784
if (dev_is_platform(port->uport_dev))
···
786
786
return cxl_setup_comp_regs(&port->dev, &port->reg_map,
787
787
component_reg_phys);
788
788
}
789
+
EXPORT_SYMBOL_NS_GPL(cxl_port_setup_regs, "CXL");
789
790
790
791
static int cxl_dport_setup_regs(struct device *host, struct cxl_dport *dport,
791
792
resource_size_t component_reg_phys)
···
823
822
824
823
static void cxl_debugfs_create_dport_dir(struct cxl_dport *dport)
825
824
{
825
+
struct cxl_port *parent = parent_port_of(dport->port);
826
826
struct dentry *dir;
827
827
828
828
if (!einj_cxl_is_initialized())
829
829
return;
830
830
831
831
/*
832
-
* dport_dev needs to be a PCIe port for CXL 2.0+ ports because
833
-
* EINJ expects a dport SBDF to be specified for 2.0 error injection.
832
+
* Protocol error injection is only available for CXL 2.0+ root ports
833
+
* and CXL 1.1 downstream ports
834
834
*/
835
-
if (!dport->rch && !dev_is_pci(dport->dport_dev))
835
+
if (!dport->rch &&
836
+
!(dev_is_pci(dport->dport_dev) && parent && is_cxl_root(parent)))
836
837
return;
837
838
838
839
dir = cxl_debugfs_create_dir(dev_name(dport->dport_dev));
···
957
954
}
958
955
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_port, "CXL");
959
956
960
-
struct cxl_root *devm_cxl_add_root(struct device *host,
961
-
const struct cxl_root_ops *ops)
957
+
struct cxl_root *devm_cxl_add_root(struct device *host)
962
958
{
963
-
struct cxl_root *cxl_root;
964
959
struct cxl_port *port;
965
960
966
961
port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL);
967
962
if (IS_ERR(port))
968
963
return ERR_CAST(port);
969
964
970
-
cxl_root = to_cxl_root(port);
971
-
cxl_root->ops = ops;
972
-
return cxl_root;
965
+
return to_cxl_root(port);
973
966
}
974
967
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_root, "CXL");
975
968
···
1065
1066
return -EBUSY;
1066
1067
}
1067
1068
1069
+
/* Arrange for dport_dev to be valid through remove_dport() */
1070
+
struct device *dev __free(put_device) = get_device(dport->dport_dev);
1071
+
1068
1072
rc = xa_insert(&port->dports, (unsigned long)dport->dport_dev, dport,
1069
1073
GFP_KERNEL);
1070
1074
if (rc)
1071
1075
return rc;
1072
1076
1077
+
retain_and_null_ptr(dev);
1073
1078
port->nr_dports++;
1074
1079
return 0;
1075
1080
}
···
1102
1099
struct cxl_dport *dport = data;
1103
1100
struct cxl_port *port = dport->port;
1104
1101
1102
+
port->nr_dports--;
1105
1103
xa_erase(&port->dports, (unsigned long) dport->dport_dev);
1106
1104
put_device(dport->dport_dev);
1107
1105
}
···
1116
1112
sprintf(link_name, "dport%d", dport->port_id);
1117
1113
sysfs_remove_link(&port->dev.kobj, link_name);
1118
1114
}
1115
+
1116
+
static void free_dport(void *dport)
1117
+
{
1118
+
kfree(dport);
1119
+
}
1120
+
1121
+
/*
1122
+
* Upon return either a group is established with one action (free_dport()), or
1123
+
* no group established and @dport is freed.
1124
+
*/
1125
+
static void *cxl_dport_open_dr_group_or_free(struct cxl_dport *dport)
1126
+
{
1127
+
int rc;
1128
+
struct device *host = dport_to_host(dport);
1129
+
void *group = devres_open_group(host, dport, GFP_KERNEL);
1130
+
1131
+
if (!group) {
1132
+
kfree(dport);
1133
+
return NULL;
1134
+
}
1135
+
1136
+
rc = devm_add_action_or_reset(host, free_dport, dport);
1137
+
if (rc) {
1138
+
devres_release_group(host, group);
1139
+
return NULL;
1140
+
}
1141
+
1142
+
return group;
1143
+
}
1144
+
1145
+
static void cxl_dport_close_dr_group(struct cxl_dport *dport, void *group)
1146
+
{
1147
+
devres_close_group(dport_to_host(dport), group);
1148
+
}
1149
+
1150
+
static void del_dport(struct cxl_dport *dport)
1151
+
{
1152
+
devres_release_group(dport_to_host(dport), dport);
1153
+
}
1154
+
1155
+
/* The dport group id is the dport */
1156
+
DEFINE_FREE(cxl_dport_release_dr_group, void *, if (_T) del_dport(_T))
1119
1157
1120
1158
static struct cxl_dport *
1121
1159
__devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
···
1184
1138
CXL_TARGET_STRLEN)
1185
1139
return ERR_PTR(-EINVAL);
1186
1140
1187
-
dport = devm_kzalloc(host, sizeof(*dport), GFP_KERNEL);
1141
+
dport = kzalloc(sizeof(*dport), GFP_KERNEL);
1188
1142
if (!dport)
1189
1143
return ERR_PTR(-ENOMEM);
1190
1144
1145
+
/* Just enough init to manage the devres group */
1191
1146
dport->dport_dev = dport_dev;
1192
1147
dport->port_id = port_id;
1193
1148
dport->port = port;
1149
+
1150
+
void *dport_dr_group __free(cxl_dport_release_dr_group) =
1151
+
cxl_dport_open_dr_group_or_free(dport);
1152
+
if (!dport_dr_group)
1153
+
return ERR_PTR(-ENOMEM);
1194
1154
1195
1155
if (rcrb == CXL_RESOURCE_NONE) {
1196
1156
rc = cxl_dport_setup_regs(&port->dev, dport,
···
1233
1181
if (rc)
1234
1182
return ERR_PTR(rc);
1235
1183
1236
-
/*
1237
-
* Setup port register if this is the first dport showed up. Having
1238
-
* a dport also means that there is at least 1 active link.
1239
-
*/
1240
-
if (port->nr_dports == 1 &&
1241
-
port->component_reg_phys != CXL_RESOURCE_NONE) {
1242
-
rc = cxl_port_setup_regs(port, port->component_reg_phys);
1243
-
if (rc) {
1244
-
xa_erase(&port->dports, (unsigned long)dport->dport_dev);
1245
-
return ERR_PTR(rc);
1246
-
}
1247
-
port->component_reg_phys = CXL_RESOURCE_NONE;
1248
-
}
1249
-
1250
-
get_device(dport_dev);
1251
1184
rc = devm_add_action_or_reset(host, cxl_dport_remove, dport);
1252
1185
if (rc)
1253
1186
return ERR_PTR(rc);
···
1249
1212
dport->link_latency = cxl_pci_get_latency(to_pci_dev(dport_dev));
1250
1213
1251
1214
cxl_debugfs_create_dport_dir(dport);
1215
+
1216
+
if (!dport->rch)
1217
+
devm_cxl_dport_ras_setup(dport);
1218
+
1219
+
/* keep the group, and mark the end of devm actions */
1220
+
cxl_dport_close_dr_group(dport, no_free_ptr(dport_dr_group));
1252
1221
1253
1222
return dport;
1254
1223
}
···
1482
1439
devm_release_action(port->dev.parent, unregister_port, port);
1483
1440
}
1484
1441
1485
-
static void del_dport(struct cxl_dport *dport)
1486
-
{
1487
-
struct cxl_port *port = dport->port;
1488
-
1489
-
devm_release_action(&port->dev, cxl_dport_unlink, dport);
1490
-
devm_release_action(&port->dev, cxl_dport_remove, dport);
1491
-
devm_kfree(&port->dev, dport);
1492
-
}
1493
-
1494
1442
static void del_dports(struct cxl_port *port)
1495
1443
{
1496
1444
struct cxl_dport *dport;
···
1588
1554
return 0;
1589
1555
1590
1556
port = to_cxl_port(dev);
1557
+
/* Endpoint ports are hosted by memdevs */
1558
+
if (is_cxl_memdev(port->uport_dev))
1559
+
return uport_dev == port->uport_dev->parent;
1591
1560
return uport_dev == port->uport_dev;
1592
1561
}
1593
1562
1594
-
/*
1563
+
/**
1564
+
* find_cxl_port_by_uport - Find a CXL port device companion
1565
+
* @uport_dev: Device that acts as a switch or endpoint in the CXL hierarchy
1566
+
*
1567
+
* In the case of endpoint ports recall that port->uport_dev points to a 'struct
1568
+
* cxl_memdev' device. So, the @uport_dev argument is the parent device of the
1569
+
* 'struct cxl_memdev' in that case.
1570
+
*
1595
1571
* Function takes a device reference on the port device. Caller should do a
1596
1572
* put_device() when done.
1597
1573
*/
···
1641
1597
return 0;
1642
1598
}
1643
1599
1644
-
DEFINE_FREE(del_cxl_dport, struct cxl_dport *, if (!IS_ERR_OR_NULL(_T)) del_dport(_T))
1645
-
static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
1646
-
struct device *dport_dev)
1600
+
void cxl_port_update_decoder_targets(struct cxl_port *port,
1601
+
struct cxl_dport *dport)
1647
1602
{
1648
-
struct cxl_dport *dport;
1649
-
int rc;
1603
+
device_for_each_child(&port->dev, dport, update_decoder_targets);
1604
+
}
1605
+
EXPORT_SYMBOL_NS_GPL(cxl_port_update_decoder_targets, "CXL");
1606
+
1607
+
static bool dport_exists(struct cxl_port *port, struct device *dport_dev)
1608
+
{
1609
+
struct cxl_dport *dport = cxl_find_dport_by_dev(port, dport_dev);
1610
+
1611
+
if (dport) {
1612
+
dev_dbg(&port->dev, "dport%d:%s already exists\n",
1613
+
dport->port_id, dev_name(dport_dev));
1614
+
return true;
1615
+
}
1616
+
1617
+
return false;
1618
+
}
1619
+
1620
+
static struct cxl_dport *probe_dport(struct cxl_port *port,
1621
+
struct device *dport_dev)
1622
+
{
1623
+
struct cxl_driver *drv;
1650
1624
1651
1625
device_lock_assert(&port->dev);
1652
1626
if (!port->dev.driver)
1653
1627
return ERR_PTR(-ENXIO);
1654
1628
1655
-
dport = cxl_find_dport_by_dev(port, dport_dev);
1656
-
if (dport) {
1657
-
dev_dbg(&port->dev, "dport%d:%s already exists\n",
1658
-
dport->port_id, dev_name(dport_dev));
1629
+
if (dport_exists(port, dport_dev))
1659
1630
return ERR_PTR(-EBUSY);
1660
-
}
1661
1631
1662
-
struct cxl_dport *new_dport __free(del_cxl_dport) =
1663
-
devm_cxl_add_dport_by_dev(port, dport_dev);
1664
-
if (IS_ERR(new_dport))
1665
-
return new_dport;
1632
+
drv = container_of(port->dev.driver, struct cxl_driver, drv);
1633
+
if (!drv->add_dport)
1634
+
return ERR_PTR(-ENXIO);
1666
1635
1667
-
cxl_switch_parse_cdat(new_dport);
1668
-
1669
-
if (ida_is_empty(&port->decoder_ida)) {
1670
-
rc = devm_cxl_switch_port_decoders_setup(port);
1671
-
if (rc)
1672
-
return ERR_PTR(rc);
1673
-
dev_dbg(&port->dev, "first dport%d:%s added with decoders\n",
1674
-
new_dport->port_id, dev_name(dport_dev));
1675
-
return no_free_ptr(new_dport);
1676
-
}
1677
-
1678
-
/* New dport added, update the decoder targets */
1679
-
device_for_each_child(&port->dev, new_dport, update_decoder_targets);
1680
-
1681
-
dev_dbg(&port->dev, "dport%d:%s added\n", new_dport->port_id,
1682
-
dev_name(dport_dev));
1683
-
1684
-
return no_free_ptr(new_dport);
1636
+
/* see cxl_port_add_dport() */
1637
+
return drv->add_dport(port, dport_dev);
1685
1638
}
1686
1639
1687
1640
static struct cxl_dport *devm_cxl_create_port(struct device *ep_dev,
···
1725
1684
}
1726
1685
1727
1686
guard(device)(&port->dev);
1728
-
return cxl_port_add_dport(port, dport_dev);
1687
+
return probe_dport(port, dport_dev);
1729
1688
}
1730
1689
1731
1690
static int add_port_attach_ep(struct cxl_memdev *cxlmd,
···
1757
1716
scoped_guard(device, &parent_port->dev) {
1758
1717
parent_dport = cxl_find_dport_by_dev(parent_port, dparent);
1759
1718
if (!parent_dport) {
1760
-
parent_dport = cxl_port_add_dport(parent_port, dparent);
1719
+
parent_dport = probe_dport(parent_port, dparent);
1761
1720
if (IS_ERR(parent_dport))
1762
1721
return PTR_ERR(parent_dport);
1763
1722
}
···
1793
1752
device_lock_assert(&port->dev);
1794
1753
dport = cxl_find_dport_by_dev(port, dport_dev);
1795
1754
if (!dport) {
1796
-
dport = cxl_port_add_dport(port, dport_dev);
1755
+
dport = probe_dport(port, dport_dev);
1797
1756
if (IS_ERR(dport))
1798
1757
return dport;
1799
1758
+198
drivers/cxl/core/ras.c
+198
drivers/cxl/core/ras.c
···
5
5
#include <linux/aer.h>
6
6
#include <cxl/event.h>
7
7
#include <cxlmem.h>
8
+
#include <cxlpci.h>
8
9
#include "trace.h"
9
10
10
11
static void cxl_cper_trace_corr_port_prot_err(struct pci_dev *pdev,
···
126
125
cxl_cper_unregister_prot_err_work(&cxl_cper_prot_err_work);
127
126
cancel_work_sync(&cxl_cper_prot_err_work);
128
127
}
128
+
129
+
static void cxl_dport_map_ras(struct cxl_dport *dport)
130
+
{
131
+
struct cxl_register_map *map = &dport->reg_map;
132
+
struct device *dev = dport->dport_dev;
133
+
134
+
if (!map->component_map.ras.valid)
135
+
dev_dbg(dev, "RAS registers not found\n");
136
+
else if (cxl_map_component_regs(map, &dport->regs.component,
137
+
BIT(CXL_CM_CAP_CAP_ID_RAS)))
138
+
dev_dbg(dev, "Failed to map RAS capability.\n");
139
+
}
140
+
141
+
/**
142
+
* devm_cxl_dport_ras_setup - Setup CXL RAS report on this dport
143
+
* @dport: the cxl_dport that needs to be initialized
144
+
*/
145
+
void devm_cxl_dport_ras_setup(struct cxl_dport *dport)
146
+
{
147
+
dport->reg_map.host = dport_to_host(dport);
148
+
cxl_dport_map_ras(dport);
149
+
}
150
+
151
+
void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
152
+
{
153
+
struct pci_host_bridge *host_bridge;
154
+
155
+
if (!dev_is_pci(dport->dport_dev))
156
+
return;
157
+
158
+
devm_cxl_dport_ras_setup(dport);
159
+
160
+
host_bridge = to_pci_host_bridge(dport->dport_dev);
161
+
if (!host_bridge->native_aer)
162
+
return;
163
+
164
+
cxl_dport_map_rch_aer(dport);
165
+
cxl_disable_rch_root_ints(dport);
166
+
}
167
+
EXPORT_SYMBOL_NS_GPL(devm_cxl_dport_rch_ras_setup, "CXL");
168
+
169
+
void devm_cxl_port_ras_setup(struct cxl_port *port)
170
+
{
171
+
struct cxl_register_map *map = &port->reg_map;
172
+
173
+
if (!map->component_map.ras.valid) {
174
+
dev_dbg(&port->dev, "RAS registers not found\n");
175
+
return;
176
+
}
177
+
178
+
map->host = &port->dev;
179
+
if (cxl_map_component_regs(map, &port->regs,
180
+
BIT(CXL_CM_CAP_CAP_ID_RAS)))
181
+
dev_dbg(&port->dev, "Failed to map RAS capability\n");
182
+
}
183
+
EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL");
184
+
185
+
void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
186
+
{
187
+
void __iomem *addr;
188
+
u32 status;
189
+
190
+
if (!ras_base)
191
+
return;
192
+
193
+
addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
194
+
status = readl(addr);
195
+
if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) {
196
+
writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr);
197
+
trace_cxl_aer_correctable_error(to_cxl_memdev(dev), status);
198
+
}
199
+
}
200
+
201
+
/* CXL spec rev3.0 8.2.4.16.1 */
202
+
static void header_log_copy(void __iomem *ras_base, u32 *log)
203
+
{
204
+
void __iomem *addr;
205
+
u32 *log_addr;
206
+
int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32);
207
+
208
+
addr = ras_base + CXL_RAS_HEADER_LOG_OFFSET;
209
+
log_addr = log;
210
+
211
+
for (i = 0; i < log_u32_size; i++) {
212
+
*log_addr = readl(addr);
213
+
log_addr++;
214
+
addr += sizeof(u32);
215
+
}
216
+
}
217
+
218
+
/*
219
+
* Log the state of the RAS status registers and prepare them to log the
220
+
* next error status. Return 1 if reset needed.
221
+
*/
222
+
bool cxl_handle_ras(struct device *dev, void __iomem *ras_base)
223
+
{
224
+
u32 hl[CXL_HEADERLOG_SIZE_U32];
225
+
void __iomem *addr;
226
+
u32 status;
227
+
u32 fe;
228
+
229
+
if (!ras_base)
230
+
return false;
231
+
232
+
addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
233
+
status = readl(addr);
234
+
if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
235
+
return false;
236
+
237
+
/* If multiple errors, log header points to first error from ctrl reg */
238
+
if (hweight32(status) > 1) {
239
+
void __iomem *rcc_addr =
240
+
ras_base + CXL_RAS_CAP_CONTROL_OFFSET;
241
+
242
+
fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK,
243
+
readl(rcc_addr)));
244
+
} else {
245
+
fe = status;
246
+
}
247
+
248
+
header_log_copy(ras_base, hl);
249
+
trace_cxl_aer_uncorrectable_error(to_cxl_memdev(dev), status, fe, hl);
250
+
writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr);
251
+
252
+
return true;
253
+
}
254
+
255
+
void cxl_cor_error_detected(struct pci_dev *pdev)
256
+
{
257
+
struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
258
+
struct cxl_memdev *cxlmd = cxlds->cxlmd;
259
+
struct device *dev = &cxlds->cxlmd->dev;
260
+
261
+
scoped_guard(device, dev) {
262
+
if (!dev->driver) {
263
+
dev_warn(&pdev->dev,
264
+
"%s: memdev disabled, abort error handling\n",
265
+
dev_name(dev));
266
+
return;
267
+
}
268
+
269
+
if (cxlds->rcd)
270
+
cxl_handle_rdport_errors(cxlds);
271
+
272
+
cxl_handle_cor_ras(&cxlds->cxlmd->dev, cxlmd->endpoint->regs.ras);
273
+
}
274
+
}
275
+
EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, "CXL");
276
+
277
+
pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
278
+
pci_channel_state_t state)
279
+
{
280
+
struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
281
+
struct cxl_memdev *cxlmd = cxlds->cxlmd;
282
+
struct device *dev = &cxlmd->dev;
283
+
bool ue;
284
+
285
+
scoped_guard(device, dev) {
286
+
if (!dev->driver) {
287
+
dev_warn(&pdev->dev,
288
+
"%s: memdev disabled, abort error handling\n",
289
+
dev_name(dev));
290
+
return PCI_ERS_RESULT_DISCONNECT;
291
+
}
292
+
293
+
if (cxlds->rcd)
294
+
cxl_handle_rdport_errors(cxlds);
295
+
/*
296
+
* A frozen channel indicates an impending reset which is fatal to
297
+
* CXL.mem operation, and will likely crash the system. On the off
298
+
* chance the situation is recoverable dump the status of the RAS
299
+
* capability registers and bounce the active state of the memdev.
300
+
*/
301
+
ue = cxl_handle_ras(&cxlds->cxlmd->dev, cxlmd->endpoint->regs.ras);
302
+
}
303
+
304
+
switch (state) {
305
+
case pci_channel_io_normal:
306
+
if (ue) {
307
+
device_release_driver(dev);
308
+
return PCI_ERS_RESULT_NEED_RESET;
309
+
}
310
+
return PCI_ERS_RESULT_CAN_RECOVER;
311
+
case pci_channel_io_frozen:
312
+
dev_warn(&pdev->dev,
313
+
"%s: frozen state error detected, disable CXL.mem\n",
314
+
dev_name(dev));
315
+
device_release_driver(dev);
316
+
return PCI_ERS_RESULT_NEED_RESET;
317
+
case pci_channel_io_perm_failure:
318
+
dev_warn(&pdev->dev,
319
+
"failure state error detected, request disconnect\n");
320
+
return PCI_ERS_RESULT_DISCONNECT;
321
+
}
322
+
return PCI_ERS_RESULT_NEED_RESET;
323
+
}
324
+
EXPORT_SYMBOL_NS_GPL(cxl_error_detected, "CXL");
+121
drivers/cxl/core/ras_rch.c
+121
drivers/cxl/core/ras_rch.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/* Copyright(c) 2025 AMD Corporation. All rights reserved. */
3
+
4
+
#include <linux/types.h>
5
+
#include <linux/aer.h>
6
+
#include "cxl.h"
7
+
#include "core.h"
8
+
#include "cxlmem.h"
9
+
10
+
void cxl_dport_map_rch_aer(struct cxl_dport *dport)
11
+
{
12
+
resource_size_t aer_phys;
13
+
struct device *host;
14
+
u16 aer_cap;
15
+
16
+
aer_cap = cxl_rcrb_to_aer(dport->dport_dev, dport->rcrb.base);
17
+
if (aer_cap) {
18
+
host = dport->reg_map.host;
19
+
aer_phys = aer_cap + dport->rcrb.base;
20
+
dport->regs.dport_aer =
21
+
devm_cxl_iomap_block(host, aer_phys,
22
+
sizeof(struct aer_capability_regs));
23
+
}
24
+
}
25
+
26
+
void cxl_disable_rch_root_ints(struct cxl_dport *dport)
27
+
{
28
+
void __iomem *aer_base = dport->regs.dport_aer;
29
+
u32 aer_cmd_mask, aer_cmd;
30
+
31
+
if (!aer_base)
32
+
return;
33
+
34
+
/*
35
+
* Disable RCH root port command interrupts.
36
+
* CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors
37
+
*
38
+
* This sequence may not be necessary. CXL spec states disabling
39
+
* the root cmd register's interrupts is required. But, PCI spec
40
+
* shows these are disabled by default on reset.
41
+
*/
42
+
aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN |
43
+
PCI_ERR_ROOT_CMD_NONFATAL_EN |
44
+
PCI_ERR_ROOT_CMD_FATAL_EN);
45
+
aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND);
46
+
aer_cmd &= ~aer_cmd_mask;
47
+
writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND);
48
+
}
49
+
50
+
/*
51
+
* Copy the AER capability registers using 32 bit read accesses.
52
+
* This is necessary because RCRB AER capability is MMIO mapped. Clear the
53
+
* status after copying.
54
+
*
55
+
* @aer_base: base address of AER capability block in RCRB
56
+
* @aer_regs: destination for copying AER capability
57
+
*/
58
+
static bool cxl_rch_get_aer_info(void __iomem *aer_base,
59
+
struct aer_capability_regs *aer_regs)
60
+
{
61
+
int read_cnt = sizeof(struct aer_capability_regs) / sizeof(u32);
62
+
u32 *aer_regs_buf = (u32 *)aer_regs;
63
+
int n;
64
+
65
+
if (!aer_base)
66
+
return false;
67
+
68
+
/* Use readl() to guarantee 32-bit accesses */
69
+
for (n = 0; n < read_cnt; n++)
70
+
aer_regs_buf[n] = readl(aer_base + n * sizeof(u32));
71
+
72
+
writel(aer_regs->uncor_status, aer_base + PCI_ERR_UNCOR_STATUS);
73
+
writel(aer_regs->cor_status, aer_base + PCI_ERR_COR_STATUS);
74
+
75
+
return true;
76
+
}
77
+
78
+
/* Get AER severity. Return false if there is no error. */
79
+
static bool cxl_rch_get_aer_severity(struct aer_capability_regs *aer_regs,
80
+
int *severity)
81
+
{
82
+
if (aer_regs->uncor_status & ~aer_regs->uncor_mask) {
83
+
if (aer_regs->uncor_status & PCI_ERR_ROOT_FATAL_RCV)
84
+
*severity = AER_FATAL;
85
+
else
86
+
*severity = AER_NONFATAL;
87
+
return true;
88
+
}
89
+
90
+
if (aer_regs->cor_status & ~aer_regs->cor_mask) {
91
+
*severity = AER_CORRECTABLE;
92
+
return true;
93
+
}
94
+
95
+
return false;
96
+
}
97
+
98
+
void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
99
+
{
100
+
struct pci_dev *pdev = to_pci_dev(cxlds->dev);
101
+
struct aer_capability_regs aer_regs;
102
+
struct cxl_dport *dport;
103
+
int severity;
104
+
105
+
struct cxl_port *port __free(put_cxl_port) =
106
+
cxl_pci_find_port(pdev, &dport);
107
+
if (!port)
108
+
return;
109
+
110
+
if (!cxl_rch_get_aer_info(dport->regs.dport_aer, &aer_regs))
111
+
return;
112
+
113
+
if (!cxl_rch_get_aer_severity(&aer_regs, &severity))
114
+
return;
115
+
116
+
pci_print_aer(pdev, severity, &aer_regs);
117
+
if (severity == AER_CORRECTABLE)
118
+
cxl_handle_cor_ras(&cxlds->cxlmd->dev, dport->regs.ras);
119
+
else
120
+
cxl_handle_ras(&cxlds->cxlmd->dev, dport->regs.ras);
121
+
}
+361
-101
drivers/cxl/core/region.c
+361
-101
drivers/cxl/core/region.c
···
489
489
struct device_attribute *attr,
490
490
const char *buf, size_t len)
491
491
{
492
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
493
-
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
494
492
struct cxl_region *cxlr = to_cxl_region(dev);
493
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
494
+
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
495
495
struct cxl_region_params *p = &cxlr->params;
496
496
unsigned int val, save;
497
497
int rc;
···
552
552
struct device_attribute *attr,
553
553
const char *buf, size_t len)
554
554
{
555
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
556
-
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
557
555
struct cxl_region *cxlr = to_cxl_region(dev);
556
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
557
+
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
558
558
struct cxl_region_params *p = &cxlr->params;
559
559
int rc, val;
560
560
u16 ig;
···
628
628
629
629
static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size)
630
630
{
631
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
631
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
632
632
struct cxl_region_params *p = &cxlr->params;
633
633
struct resource *res;
634
634
u64 remainder = 0;
···
663
663
PTR_ERR(res), &size, cxlrd->res->name, cxlrd->res);
664
664
return PTR_ERR(res);
665
665
}
666
+
667
+
cxlr->hpa_range = DEFINE_RANGE(res->start, res->end);
666
668
667
669
p->res = res;
668
670
p->state = CXL_CONFIG_INTERLEAVE_ACTIVE;
···
701
699
702
700
if (p->state >= CXL_CONFIG_ACTIVE)
703
701
return -EBUSY;
702
+
703
+
cxlr->hpa_range = DEFINE_RANGE(0, -1);
704
704
705
705
cxl_region_iomem_release(cxlr);
706
706
p->state = CXL_CONFIG_IDLE;
···
1097
1093
return 0;
1098
1094
}
1099
1095
1100
-
static void cxl_region_set_lock(struct cxl_region *cxlr,
1101
-
struct cxl_decoder *cxld)
1096
+
static void cxl_region_setup_flags(struct cxl_region *cxlr,
1097
+
struct cxl_decoder *cxld)
1102
1098
{
1103
-
if (!test_bit(CXL_DECODER_F_LOCK, &cxld->flags))
1104
-
return;
1099
+
if (test_bit(CXL_DECODER_F_LOCK, &cxld->flags)) {
1100
+
set_bit(CXL_REGION_F_LOCK, &cxlr->flags);
1101
+
clear_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags);
1102
+
}
1105
1103
1106
-
set_bit(CXL_REGION_F_LOCK, &cxlr->flags);
1107
-
clear_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags);
1104
+
if (test_bit(CXL_DECODER_F_NORMALIZED_ADDRESSING, &cxld->flags))
1105
+
set_bit(CXL_REGION_F_NORMALIZED_ADDRESSING, &cxlr->flags);
1108
1106
}
1109
1107
1110
1108
/**
···
1220
1214
}
1221
1215
}
1222
1216
1223
-
cxl_region_set_lock(cxlr, cxld);
1217
+
cxl_region_setup_flags(cxlr, cxld);
1224
1218
1225
1219
rc = cxl_rr_ep_add(cxl_rr, cxled);
1226
1220
if (rc) {
···
1379
1373
struct cxl_region *cxlr,
1380
1374
struct cxl_endpoint_decoder *cxled)
1381
1375
{
1382
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
1376
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
1383
1377
int parent_iw, parent_ig, ig, iw, rc, pos = cxled->pos;
1384
1378
struct cxl_port *parent_port = to_cxl_port(port->dev.parent);
1385
1379
struct cxl_region_ref *cxl_rr = cxl_rr_load(port, cxlr);
···
1737
1731
}
1738
1732
1739
1733
static int cxl_region_attach_position(struct cxl_region *cxlr,
1740
-
struct cxl_root_decoder *cxlrd,
1741
1734
struct cxl_endpoint_decoder *cxled,
1742
1735
const struct cxl_dport *dport, int pos)
1743
1736
{
1737
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
1744
1738
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
1745
1739
struct cxl_switch_decoder *cxlsd = &cxlrd->cxlsd;
1746
1740
struct cxl_decoder *cxld = &cxlsd->cxld;
···
1880
1874
/**
1881
1875
* cxl_calc_interleave_pos() - calculate an endpoint position in a region
1882
1876
* @cxled: endpoint decoder member of given region
1877
+
* @hpa_range: translated HPA range of the endpoint
1883
1878
*
1884
1879
* The endpoint position is calculated by traversing the topology from
1885
1880
* the endpoint to the root decoder and iteratively applying this
···
1893
1886
* Return: position >= 0 on success
1894
1887
* -ENXIO on failure
1895
1888
*/
1896
-
static int cxl_calc_interleave_pos(struct cxl_endpoint_decoder *cxled)
1889
+
static int cxl_calc_interleave_pos(struct cxl_endpoint_decoder *cxled,
1890
+
struct range *hpa_range)
1897
1891
{
1898
1892
struct cxl_port *iter, *port = cxled_to_port(cxled);
1899
1893
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
1900
-
struct range *range = &cxled->cxld.hpa_range;
1901
1894
int parent_ways = 0, parent_pos = 0, pos = 0;
1902
1895
int rc;
1903
1896
···
1935
1928
if (is_cxl_root(iter))
1936
1929
break;
1937
1930
1938
-
rc = find_pos_and_ways(iter, range, &parent_pos, &parent_ways);
1931
+
rc = find_pos_and_ways(iter, hpa_range, &parent_pos,
1932
+
&parent_ways);
1939
1933
if (rc)
1940
1934
return rc;
1941
1935
···
1946
1938
dev_dbg(&cxlmd->dev,
1947
1939
"decoder:%s parent:%s port:%s range:%#llx-%#llx pos:%d\n",
1948
1940
dev_name(&cxled->cxld.dev), dev_name(cxlmd->dev.parent),
1949
-
dev_name(&port->dev), range->start, range->end, pos);
1941
+
dev_name(&port->dev), hpa_range->start, hpa_range->end, pos);
1950
1942
1951
1943
return pos;
1952
1944
}
···
1959
1951
for (i = 0; i < p->nr_targets; i++) {
1960
1952
struct cxl_endpoint_decoder *cxled = p->targets[i];
1961
1953
1962
-
cxled->pos = cxl_calc_interleave_pos(cxled);
1954
+
cxled->pos = cxl_calc_interleave_pos(cxled, &cxlr->hpa_range);
1963
1955
/*
1964
1956
* Record that sorting failed, but still continue to calc
1965
1957
* cxled->pos so that follow-on code paths can reliably
···
1979
1971
static int cxl_region_attach(struct cxl_region *cxlr,
1980
1972
struct cxl_endpoint_decoder *cxled, int pos)
1981
1973
{
1982
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
1974
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
1983
1975
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
1984
1976
struct cxl_dev_state *cxlds = cxlmd->cxlds;
1985
1977
struct cxl_region_params *p = &cxlr->params;
···
2084
2076
ep_port = cxled_to_port(cxled);
2085
2077
dport = cxl_find_dport_by_dev(root_port,
2086
2078
ep_port->host_bridge);
2087
-
rc = cxl_region_attach_position(cxlr, cxlrd, cxled,
2088
-
dport, i);
2079
+
rc = cxl_region_attach_position(cxlr, cxled, dport, i);
2089
2080
if (rc)
2090
2081
return rc;
2091
2082
}
···
2107
2100
if (rc)
2108
2101
return rc;
2109
2102
2110
-
rc = cxl_region_attach_position(cxlr, cxlrd, cxled, dport, pos);
2103
+
rc = cxl_region_attach_position(cxlr, cxled, dport, pos);
2111
2104
if (rc)
2112
2105
return rc;
2113
2106
···
2143
2136
struct cxl_endpoint_decoder *cxled = p->targets[i];
2144
2137
int test_pos;
2145
2138
2146
-
test_pos = cxl_calc_interleave_pos(cxled);
2139
+
test_pos = cxl_calc_interleave_pos(cxled, &cxlr->hpa_range);
2147
2140
dev_dbg(&cxled->cxld.dev,
2148
2141
"Test cxl_calc_interleave_pos(): %s test_pos:%d cxled->pos:%d\n",
2149
2142
(test_pos == cxled->pos) ? "success" : "fail",
···
2403
2396
2404
2397
static void cxl_region_release(struct device *dev)
2405
2398
{
2406
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
2407
2399
struct cxl_region *cxlr = to_cxl_region(dev);
2400
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
2408
2401
int id = atomic_read(&cxlrd->region_id);
2409
2402
2410
2403
/*
···
2461
2454
for (i = 0; i < p->interleave_ways; i++)
2462
2455
detach_target(cxlr, i);
2463
2456
2457
+
cxlr->hpa_range = DEFINE_RANGE(0, -1);
2458
+
2464
2459
cxl_region_iomem_release(cxlr);
2465
2460
put_device(&cxlr->dev);
2466
2461
}
···
2489
2480
* region id allocations
2490
2481
*/
2491
2482
get_device(dev->parent);
2483
+
cxlr->cxlrd = cxlrd;
2484
+
cxlr->id = id;
2485
+
2492
2486
device_set_pm_not_required(dev);
2493
2487
dev->bus = &cxl_bus_type;
2494
2488
dev->type = &cxl_region_type;
2495
-
cxlr->id = id;
2496
-
cxl_region_set_lock(cxlr, &cxlrd->cxlsd.cxld);
2489
+
cxl_region_setup_flags(cxlr, &cxlrd->cxlsd.cxld);
2497
2490
2498
2491
return cxlr;
2499
2492
}
···
3123
3112
}
3124
3113
EXPORT_SYMBOL_FOR_MODULES(cxl_calculate_hpa_offset, "cxl_translate");
3125
3114
3115
+
static int decode_pos(int region_ways, int hb_ways, int pos, int *pos_port,
3116
+
int *pos_hb)
3117
+
{
3118
+
int devices_per_hb;
3119
+
3120
+
/*
3121
+
* Decode for 3-6-12 way interleaves as defined in the CXL
3122
+
* Spec 4.0 9.13.1.1 Legal Interleaving Configurations.
3123
+
* Region creation should prevent invalid combinations but
3124
+
* sanity check here to avoid a silent bad decode.
3125
+
*/
3126
+
switch (hb_ways) {
3127
+
case 3:
3128
+
if (region_ways != 3 && region_ways != 6 && region_ways != 12)
3129
+
return -EINVAL;
3130
+
break;
3131
+
case 6:
3132
+
if (region_ways != 6 && region_ways != 12)
3133
+
return -EINVAL;
3134
+
break;
3135
+
case 12:
3136
+
if (region_ways != 12)
3137
+
return -EINVAL;
3138
+
break;
3139
+
default:
3140
+
return -EINVAL;
3141
+
}
3142
+
/*
3143
+
* Each host bridge contributes an equal number of endpoints
3144
+
* that are laid out contiguously per host bridge. Modulo
3145
+
* selects the port within a host bridge and division selects
3146
+
* the host bridge position.
3147
+
*/
3148
+
devices_per_hb = region_ways / hb_ways;
3149
+
*pos_port = pos % devices_per_hb;
3150
+
*pos_hb = pos / devices_per_hb;
3151
+
3152
+
return 0;
3153
+
}
3154
+
3155
+
/*
3156
+
* restore_parent() reconstruct the address in parent
3157
+
*
3158
+
* This math, specifically the bitmask creation 'mask = gran - 1' relies
3159
+
* on the CXL Spec requirement that interleave granularity is always a
3160
+
* power of two.
3161
+
*
3162
+
* [mask] isolate the offset with the granularity
3163
+
* [addr & ~mask] remove the offset leaving the aligned portion
3164
+
* [* ways] distribute across all interleave ways
3165
+
* [+ (pos * gran)] add the positional offset
3166
+
* [+ (addr & mask)] restore the masked offset
3167
+
*/
3168
+
static u64 restore_parent(u64 addr, u64 pos, u64 gran, u64 ways)
3169
+
{
3170
+
u64 mask = gran - 1;
3171
+
3172
+
return ((addr & ~mask) * ways) + (pos * gran) + (addr & mask);
3173
+
}
3174
+
3175
+
/*
3176
+
* unaligned_dpa_to_hpa() translates a DPA to HPA when the region resource
3177
+
* start address is not aligned at Host Bridge Interleave Ways * 256MB.
3178
+
*
3179
+
* Unaligned start addresses only occur with MOD3 interleaves. All power-
3180
+
* of-two interleaves are guaranteed aligned.
3181
+
*/
3182
+
static u64 unaligned_dpa_to_hpa(struct cxl_decoder *cxld,
3183
+
struct cxl_region_params *p, int pos, u64 dpa)
3184
+
{
3185
+
int ways_port = p->interleave_ways / cxld->interleave_ways;
3186
+
int gran_port = p->interleave_granularity;
3187
+
int gran_hb = cxld->interleave_granularity;
3188
+
int ways_hb = cxld->interleave_ways;
3189
+
int pos_port, pos_hb, gran_shift;
3190
+
u64 hpa_port = 0;
3191
+
3192
+
/* Decode an endpoint 'pos' into port and host-bridge components */
3193
+
if (decode_pos(p->interleave_ways, ways_hb, pos, &pos_port, &pos_hb)) {
3194
+
dev_dbg(&cxld->dev, "not supported for region ways:%d\n",
3195
+
p->interleave_ways);
3196
+
return ULLONG_MAX;
3197
+
}
3198
+
3199
+
/* Restore the port parent address if needed */
3200
+
if (gran_hb != gran_port)
3201
+
hpa_port = restore_parent(dpa, pos_port, gran_port, ways_port);
3202
+
else
3203
+
hpa_port = dpa;
3204
+
3205
+
/*
3206
+
* Complete the HPA reconstruction by restoring the address as if
3207
+
* each HB position is a candidate. Test against expected pos_hb
3208
+
* to confirm match.
3209
+
*/
3210
+
gran_shift = ilog2(gran_hb);
3211
+
for (int position = 0; position < ways_hb; position++) {
3212
+
u64 shifted, hpa;
3213
+
3214
+
hpa = restore_parent(hpa_port, position, gran_hb, ways_hb);
3215
+
hpa += p->res->start;
3216
+
3217
+
shifted = hpa >> gran_shift;
3218
+
if (do_div(shifted, ways_hb) == pos_hb)
3219
+
return hpa;
3220
+
}
3221
+
3222
+
dev_dbg(&cxld->dev, "fail dpa:%#llx region:%pr pos:%d\n", dpa, p->res,
3223
+
pos);
3224
+
dev_dbg(&cxld->dev, " port-w/g/p:%d/%d/%d hb-w/g/p:%d/%d/%d\n",
3225
+
ways_port, gran_port, pos_port, ways_hb, gran_hb, pos_hb);
3226
+
3227
+
return ULLONG_MAX;
3228
+
}
3229
+
3230
+
static bool region_is_unaligned_mod3(struct cxl_region *cxlr)
3231
+
{
3232
+
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
3233
+
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
3234
+
struct cxl_region_params *p = &cxlr->params;
3235
+
int hbiw = cxld->interleave_ways;
3236
+
u64 rem;
3237
+
3238
+
if (is_power_of_2(hbiw))
3239
+
return false;
3240
+
3241
+
div64_u64_rem(p->res->start, (u64)hbiw * SZ_256M, &rem);
3242
+
3243
+
return (rem != 0);
3244
+
}
3245
+
3126
3246
u64 cxl_dpa_to_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
3127
3247
u64 dpa)
3128
3248
{
3129
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
3249
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
3250
+
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
3130
3251
struct cxl_region_params *p = &cxlr->params;
3131
3252
struct cxl_endpoint_decoder *cxled = NULL;
3132
3253
u64 base, dpa_offset, hpa_offset, hpa;
3254
+
bool unaligned = false;
3133
3255
u16 eig = 0;
3134
3256
u8 eiw = 0;
3135
3257
int pos;
3258
+
3259
+
/*
3260
+
* Conversion between SPA and DPA is not supported in
3261
+
* Normalized Address mode.
3262
+
*/
3263
+
if (test_bit(CXL_REGION_F_NORMALIZED_ADDRESSING, &cxlr->flags))
3264
+
return ULLONG_MAX;
3136
3265
3137
3266
for (int i = 0; i < p->nr_targets; i++) {
3138
3267
if (cxlmd == cxled_to_memdev(p->targets[i])) {
···
3283
3132
if (!cxled)
3284
3133
return ULLONG_MAX;
3285
3134
3286
-
pos = cxled->pos;
3287
-
ways_to_eiw(p->interleave_ways, &eiw);
3288
-
granularity_to_eig(p->interleave_granularity, &eig);
3289
-
3290
3135
base = cxl_dpa_resource_start(cxled);
3291
3136
if (base == RESOURCE_SIZE_MAX)
3292
3137
return ULLONG_MAX;
3293
3138
3294
3139
dpa_offset = dpa - base;
3140
+
3141
+
/* Unaligned calc for MOD3 interleaves not hbiw * 256MB aligned */
3142
+
unaligned = region_is_unaligned_mod3(cxlr);
3143
+
if (unaligned) {
3144
+
hpa = unaligned_dpa_to_hpa(cxld, p, cxled->pos, dpa_offset);
3145
+
if (hpa == ULLONG_MAX)
3146
+
return ULLONG_MAX;
3147
+
3148
+
goto skip_aligned;
3149
+
}
3150
+
/*
3151
+
* Aligned calc for all power-of-2 interleaves and for MOD3
3152
+
* interleaves that are aligned at hbiw * 256MB
3153
+
*/
3154
+
pos = cxled->pos;
3155
+
ways_to_eiw(p->interleave_ways, &eiw);
3156
+
granularity_to_eig(p->interleave_granularity, &eig);
3157
+
3295
3158
hpa_offset = cxl_calculate_hpa_offset(dpa_offset, pos, eiw, eig);
3296
3159
if (hpa_offset == ULLONG_MAX)
3297
3160
return ULLONG_MAX;
3298
3161
3299
3162
/* Apply the hpa_offset to the region base address */
3300
-
hpa = hpa_offset + p->res->start + p->cache_size;
3163
+
hpa = hpa_offset + p->res->start;
3164
+
3165
+
skip_aligned:
3166
+
hpa += p->cache_size;
3301
3167
3302
3168
/* Root decoder translation overrides typical modulo decode */
3303
3169
if (cxlrd->ops.hpa_to_spa)
···
3328
3160
"Addr trans fail: hpa 0x%llx not in region\n", hpa);
3329
3161
return ULLONG_MAX;
3330
3162
}
3331
-
3332
-
/* Simple chunk check, by pos & gran, only applies to modulo decodes */
3333
-
if (!cxlrd->ops.hpa_to_spa && !cxl_is_hpa_in_chunk(hpa, cxlr, pos))
3163
+
/* Chunk check applies to aligned modulo decodes only */
3164
+
if (!unaligned && !cxlrd->ops.hpa_to_spa &&
3165
+
!cxl_is_hpa_in_chunk(hpa, cxlr, pos))
3334
3166
return ULLONG_MAX;
3335
3167
3336
3168
return hpa;
···
3341
3173
u64 dpa;
3342
3174
};
3343
3175
3176
+
static int unaligned_region_offset_to_dpa_result(struct cxl_region *cxlr,
3177
+
u64 offset,
3178
+
struct dpa_result *result)
3179
+
{
3180
+
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
3181
+
struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
3182
+
struct cxl_region_params *p = &cxlr->params;
3183
+
u64 interleave_width, interleave_index;
3184
+
u64 gran, gran_offset, dpa_offset;
3185
+
u64 hpa = p->res->start + offset;
3186
+
u64 tmp = offset;
3187
+
3188
+
/*
3189
+
* Unaligned addresses are not algebraically invertible. Calculate
3190
+
* a dpa_offset independent of the target device and then enumerate
3191
+
* and test that dpa_offset against each candidate endpoint decoder.
3192
+
*/
3193
+
gran = cxld->interleave_granularity;
3194
+
interleave_width = gran * cxld->interleave_ways;
3195
+
interleave_index = div64_u64(offset, interleave_width);
3196
+
gran_offset = do_div(tmp, gran);
3197
+
3198
+
dpa_offset = interleave_index * gran + gran_offset;
3199
+
3200
+
for (int i = 0; i < p->nr_targets; i++) {
3201
+
struct cxl_endpoint_decoder *cxled = p->targets[i];
3202
+
int pos = cxled->pos;
3203
+
u64 test_hpa;
3204
+
3205
+
test_hpa = unaligned_dpa_to_hpa(cxld, p, pos, dpa_offset);
3206
+
if (test_hpa == hpa) {
3207
+
result->cxlmd = cxled_to_memdev(cxled);
3208
+
result->dpa =
3209
+
cxl_dpa_resource_start(cxled) + dpa_offset;
3210
+
return 0;
3211
+
}
3212
+
}
3213
+
dev_err(&cxlr->dev,
3214
+
"failed to resolve HPA %#llx in unaligned MOD3 region\n", hpa);
3215
+
3216
+
return -ENXIO;
3217
+
}
3218
+
3344
3219
static int region_offset_to_dpa_result(struct cxl_region *cxlr, u64 offset,
3345
3220
struct dpa_result *result)
3346
3221
{
3347
3222
struct cxl_region_params *p = &cxlr->params;
3348
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
3223
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
3349
3224
struct cxl_endpoint_decoder *cxled;
3350
3225
u64 hpa_offset = offset;
3351
3226
u64 dpa, dpa_offset;
···
3416
3205
}
3417
3206
hpa_offset -= p->res->start;
3418
3207
}
3208
+
3209
+
if (region_is_unaligned_mod3(cxlr))
3210
+
return unaligned_region_offset_to_dpa_result(cxlr, offset,
3211
+
result);
3419
3212
3420
3213
pos = cxl_calculate_position(hpa_offset, eiw, eig);
3421
3214
if (pos < 0 || pos >= p->nr_targets) {
···
3693
3478
return rc;
3694
3479
}
3695
3480
3696
-
static int match_decoder_by_range(struct device *dev, const void *data)
3481
+
static int match_root_decoder(struct device *dev, const void *data)
3697
3482
{
3698
3483
const struct range *r1, *r2 = data;
3699
-
struct cxl_decoder *cxld;
3484
+
struct cxl_root_decoder *cxlrd;
3700
3485
3701
-
if (!is_switch_decoder(dev))
3486
+
if (!is_root_decoder(dev))
3702
3487
return 0;
3703
3488
3704
-
cxld = to_cxl_decoder(dev);
3705
-
r1 = &cxld->hpa_range;
3489
+
cxlrd = to_cxl_root_decoder(dev);
3490
+
r1 = &cxlrd->cxlsd.cxld.hpa_range;
3491
+
3706
3492
return range_contains(r1, r2);
3707
3493
}
3708
3494
3709
-
static struct cxl_decoder *
3710
-
cxl_port_find_switch_decoder(struct cxl_port *port, struct range *hpa)
3495
+
static int cxl_root_setup_translation(struct cxl_root *cxl_root,
3496
+
struct cxl_region_context *ctx)
3711
3497
{
3712
-
struct device *cxld_dev = device_find_child(&port->dev, hpa,
3713
-
match_decoder_by_range);
3498
+
if (!cxl_root->ops.translation_setup_root)
3499
+
return 0;
3714
3500
3715
-
return cxld_dev ? to_cxl_decoder(cxld_dev) : NULL;
3501
+
return cxl_root->ops.translation_setup_root(cxl_root, ctx);
3716
3502
}
3717
3503
3504
+
/*
3505
+
* Note, when finished with the device, drop the reference with
3506
+
* put_device() or use the put_cxl_root_decoder helper.
3507
+
*/
3718
3508
static struct cxl_root_decoder *
3719
-
cxl_find_root_decoder(struct cxl_endpoint_decoder *cxled)
3509
+
get_cxl_root_decoder(struct cxl_endpoint_decoder *cxled,
3510
+
struct cxl_region_context *ctx)
3720
3511
{
3721
3512
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
3722
3513
struct cxl_port *port = cxled_to_port(cxled);
3723
3514
struct cxl_root *cxl_root __free(put_cxl_root) = find_cxl_root(port);
3724
-
struct cxl_decoder *root, *cxld = &cxled->cxld;
3725
-
struct range *hpa = &cxld->hpa_range;
3515
+
struct device *cxlrd_dev;
3516
+
int rc;
3726
3517
3727
-
root = cxl_port_find_switch_decoder(&cxl_root->port, hpa);
3728
-
if (!root) {
3518
+
/*
3519
+
* Adjust the endpoint's HPA range and interleaving
3520
+
* configuration to the root decoder’s memory space before
3521
+
* setting up the root decoder.
3522
+
*/
3523
+
rc = cxl_root_setup_translation(cxl_root, ctx);
3524
+
if (rc) {
3729
3525
dev_err(cxlmd->dev.parent,
3730
-
"%s:%s no CXL window for range %#llx:%#llx\n",
3731
-
dev_name(&cxlmd->dev), dev_name(&cxld->dev),
3732
-
cxld->hpa_range.start, cxld->hpa_range.end);
3733
-
return NULL;
3526
+
"%s:%s Failed to setup translation for address range %#llx:%#llx\n",
3527
+
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
3528
+
ctx->hpa_range.start, ctx->hpa_range.end);
3529
+
return ERR_PTR(rc);
3734
3530
}
3735
3531
3736
-
return to_cxl_root_decoder(&root->dev);
3532
+
cxlrd_dev = device_find_child(&cxl_root->port.dev, &ctx->hpa_range,
3533
+
match_root_decoder);
3534
+
if (!cxlrd_dev) {
3535
+
dev_err(cxlmd->dev.parent,
3536
+
"%s:%s no CXL window for range %#llx:%#llx\n",
3537
+
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
3538
+
ctx->hpa_range.start, ctx->hpa_range.end);
3539
+
return ERR_PTR(-ENXIO);
3540
+
}
3541
+
3542
+
return to_cxl_root_decoder(cxlrd_dev);
3737
3543
}
3738
3544
3739
3545
static int match_region_by_range(struct device *dev, const void *data)
···
3776
3540
static int cxl_extended_linear_cache_resize(struct cxl_region *cxlr,
3777
3541
struct resource *res)
3778
3542
{
3779
-
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
3543
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
3780
3544
struct cxl_region_params *p = &cxlr->params;
3781
3545
resource_size_t size = resource_size(res);
3782
3546
resource_size_t cache_size, start;
···
3812
3576
}
3813
3577
3814
3578
static int __construct_region(struct cxl_region *cxlr,
3815
-
struct cxl_root_decoder *cxlrd,
3816
-
struct cxl_endpoint_decoder *cxled)
3579
+
struct cxl_region_context *ctx)
3817
3580
{
3581
+
struct cxl_endpoint_decoder *cxled = ctx->cxled;
3582
+
struct cxl_root_decoder *cxlrd = cxlr->cxlrd;
3818
3583
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
3819
-
struct range *hpa = &cxled->cxld.hpa_range;
3584
+
struct range *hpa_range = &ctx->hpa_range;
3820
3585
struct cxl_region_params *p;
3821
3586
struct resource *res;
3822
3587
int rc;
···
3833
3596
}
3834
3597
3835
3598
set_bit(CXL_REGION_F_AUTO, &cxlr->flags);
3599
+
cxlr->hpa_range = *hpa_range;
3836
3600
3837
3601
res = kmalloc(sizeof(*res), GFP_KERNEL);
3838
3602
if (!res)
3839
3603
return -ENOMEM;
3840
3604
3841
-
*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
3605
+
*res = DEFINE_RES_MEM_NAMED(hpa_range->start, range_len(hpa_range),
3842
3606
dev_name(&cxlr->dev));
3843
3607
3844
3608
rc = cxl_extended_linear_cache_resize(cxlr, res);
···
3870
3632
}
3871
3633
3872
3634
p->res = res;
3873
-
p->interleave_ways = cxled->cxld.interleave_ways;
3874
-
p->interleave_granularity = cxled->cxld.interleave_granularity;
3635
+
p->interleave_ways = ctx->interleave_ways;
3636
+
p->interleave_granularity = ctx->interleave_granularity;
3875
3637
p->state = CXL_CONFIG_INTERLEAVE_ACTIVE;
3876
3638
3877
3639
rc = sysfs_update_group(&cxlr->dev.kobj, get_cxl_region_target_group());
···
3891
3653
3892
3654
/* Establish an empty region covering the given HPA range */
3893
3655
static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
3894
-
struct cxl_endpoint_decoder *cxled)
3656
+
struct cxl_region_context *ctx)
3895
3657
{
3658
+
struct cxl_endpoint_decoder *cxled = ctx->cxled;
3896
3659
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
3897
3660
struct cxl_port *port = cxlrd_to_port(cxlrd);
3898
3661
struct cxl_dev_state *cxlds = cxlmd->cxlds;
···
3913
3674
return cxlr;
3914
3675
}
3915
3676
3916
-
rc = __construct_region(cxlr, cxlrd, cxled);
3677
+
rc = __construct_region(cxlr, ctx);
3917
3678
if (rc) {
3918
3679
devm_release_action(port->uport_dev, unregister_region, cxlr);
3919
3680
return ERR_PTR(rc);
···
3923
3684
}
3924
3685
3925
3686
static struct cxl_region *
3926
-
cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
3687
+
cxl_find_region_by_range(struct cxl_root_decoder *cxlrd,
3688
+
struct range *hpa_range)
3927
3689
{
3928
3690
struct device *region_dev;
3929
3691
3930
-
region_dev = device_find_child(&cxlrd->cxlsd.cxld.dev, hpa,
3692
+
region_dev = device_find_child(&cxlrd->cxlsd.cxld.dev, hpa_range,
3931
3693
match_region_by_range);
3932
3694
if (!region_dev)
3933
3695
return NULL;
···
3938
3698
3939
3699
int cxl_add_to_region(struct cxl_endpoint_decoder *cxled)
3940
3700
{
3941
-
struct range *hpa = &cxled->cxld.hpa_range;
3701
+
struct cxl_region_context ctx;
3942
3702
struct cxl_region_params *p;
3943
3703
bool attach = false;
3944
3704
int rc;
3945
3705
3706
+
ctx = (struct cxl_region_context) {
3707
+
.cxled = cxled,
3708
+
.hpa_range = cxled->cxld.hpa_range,
3709
+
.interleave_ways = cxled->cxld.interleave_ways,
3710
+
.interleave_granularity = cxled->cxld.interleave_granularity,
3711
+
};
3712
+
3946
3713
struct cxl_root_decoder *cxlrd __free(put_cxl_root_decoder) =
3947
-
cxl_find_root_decoder(cxled);
3948
-
if (!cxlrd)
3949
-
return -ENXIO;
3714
+
get_cxl_root_decoder(cxled, &ctx);
3715
+
3716
+
if (IS_ERR(cxlrd))
3717
+
return PTR_ERR(cxlrd);
3950
3718
3951
3719
/*
3952
-
* Ensure that if multiple threads race to construct_region() for @hpa
3953
-
* one does the construction and the others add to that.
3720
+
* Ensure that, if multiple threads race to construct_region()
3721
+
* for the HPA range, one does the construction and the others
3722
+
* add to that.
3954
3723
*/
3955
3724
mutex_lock(&cxlrd->range_lock);
3956
3725
struct cxl_region *cxlr __free(put_cxl_region) =
3957
-
cxl_find_region_by_range(cxlrd, hpa);
3726
+
cxl_find_region_by_range(cxlrd, &ctx.hpa_range);
3958
3727
if (!cxlr)
3959
-
cxlr = construct_region(cxlrd, cxled);
3728
+
cxlr = construct_region(cxlrd, &ctx);
3960
3729
mutex_unlock(&cxlrd->range_lock);
3961
3730
3962
3731
rc = PTR_ERR_OR_ZERO(cxlr);
···
4140
3891
DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
4141
3892
cxl_region_debugfs_poison_clear, "%llx\n");
4142
3893
3894
+
static int cxl_region_setup_poison(struct cxl_region *cxlr)
3895
+
{
3896
+
struct device *dev = &cxlr->dev;
3897
+
struct cxl_region_params *p = &cxlr->params;
3898
+
struct dentry *dentry;
3899
+
3900
+
/*
3901
+
* Do not enable poison injection in Normalized Address mode.
3902
+
* Conversion between SPA and DPA is required for this, but it is
3903
+
* not supported in this mode.
3904
+
*/
3905
+
if (test_bit(CXL_REGION_F_NORMALIZED_ADDRESSING, &cxlr->flags))
3906
+
return 0;
3907
+
3908
+
/* Create poison attributes if all memdevs support the capabilities */
3909
+
for (int i = 0; i < p->nr_targets; i++) {
3910
+
struct cxl_endpoint_decoder *cxled = p->targets[i];
3911
+
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
3912
+
3913
+
if (!cxl_memdev_has_poison_cmd(cxlmd, CXL_POISON_ENABLED_INJECT) ||
3914
+
!cxl_memdev_has_poison_cmd(cxlmd, CXL_POISON_ENABLED_CLEAR))
3915
+
return 0;
3916
+
}
3917
+
3918
+
dentry = cxl_debugfs_create_dir(dev_name(dev));
3919
+
debugfs_create_file("inject_poison", 0200, dentry, cxlr,
3920
+
&cxl_poison_inject_fops);
3921
+
debugfs_create_file("clear_poison", 0200, dentry, cxlr,
3922
+
&cxl_poison_clear_fops);
3923
+
3924
+
return devm_add_action_or_reset(dev, remove_debugfs, dentry);
3925
+
}
3926
+
4143
3927
static int cxl_region_can_probe(struct cxl_region *cxlr)
4144
3928
{
4145
3929
struct cxl_region_params *p = &cxlr->params;
···
4202
3920
{
4203
3921
struct cxl_region *cxlr = to_cxl_region(dev);
4204
3922
struct cxl_region_params *p = &cxlr->params;
4205
-
bool poison_supported = true;
4206
3923
int rc;
4207
3924
4208
3925
rc = cxl_region_can_probe(cxlr);
···
4225
3944
if (rc)
4226
3945
return rc;
4227
3946
4228
-
/* Create poison attributes if all memdevs support the capabilities */
4229
-
for (int i = 0; i < p->nr_targets; i++) {
4230
-
struct cxl_endpoint_decoder *cxled = p->targets[i];
4231
-
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
4232
-
4233
-
if (!cxl_memdev_has_poison_cmd(cxlmd, CXL_POISON_ENABLED_INJECT) ||
4234
-
!cxl_memdev_has_poison_cmd(cxlmd, CXL_POISON_ENABLED_CLEAR)) {
4235
-
poison_supported = false;
4236
-
break;
4237
-
}
4238
-
}
4239
-
4240
-
if (poison_supported) {
4241
-
struct dentry *dentry;
4242
-
4243
-
dentry = cxl_debugfs_create_dir(dev_name(dev));
4244
-
debugfs_create_file("inject_poison", 0200, dentry, cxlr,
4245
-
&cxl_poison_inject_fops);
4246
-
debugfs_create_file("clear_poison", 0200, dentry, cxlr,
4247
-
&cxl_poison_clear_fops);
4248
-
rc = devm_add_action_or_reset(dev, remove_debugfs, dentry);
4249
-
if (rc)
4250
-
return rc;
4251
-
}
3947
+
rc = cxl_region_setup_poison(cxlr);
3948
+
if (rc)
3949
+
return rc;
4252
3950
4253
3951
switch (cxlr->mode) {
4254
3952
case CXL_PARTMODE_PMEM:
+7
-7
drivers/cxl/core/regs.c
+7
-7
drivers/cxl/core/regs.c
···
271
271
static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,
272
272
struct cxl_register_map *map)
273
273
{
274
-
u8 reg_type = FIELD_GET(CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK, reg_lo);
275
-
int bar = FIELD_GET(CXL_DVSEC_REG_LOCATOR_BIR_MASK, reg_lo);
274
+
u8 reg_type = FIELD_GET(PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_ID, reg_lo);
275
+
int bar = FIELD_GET(PCI_DVSEC_CXL_REG_LOCATOR_BIR, reg_lo);
276
276
u64 offset = ((u64)reg_hi << 32) |
277
-
(reg_lo & CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK);
277
+
(reg_lo & PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW);
278
278
279
279
if (offset > pci_resource_len(pdev, bar)) {
280
280
dev_warn(&pdev->dev,
···
311
311
};
312
312
313
313
regloc = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
314
-
CXL_DVSEC_REG_LOCATOR);
314
+
PCI_DVSEC_CXL_REG_LOCATOR);
315
315
if (!regloc)
316
316
return -ENXIO;
317
317
318
318
pci_read_config_dword(pdev, regloc + PCI_DVSEC_HEADER1, ®loc_size);
319
-
regloc_size = FIELD_GET(PCI_DVSEC_HEADER1_LENGTH_MASK, regloc_size);
319
+
regloc_size = PCI_DVSEC_HEADER1_LEN(regloc_size);
320
320
321
-
regloc += CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET;
322
-
regblocks = (regloc_size - CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET) / 8;
321
+
regloc += PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1;
322
+
regblocks = (regloc_size - PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1) / 8;
323
323
324
324
for (i = 0; i < regblocks; i++, regloc += 8) {
325
325
u32 reg_lo, reg_hi;
+45
-35
drivers/cxl/cxl.h
+45
-35
drivers/cxl/cxl.h
···
332
332
#define CXL_DECODER_F_TYPE3 BIT(3)
333
333
#define CXL_DECODER_F_LOCK BIT(4)
334
334
#define CXL_DECODER_F_ENABLE BIT(5)
335
-
#define CXL_DECODER_F_MASK GENMASK(5, 0)
335
+
#define CXL_DECODER_F_NORMALIZED_ADDRESSING BIT(6)
336
336
337
337
enum cxl_decoder_type {
338
338
CXL_DECODER_DEVMEM = 2,
···
525
525
*/
526
526
#define CXL_REGION_F_LOCK 2
527
527
528
+
/*
529
+
* Indicate Normalized Addressing. Use it to disable SPA conversion if
530
+
* HPA != SPA and an address translation callback handler does not
531
+
* exist. Flag is needed by AMD Zen5 platforms.
532
+
*/
533
+
#define CXL_REGION_F_NORMALIZED_ADDRESSING 3
534
+
528
535
/**
529
536
* struct cxl_region - CXL region
530
537
* @dev: This region's device
531
538
* @id: This region's id. Id is globally unique across all regions
539
+
* @cxlrd: Region's root decoder
540
+
* @hpa_range: Address range occupied by the region
532
541
* @mode: Operational mode of the mapped capacity
533
542
* @type: Endpoint decoder target type
534
543
* @cxl_nvb: nvdimm bridge for coordinating @cxlr_pmem setup / shutdown
···
551
542
struct cxl_region {
552
543
struct device dev;
553
544
int id;
545
+
struct cxl_root_decoder *cxlrd;
546
+
struct range hpa_range;
554
547
enum cxl_partition_mode mode;
555
548
enum cxl_decoder_type type;
556
549
struct cxl_nvdimm_bridge *cxl_nvb;
···
618
607
* @parent_dport: dport that points to this port in the parent
619
608
* @decoder_ida: allocator for decoder ids
620
609
* @reg_map: component and ras register mapping parameters
610
+
* @regs: mapped component registers
621
611
* @nr_dports: number of entries in @dports
622
612
* @hdm_end: track last allocated HDM decoder instance for allocation ordering
623
613
* @commit_end: cursor to track highest committed decoder for commit ordering
···
640
628
struct cxl_dport *parent_dport;
641
629
struct ida decoder_ida;
642
630
struct cxl_register_map reg_map;
631
+
struct cxl_component_regs regs;
643
632
int nr_dports;
644
633
int hdm_end;
645
634
int commit_end;
···
655
642
resource_size_t component_reg_phys;
656
643
};
657
644
645
+
struct cxl_root;
646
+
647
+
struct cxl_root_ops {
648
+
int (*qos_class)(struct cxl_root *cxl_root,
649
+
struct access_coordinate *coord, int entries,
650
+
int *qos_class);
651
+
int (*translation_setup_root)(struct cxl_root *cxl_root, void *data);
652
+
};
653
+
658
654
/**
659
655
* struct cxl_root - logical collection of root cxl_port items
660
656
*
···
672
650
*/
673
651
struct cxl_root {
674
652
struct cxl_port port;
675
-
const struct cxl_root_ops *ops;
653
+
struct cxl_root_ops ops;
676
654
};
677
655
678
656
static inline struct cxl_root *
···
680
658
{
681
659
return container_of(port, struct cxl_root, port);
682
660
}
683
-
684
-
struct cxl_root_ops {
685
-
int (*qos_class)(struct cxl_root *cxl_root,
686
-
struct access_coordinate *coord, int entries,
687
-
int *qos_class);
688
-
};
689
661
690
662
static inline struct cxl_dport *
691
663
cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
···
794
778
struct device *uport_dev,
795
779
resource_size_t component_reg_phys,
796
780
struct cxl_dport *parent_dport);
797
-
struct cxl_root *devm_cxl_add_root(struct device *host,
798
-
const struct cxl_root_ops *ops);
781
+
struct cxl_root *devm_cxl_add_root(struct device *host);
782
+
int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
783
+
struct cxl_dport *parent_dport);
799
784
struct cxl_root *find_cxl_root(struct cxl_port *port);
800
785
801
786
DEFINE_FREE(put_cxl_root, struct cxl_root *, if (_T) put_device(&_T->port.dev))
···
820
803
struct device *dport_dev, int port_id,
821
804
resource_size_t rcrb);
822
805
823
-
#ifdef CONFIG_PCIEAER_CXL
824
-
void cxl_setup_parent_dport(struct device *host, struct cxl_dport *dport);
825
-
void cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host);
806
+
#ifdef CONFIG_CXL_ATL
807
+
void cxl_setup_prm_address_translation(struct cxl_root *cxl_root);
826
808
#else
827
-
static inline void cxl_dport_init_ras_reporting(struct cxl_dport *dport,
828
-
struct device *host) { }
809
+
static inline
810
+
void cxl_setup_prm_address_translation(struct cxl_root *cxl_root) {}
829
811
#endif
830
812
831
813
struct cxl_decoder *to_cxl_decoder(struct device *dev);
···
864
848
};
865
849
866
850
int devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
867
-
int __devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
868
851
int devm_cxl_endpoint_decoders_setup(struct cxl_port *port);
852
+
void cxl_port_update_decoder_targets(struct cxl_port *port,
853
+
struct cxl_dport *dport);
854
+
int cxl_port_setup_regs(struct cxl_port *port,
855
+
resource_size_t component_reg_phys);
869
856
870
857
struct cxl_dev_state;
871
858
int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds,
···
878
859
879
860
extern const struct bus_type cxl_bus_type;
880
861
862
+
/*
863
+
* Note, add_dport() is expressly for the cxl_port driver. TODO: investigate a
864
+
* type-safe driver model where probe()/remove() take the type of object implied
865
+
* by @id and the add_dport() op only defined for the CXL_DEVICE_PORT driver
866
+
* template.
867
+
*/
881
868
struct cxl_driver {
882
869
const char *name;
883
870
int (*probe)(struct device *dev);
884
871
void (*remove)(struct device *dev);
872
+
struct cxl_dport *(*add_dport)(struct cxl_port *port,
873
+
struct device *dport_dev);
885
874
struct device_driver drv;
886
875
int id;
887
876
};
···
922
895
struct cxl_port *port);
923
896
struct cxl_nvdimm *to_cxl_nvdimm(struct device *dev);
924
897
bool is_cxl_nvdimm(struct device *dev);
925
-
int devm_cxl_add_nvdimm(struct cxl_port *parent_port, struct cxl_memdev *cxlmd);
898
+
int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
899
+
struct cxl_memdev *cxlmd);
926
900
struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_port *port);
927
901
928
902
#ifdef CONFIG_CXL_REGION
···
974
946
bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
975
947
struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
976
948
struct device *dport_dev);
977
-
struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
978
-
struct device *dport_dev);
979
949
980
950
/*
981
951
* Unit test builds overrides this to __weak, find the 'strong' version
···
984
958
#endif
985
959
986
960
u16 cxl_gpf_get_dvsec(struct device *dev);
987
-
988
-
/*
989
-
* Declaration for functions that are mocked by cxl_test that are called by
990
-
* cxl_core. The respective functions are defined as __foo() and called by
991
-
* cxl_core as foo(). The macros below ensures that those functions would
992
-
* exist as foo(). See tools/testing/cxl/cxl_core_exports.c and
993
-
* tools/testing/cxl/exports.h for setting up the mock functions. The dance
994
-
* is done to avoid a circular dependency where cxl_core calls a function that
995
-
* ends up being a mock function and goes to * cxl_test where it calls a
996
-
* cxl_core function.
997
-
*/
998
-
#ifndef CXL_TEST_ENABLE
999
-
#define DECLARE_TESTABLE(x) __##x
1000
-
#define devm_cxl_add_dport_by_dev DECLARE_TESTABLE(devm_cxl_add_dport_by_dev)
1001
-
#define devm_cxl_switch_port_decoders_setup DECLARE_TESTABLE(devm_cxl_switch_port_decoders_setup)
1002
-
#endif
1003
961
1004
962
#endif /* __CXL_H__ */
+13
-8
drivers/cxl/cxlmem.h
+13
-8
drivers/cxl/cxlmem.h
···
34
34
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
35
35
CXLMDEV_RESET_NEEDED_NOT)
36
36
37
+
struct cxl_memdev_attach {
38
+
int (*probe)(struct cxl_memdev *cxlmd);
39
+
};
40
+
37
41
/**
38
42
* struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
39
43
* @dev: driver core device object
···
47
43
* @cxl_nvb: coordinate removal of @cxl_nvd if present
48
44
* @cxl_nvd: optional bridge to an nvdimm if the device supports pmem
49
45
* @endpoint: connection to the CXL port topology for this memory device
46
+
* @attach: creator of this memdev depends on CXL link attach to operate
50
47
* @id: id number of this memdev instance.
51
48
* @depth: endpoint port depth
52
49
* @scrub_cycle: current scrub cycle set for this device
···
64
59
struct cxl_nvdimm_bridge *cxl_nvb;
65
60
struct cxl_nvdimm *cxl_nvd;
66
61
struct cxl_port *endpoint;
62
+
const struct cxl_memdev_attach *attach;
67
63
int id;
68
64
int depth;
69
65
u8 scrub_cycle;
70
66
int scrub_region_id;
71
-
void *err_rec_array;
67
+
struct cxl_mem_err_rec *err_rec_array;
72
68
};
73
69
74
70
static inline struct cxl_memdev *to_cxl_memdev(struct device *dev)
···
101
95
return is_cxl_memdev(port->uport_dev);
102
96
}
103
97
104
-
struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
105
-
struct cxl_dev_state *cxlds);
98
+
struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds,
99
+
const struct cxl_memdev_attach *attach);
100
+
struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds,
101
+
const struct cxl_memdev_attach *attach);
106
102
int devm_cxl_sanitize_setup_notifier(struct device *host,
107
103
struct cxl_memdev *cxlmd);
108
104
struct cxl_memdev_state;
···
423
415
* @dev: The device associated with this CXL state
424
416
* @cxlmd: The device representing the CXL.mem capabilities of @dev
425
417
* @reg_map: component and ras register mapping parameters
426
-
* @regs: Parsed register blocks
418
+
* @regs: Class device "Device" registers
427
419
* @cxl_dvsec: Offset to the PCIe device DVSEC
428
420
* @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
429
421
* @media_ready: Indicate whether the device media is usable
···
439
431
struct device *dev;
440
432
struct cxl_memdev *cxlmd;
441
433
struct cxl_register_map reg_map;
442
-
struct cxl_regs regs;
434
+
struct cxl_device_regs regs;
443
435
int cxl_dvsec;
444
436
bool rcd;
445
437
bool media_ready;
···
885
877
int devm_cxl_region_edac_register(struct cxl_region *cxlr);
886
878
int cxl_store_rec_gen_media(struct cxl_memdev *cxlmd, union cxl_event *evt);
887
879
int cxl_store_rec_dram(struct cxl_memdev *cxlmd, union cxl_event *evt);
888
-
void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd);
889
880
#else
890
881
static inline int devm_cxl_memdev_edac_register(struct cxl_memdev *cxlmd)
891
882
{ return 0; }
···
896
889
static inline int cxl_store_rec_dram(struct cxl_memdev *cxlmd,
897
890
union cxl_event *evt)
898
891
{ return 0; }
899
-
static inline void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd)
900
-
{ return; }
901
892
#endif
902
893
903
894
#ifdef CONFIG_CXL_SUSPEND
+22
-53
drivers/cxl/cxlpci.h
+22
-53
drivers/cxl/cxlpci.h
···
8
8
#define CXL_MEMORY_PROGIF 0x10
9
9
10
10
/*
11
-
* See section 8.1 Configuration Space Registers in the CXL 2.0
12
-
* Specification. Names are taken straight from the specification with "CXL" and
13
-
* "DVSEC" redundancies removed. When obvious, abbreviations may be used.
14
-
*/
15
-
#define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20)
16
-
17
-
/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
18
-
#define CXL_DVSEC_PCIE_DEVICE 0
19
-
#define CXL_DVSEC_CAP_OFFSET 0xA
20
-
#define CXL_DVSEC_MEM_CAPABLE BIT(2)
21
-
#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4)
22
-
#define CXL_DVSEC_CTRL_OFFSET 0xC
23
-
#define CXL_DVSEC_MEM_ENABLE BIT(2)
24
-
#define CXL_DVSEC_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10))
25
-
#define CXL_DVSEC_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10))
26
-
#define CXL_DVSEC_MEM_INFO_VALID BIT(0)
27
-
#define CXL_DVSEC_MEM_ACTIVE BIT(1)
28
-
#define CXL_DVSEC_MEM_SIZE_LOW_MASK GENMASK(31, 28)
29
-
#define CXL_DVSEC_RANGE_BASE_HIGH(i) (0x20 + (i * 0x10))
30
-
#define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + (i * 0x10))
31
-
#define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28)
32
-
33
-
#define CXL_DVSEC_RANGE_MAX 2
34
-
35
-
/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */
36
-
#define CXL_DVSEC_FUNCTION_MAP 2
37
-
38
-
/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */
39
-
#define CXL_DVSEC_PORT_EXTENSIONS 3
40
-
41
-
/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */
42
-
#define CXL_DVSEC_PORT_GPF 4
43
-
#define CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET 0x0C
44
-
#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK GENMASK(3, 0)
45
-
#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK GENMASK(11, 8)
46
-
#define CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET 0xE
47
-
#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK GENMASK(3, 0)
48
-
#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK GENMASK(11, 8)
49
-
50
-
/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */
51
-
#define CXL_DVSEC_DEVICE_GPF 5
52
-
53
-
/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */
54
-
#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7
55
-
56
-
/* CXL 2.0 8.1.9: Register Locator DVSEC */
57
-
#define CXL_DVSEC_REG_LOCATOR 8
58
-
#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC
59
-
#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0)
60
-
#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8)
61
-
#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16)
62
-
63
-
/*
64
11
* NOTE: Currently all the functions which are enabled for CXL require their
65
12
* vectors to be in the first 16. Use this as the default max.
66
13
*/
···
76
129
77
130
struct cxl_dev_state;
78
131
void read_cdat_data(struct cxl_port *port);
132
+
133
+
#ifdef CONFIG_CXL_RAS
79
134
void cxl_cor_error_detected(struct pci_dev *pdev);
80
135
pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
81
136
pci_channel_state_t state);
137
+
void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport);
138
+
void devm_cxl_port_ras_setup(struct cxl_port *port);
139
+
#else
140
+
static inline void cxl_cor_error_detected(struct pci_dev *pdev) { }
141
+
142
+
static inline pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
143
+
pci_channel_state_t state)
144
+
{
145
+
return PCI_ERS_RESULT_NONE;
146
+
}
147
+
148
+
static inline void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
149
+
{
150
+
}
151
+
152
+
static inline void devm_cxl_port_ras_setup(struct cxl_port *port)
153
+
{
154
+
}
155
+
#endif
156
+
82
157
#endif /* __CXL_PCI_H__ */
+31
-46
drivers/cxl/mem.c
+31
-46
drivers/cxl/mem.c
···
45
45
return 0;
46
46
}
47
47
48
-
static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
49
-
struct cxl_dport *parent_dport)
50
-
{
51
-
struct cxl_port *parent_port = parent_dport->port;
52
-
struct cxl_port *endpoint, *iter, *down;
53
-
int rc;
54
-
55
-
/*
56
-
* Now that the path to the root is established record all the
57
-
* intervening ports in the chain.
58
-
*/
59
-
for (iter = parent_port, down = NULL; !is_cxl_root(iter);
60
-
down = iter, iter = to_cxl_port(iter->dev.parent)) {
61
-
struct cxl_ep *ep;
62
-
63
-
ep = cxl_ep_load(iter, cxlmd);
64
-
ep->next = down;
65
-
}
66
-
67
-
/* Note: endpoint port component registers are derived from @cxlds */
68
-
endpoint = devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE,
69
-
parent_dport);
70
-
if (IS_ERR(endpoint))
71
-
return PTR_ERR(endpoint);
72
-
73
-
rc = cxl_endpoint_autoremove(cxlmd, endpoint);
74
-
if (rc)
75
-
return rc;
76
-
77
-
if (!endpoint->dev.driver) {
78
-
dev_err(&cxlmd->dev, "%s failed probe\n",
79
-
dev_name(&endpoint->dev));
80
-
return -ENXIO;
81
-
}
82
-
83
-
return 0;
84
-
}
85
-
86
48
static int cxl_debugfs_poison_inject(void *data, u64 dpa)
87
49
{
88
50
struct cxl_memdev *cxlmd = data;
···
115
153
}
116
154
117
155
if (cxl_pmem_size(cxlds) && IS_ENABLED(CONFIG_CXL_PMEM)) {
118
-
rc = devm_cxl_add_nvdimm(parent_port, cxlmd);
156
+
rc = devm_cxl_add_nvdimm(dev, parent_port, cxlmd);
119
157
if (rc) {
120
158
if (rc == -ENODEV)
121
159
dev_info(dev, "PMEM disabled by platform\n");
···
128
166
else
129
167
endpoint_parent = &parent_port->dev;
130
168
131
-
cxl_dport_init_ras_reporting(dport, dev);
132
-
133
169
scoped_guard(device, endpoint_parent) {
134
170
if (!endpoint_parent->driver) {
135
171
dev_err(dev, "CXL port topology %s not enabled\n",
···
136
176
}
137
177
138
178
rc = devm_cxl_add_endpoint(endpoint_parent, cxlmd, dport);
179
+
if (rc)
180
+
return rc;
181
+
}
182
+
183
+
if (cxlmd->attach) {
184
+
rc = cxlmd->attach->probe(cxlmd);
139
185
if (rc)
140
186
return rc;
141
187
}
···
166
200
cxl_mem_active_inc();
167
201
return devm_add_action_or_reset(dev, enable_suspend, NULL);
168
202
}
203
+
204
+
/**
205
+
* devm_cxl_add_memdev - Add a CXL memory device
206
+
* @cxlds: CXL device state to associate with the memdev
207
+
* @attach: Caller depends on CXL topology attachment
208
+
*
209
+
* Upon return the device will have had a chance to attach to the
210
+
* cxl_mem driver, but may fail to attach if the CXL topology is not ready
211
+
* (hardware CXL link down, or software platform CXL root not attached).
212
+
*
213
+
* When @attach is NULL it indicates the caller wants the memdev to remain
214
+
* registered even if it does not immediately attach to the CXL hierarchy. When
215
+
* @attach is provided a cxl_mem_probe() failure leads to failure of this routine.
216
+
*
217
+
* The parent of the resulting device and the devm context for allocations is
218
+
* @cxlds->dev.
219
+
*/
220
+
struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds,
221
+
const struct cxl_memdev_attach *attach)
222
+
{
223
+
return __devm_cxl_add_memdev(cxlds, attach);
224
+
}
225
+
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
169
226
170
227
static ssize_t trigger_poison_list_store(struct device *dev,
171
228
struct device_attribute *attr,
···
237
248
.probe = cxl_mem_probe,
238
249
.id = CXL_DEVICE_MEMORY_EXPANDER,
239
250
.drv = {
251
+
.probe_type = PROBE_FORCE_SYNCHRONOUS,
240
252
.dev_groups = cxl_mem_groups,
241
253
},
242
254
};
···
248
258
MODULE_LICENSE("GPL v2");
249
259
MODULE_IMPORT_NS("CXL");
250
260
MODULE_ALIAS_CXL(CXL_DEVICE_MEMORY_EXPANDER);
251
-
/*
252
-
* create_endpoint() wants to validate port driver attach immediately after
253
-
* endpoint registration.
254
-
*/
255
-
MODULE_SOFTDEP("pre: cxl_port");
+3
-64
drivers/cxl/pci.c
+3
-64
drivers/cxl/pci.c
···
535
535
return cxl_setup_regs(map);
536
536
}
537
537
538
-
static int cxl_pci_ras_unmask(struct pci_dev *pdev)
539
-
{
540
-
struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
541
-
void __iomem *addr;
542
-
u32 orig_val, val, mask;
543
-
u16 cap;
544
-
int rc;
545
-
546
-
if (!cxlds->regs.ras) {
547
-
dev_dbg(&pdev->dev, "No RAS registers.\n");
548
-
return 0;
549
-
}
550
-
551
-
/* BIOS has PCIe AER error control */
552
-
if (!pcie_aer_is_native(pdev))
553
-
return 0;
554
-
555
-
rc = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL, &cap);
556
-
if (rc)
557
-
return rc;
558
-
559
-
if (cap & PCI_EXP_DEVCTL_URRE) {
560
-
addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_MASK_OFFSET;
561
-
orig_val = readl(addr);
562
-
563
-
mask = CXL_RAS_UNCORRECTABLE_MASK_MASK |
564
-
CXL_RAS_UNCORRECTABLE_MASK_F256B_MASK;
565
-
val = orig_val & ~mask;
566
-
writel(val, addr);
567
-
dev_dbg(&pdev->dev,
568
-
"Uncorrectable RAS Errors Mask: %#x -> %#x\n",
569
-
orig_val, val);
570
-
}
571
-
572
-
if (cap & PCI_EXP_DEVCTL_CERE) {
573
-
addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_MASK_OFFSET;
574
-
orig_val = readl(addr);
575
-
val = orig_val & ~CXL_RAS_CORRECTABLE_MASK_MASK;
576
-
writel(val, addr);
577
-
dev_dbg(&pdev->dev, "Correctable RAS Errors Mask: %#x -> %#x\n",
578
-
orig_val, val);
579
-
}
580
-
581
-
return 0;
582
-
}
583
-
584
538
static void free_event_buf(void *buf)
585
539
{
586
540
kvfree(buf);
···
866
912
unsigned int i;
867
913
bool irq_avail;
868
914
869
-
/*
870
-
* Double check the anonymous union trickery in struct cxl_regs
871
-
* FIXME switch to struct_group()
872
-
*/
873
-
BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
874
-
offsetof(struct cxl_regs, device_regs.memdev));
875
-
876
915
rc = pcim_enable_device(pdev);
877
916
if (rc)
878
917
return rc;
···
880
933
cxlds->rcd = is_cxl_restricted(pdev);
881
934
cxlds->serial = pci_get_dsn(pdev);
882
935
cxlds->cxl_dvsec = pci_find_dvsec_capability(
883
-
pdev, PCI_VENDOR_ID_CXL, CXL_DVSEC_PCIE_DEVICE);
936
+
pdev, PCI_VENDOR_ID_CXL, PCI_DVSEC_CXL_DEVICE);
884
937
if (!cxlds->cxl_dvsec)
885
938
dev_warn(&pdev->dev,
886
939
"Device DVSEC not present, skip CXL.mem init\n");
···
889
942
if (rc)
890
943
return rc;
891
944
892
-
rc = cxl_map_device_regs(&map, &cxlds->regs.device_regs);
945
+
rc = cxl_map_device_regs(&map, &cxlds->regs);
893
946
if (rc)
894
947
return rc;
895
948
···
903
956
dev_warn(&pdev->dev, "No component registers (%d)\n", rc);
904
957
else if (!cxlds->reg_map.component_map.ras.valid)
905
958
dev_dbg(&pdev->dev, "RAS registers not found\n");
906
-
907
-
rc = cxl_map_component_regs(&cxlds->reg_map, &cxlds->regs.component,
908
-
BIT(CXL_CM_CAP_CAP_ID_RAS));
909
-
if (rc)
910
-
dev_dbg(&pdev->dev, "Failed to map RAS capability.\n");
911
959
912
960
rc = cxl_pci_type3_init_mailbox(cxlds);
913
961
if (rc)
···
948
1006
if (rc)
949
1007
dev_dbg(&pdev->dev, "No CXL Features discovered\n");
950
1008
951
-
cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds);
1009
+
cxlmd = devm_cxl_add_memdev(cxlds, NULL);
952
1010
if (IS_ERR(cxlmd))
953
1011
return PTR_ERR(cxlmd);
954
1012
···
993
1051
rc = cxl_event_config(host_bridge, mds, irq_avail);
994
1052
if (rc)
995
1053
return rc;
996
-
997
-
if (cxl_pci_ras_unmask(pdev))
998
-
dev_dbg(&pdev->dev, "No RAS reporting unmasked\n");
999
1054
1000
1055
pci_save_state(pdev);
1001
1056
+162
drivers/cxl/port.c
+162
drivers/cxl/port.c
···
1
1
// SPDX-License-Identifier: GPL-2.0-only
2
2
/* Copyright(c) 2022 Intel Corporation. All rights reserved. */
3
+
#include <linux/aer.h>
3
4
#include <linux/device.h>
4
5
#include <linux/module.h>
5
6
#include <linux/slab.h>
···
69
68
return 0;
70
69
}
71
70
71
+
static int cxl_ras_unmask(struct cxl_port *port)
72
+
{
73
+
struct pci_dev *pdev;
74
+
void __iomem *addr;
75
+
u32 orig_val, val, mask;
76
+
u16 cap;
77
+
int rc;
78
+
79
+
if (!dev_is_pci(port->uport_dev))
80
+
return 0;
81
+
pdev = to_pci_dev(port->uport_dev);
82
+
83
+
if (!port->regs.ras) {
84
+
pci_dbg(pdev, "No RAS registers.\n");
85
+
return 0;
86
+
}
87
+
88
+
/* BIOS has PCIe AER error control */
89
+
if (!pcie_aer_is_native(pdev))
90
+
return 0;
91
+
92
+
rc = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL, &cap);
93
+
if (rc)
94
+
return rc;
95
+
96
+
if (cap & PCI_EXP_DEVCTL_URRE) {
97
+
addr = port->regs.ras + CXL_RAS_UNCORRECTABLE_MASK_OFFSET;
98
+
orig_val = readl(addr);
99
+
100
+
mask = CXL_RAS_UNCORRECTABLE_MASK_MASK |
101
+
CXL_RAS_UNCORRECTABLE_MASK_F256B_MASK;
102
+
val = orig_val & ~mask;
103
+
writel(val, addr);
104
+
pci_dbg(pdev, "Uncorrectable RAS Errors Mask: %#x -> %#x\n",
105
+
orig_val, val);
106
+
}
107
+
108
+
if (cap & PCI_EXP_DEVCTL_CERE) {
109
+
addr = port->regs.ras + CXL_RAS_CORRECTABLE_MASK_OFFSET;
110
+
orig_val = readl(addr);
111
+
val = orig_val & ~CXL_RAS_CORRECTABLE_MASK_MASK;
112
+
writel(val, addr);
113
+
pci_dbg(pdev, "Correctable RAS Errors Mask: %#x -> %#x\n",
114
+
orig_val, val);
115
+
}
116
+
117
+
return 0;
118
+
}
119
+
72
120
static int cxl_endpoint_port_probe(struct cxl_port *port)
73
121
{
74
122
struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
123
+
struct cxl_dport *dport = port->parent_dport;
75
124
int rc;
76
125
77
126
/* Cache the data early to ensure is_visible() works */
···
136
85
rc = devm_cxl_endpoint_decoders_setup(port);
137
86
if (rc)
138
87
return rc;
88
+
89
+
/*
90
+
* With VH (CXL Virtual Host) topology the cxl_port::add_dport() method
91
+
* handles RAS setup for downstream ports. With RCH (CXL Restricted CXL
92
+
* Host) topologies the downstream port is enumerated early by platform
93
+
* firmware, but the RCRB (root complex register block) is not mapped
94
+
* until after the cxl_pci driver attaches to the RCIeP (root complex
95
+
* integrated endpoint).
96
+
*/
97
+
if (dport->rch)
98
+
devm_cxl_dport_rch_ras_setup(dport);
99
+
100
+
devm_cxl_port_ras_setup(port);
101
+
if (cxl_ras_unmask(port))
102
+
dev_dbg(&port->dev, "failed to unmask RAS interrupts\n");
139
103
140
104
/*
141
105
* Now that all endpoint decoders are successfully enumerated, try to
···
217
151
NULL,
218
152
};
219
153
154
+
/* note this implicitly casts the group back to its @port */
155
+
DEFINE_FREE(cxl_port_release_dr_group, struct cxl_port *,
156
+
if (_T) devres_release_group(&_T->dev, _T))
157
+
158
+
static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
159
+
struct device *dport_dev)
160
+
{
161
+
struct cxl_dport *dport;
162
+
int rc;
163
+
164
+
/* Temp group for all "first dport" and "per dport" setup actions */
165
+
void *port_dr_group __free(cxl_port_release_dr_group) =
166
+
devres_open_group(&port->dev, port, GFP_KERNEL);
167
+
if (!port_dr_group)
168
+
return ERR_PTR(-ENOMEM);
169
+
170
+
if (port->nr_dports == 0) {
171
+
/*
172
+
* Some host bridges are known to not have component regsisters
173
+
* available until a root port has trained CXL. Perform that
174
+
* setup now.
175
+
*/
176
+
rc = cxl_port_setup_regs(port, port->component_reg_phys);
177
+
if (rc)
178
+
return ERR_PTR(rc);
179
+
180
+
rc = devm_cxl_switch_port_decoders_setup(port);
181
+
if (rc)
182
+
return ERR_PTR(rc);
183
+
184
+
/*
185
+
* RAS setup is optional, either driver operation can continue
186
+
* on failure, or the device does not implement RAS registers.
187
+
*/
188
+
devm_cxl_port_ras_setup(port);
189
+
}
190
+
191
+
dport = devm_cxl_add_dport_by_dev(port, dport_dev);
192
+
if (IS_ERR(dport))
193
+
return dport;
194
+
195
+
/* This group was only needed for early exit above */
196
+
devres_remove_group(&port->dev, no_free_ptr(port_dr_group));
197
+
198
+
cxl_switch_parse_cdat(dport);
199
+
200
+
/* New dport added, update the decoder targets */
201
+
cxl_port_update_decoder_targets(port, dport);
202
+
203
+
dev_dbg(&port->dev, "dport%d:%s added\n", dport->port_id,
204
+
dev_name(dport_dev));
205
+
206
+
return dport;
207
+
}
208
+
220
209
static struct cxl_driver cxl_port_driver = {
221
210
.name = "cxl_port",
222
211
.probe = cxl_port_probe,
212
+
.add_dport = cxl_port_add_dport,
223
213
.id = CXL_DEVICE_PORT,
224
214
.drv = {
215
+
.probe_type = PROBE_FORCE_SYNCHRONOUS,
225
216
.dev_groups = cxl_port_attribute_groups,
226
217
},
227
218
};
219
+
220
+
int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
221
+
struct cxl_dport *parent_dport)
222
+
{
223
+
struct cxl_port *parent_port = parent_dport->port;
224
+
struct cxl_port *endpoint, *iter, *down;
225
+
int rc;
226
+
227
+
/*
228
+
* Now that the path to the root is established record all the
229
+
* intervening ports in the chain.
230
+
*/
231
+
for (iter = parent_port, down = NULL; !is_cxl_root(iter);
232
+
down = iter, iter = to_cxl_port(iter->dev.parent)) {
233
+
struct cxl_ep *ep;
234
+
235
+
ep = cxl_ep_load(iter, cxlmd);
236
+
ep->next = down;
237
+
}
238
+
239
+
/* Note: endpoint port component registers are derived from @cxlds */
240
+
endpoint = devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE,
241
+
parent_dport);
242
+
if (IS_ERR(endpoint))
243
+
return PTR_ERR(endpoint);
244
+
245
+
rc = cxl_endpoint_autoremove(cxlmd, endpoint);
246
+
if (rc)
247
+
return rc;
248
+
249
+
if (!endpoint->dev.driver) {
250
+
dev_err(&cxlmd->dev, "%s failed probe\n",
251
+
dev_name(&endpoint->dev));
252
+
return -ENXIO;
253
+
}
254
+
255
+
return 0;
256
+
}
257
+
EXPORT_SYMBOL_FOR_MODULES(devm_cxl_add_endpoint, "cxl_mem");
228
258
229
259
static int __init cxl_port_init(void)
230
260
{
+1
-2
drivers/dax/hmem/device.c
+1
-2
drivers/dax/hmem/device.c
+3
-2
drivers/dax/hmem/hmem.c
+3
-2
drivers/dax/hmem/hmem.c
···
73
73
return 0;
74
74
}
75
75
76
-
rc = region_intersects(res->start, resource_size(res), IORESOURCE_MEM,
77
-
IORES_DESC_SOFT_RESERVED);
76
+
rc = region_intersects_soft_reserve(res->start, resource_size(res));
78
77
if (rc != REGION_INTERSECTS)
79
78
return 0;
79
+
80
+
/* TODO: Add Soft-Reserved memory back to iomem */
80
81
81
82
id = memregion_alloc(GFP_KERNEL);
82
83
if (id < 0) {
+32
-7
drivers/pci/pci.h
+32
-7
drivers/pci/pci.h
···
816
816
817
817
#define AER_MAX_MULTI_ERR_DEVICES 5 /* Not likely to have more */
818
818
819
+
/**
820
+
* struct aer_err_info - AER Error Information
821
+
* @dev: Devices reporting error
822
+
* @ratelimit_print: Flag to log or not log the devices' error. 0=NotLog/1=Log
823
+
* @__pad1: Padding for alignment
824
+
* @error_dev_num: Number of devices reporting an error
825
+
* @level: printk level to use in logging
826
+
* @id: Value from register PCI_ERR_ROOT_ERR_SRC
827
+
* @severity: AER severity, 0-UNCOR Non-fatal, 1-UNCOR fatal, 2-COR
828
+
* @root_ratelimit_print: Flag to log or not log the root's error. 0=NotLog/1=Log
829
+
* @multi_error_valid: If multiple errors are reported
830
+
* @first_error: First reported error
831
+
* @__pad2: Padding for alignment
832
+
* @is_cxl: Bus type error: 0-PCI Bus error, 1-CXL Bus error
833
+
* @tlp_header_valid: Indicates if TLP field contains error information
834
+
* @status: COR/UNCOR error status
835
+
* @mask: COR/UNCOR mask
836
+
* @tlp: Transaction packet information
837
+
*/
819
838
struct aer_err_info {
820
839
struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES];
821
840
int ratelimit_print[AER_MAX_MULTI_ERR_DEVICES];
822
841
int error_dev_num;
823
-
const char *level; /* printk level */
842
+
const char *level;
824
843
825
844
unsigned int id:16;
826
845
827
-
unsigned int severity:2; /* 0:NONFATAL | 1:FATAL | 2:COR */
828
-
unsigned int root_ratelimit_print:1; /* 0=skip, 1=print */
846
+
unsigned int severity:2;
847
+
unsigned int root_ratelimit_print:1;
829
848
unsigned int __pad1:4;
830
849
unsigned int multi_error_valid:1;
831
850
832
851
unsigned int first_error:5;
833
-
unsigned int __pad2:2;
852
+
unsigned int __pad2:1;
853
+
unsigned int is_cxl:1;
834
854
unsigned int tlp_header_valid:1;
835
855
836
-
unsigned int status; /* COR/UNCOR Error Status */
837
-
unsigned int mask; /* COR/UNCOR Error Mask */
838
-
struct pcie_tlp_log tlp; /* TLP Header */
856
+
unsigned int status;
857
+
unsigned int mask;
858
+
struct pcie_tlp_log tlp;
839
859
};
840
860
841
861
int aer_get_device_error_info(struct aer_err_info *info, int i);
842
862
void aer_print_error(struct aer_err_info *info, int i);
863
+
864
+
static inline const char *aer_err_bus(struct aer_err_info *info)
865
+
{
866
+
return info->is_cxl ? "CXL" : "PCIe";
867
+
}
843
868
844
869
int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2,
845
870
unsigned int tlp_len, bool flit,
-9
drivers/pci/pcie/Kconfig
-9
drivers/pci/pcie/Kconfig
···
49
49
gotten from:
50
50
https://github.com/intel/aer-inject.git
51
51
52
-
config PCIEAER_CXL
53
-
bool "PCI Express CXL RAS support"
54
-
default y
55
-
depends on PCIEAER && CXL_PCI
56
-
help
57
-
Enables CXL error handling.
58
-
59
-
If unsure, say Y.
60
-
61
52
#
62
53
# PCI Express ECRC
63
54
#
+1
drivers/pci/pcie/Makefile
+1
drivers/pci/pcie/Makefile
+21
-114
drivers/pci/pcie/aer.c
+21
-114
drivers/pci/pcie/aer.c
···
867
867
struct pci_dev *dev;
868
868
int layer, agent, id;
869
869
const char *level = info->level;
870
+
const char *bus_type = aer_err_bus(info);
870
871
871
872
if (WARN_ON_ONCE(i >= AER_MAX_MULTI_ERR_DEVICES))
872
873
return;
···
877
876
878
877
pci_dev_aer_stats_incr(dev, info);
879
878
trace_aer_event(pci_name(dev), (info->status & ~info->mask),
880
-
info->severity, info->tlp_header_valid, &info->tlp);
879
+
info->severity, info->tlp_header_valid, &info->tlp, bus_type);
881
880
882
881
if (!info->ratelimit_print[i])
883
882
return;
884
883
885
884
if (!info->status) {
886
-
pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
887
-
aer_error_severity_string[info->severity]);
885
+
pci_err(dev, "%s Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
886
+
bus_type, aer_error_severity_string[info->severity]);
888
887
goto out;
889
888
}
890
889
891
890
layer = AER_GET_LAYER_ERROR(info->severity, info->status);
892
891
agent = AER_GET_AGENT(info->severity, info->status);
893
892
894
-
aer_printk(level, dev, "PCIe Bus Error: severity=%s, type=%s, (%s)\n",
895
-
aer_error_severity_string[info->severity],
893
+
aer_printk(level, dev, "%s Bus Error: severity=%s, type=%s, (%s)\n",
894
+
bus_type, aer_error_severity_string[info->severity],
896
895
aer_error_layer[layer], aer_agent_string[agent]);
897
896
898
897
aer_printk(level, dev, " device [%04x:%04x] error status/mask=%08x/%08x\n",
···
926
925
void pci_print_aer(struct pci_dev *dev, int aer_severity,
927
926
struct aer_capability_regs *aer)
928
927
{
928
+
const char *bus_type;
929
929
int layer, agent, tlp_header_valid = 0;
930
930
u32 status, mask;
931
931
struct aer_err_info info = {
···
947
945
948
946
info.status = status;
949
947
info.mask = mask;
948
+
info.is_cxl = pcie_is_cxl(dev);
949
+
950
+
bus_type = aer_err_bus(&info);
950
951
951
952
pci_dev_aer_stats_incr(dev, &info);
952
-
trace_aer_event(pci_name(dev), (status & ~mask),
953
-
aer_severity, tlp_header_valid, &aer->header_log);
953
+
trace_aer_event(pci_name(dev), (status & ~mask), aer_severity,
954
+
tlp_header_valid, &aer->header_log, bus_type);
954
955
955
956
if (!aer_ratelimit(dev, info.severity))
956
957
return;
···
1122
1117
return true;
1123
1118
}
1124
1119
1125
-
#ifdef CONFIG_PCIEAER_CXL
1126
-
1127
1120
/**
1128
1121
* pci_aer_unmask_internal_errors - unmask internal errors
1129
1122
* @dev: pointer to the pci_dev data structure
···
1132
1129
* Note: AER must be enabled and supported by the device which must be
1133
1130
* checked in advance, e.g. with pcie_aer_is_native().
1134
1131
*/
1135
-
static void pci_aer_unmask_internal_errors(struct pci_dev *dev)
1132
+
void pci_aer_unmask_internal_errors(struct pci_dev *dev)
1136
1133
{
1137
1134
int aer = dev->aer_cap;
1138
1135
u32 mask;
···
1146
1143
pci_write_config_dword(dev, aer + PCI_ERR_COR_MASK, mask);
1147
1144
}
1148
1145
1149
-
static bool is_cxl_mem_dev(struct pci_dev *dev)
1150
-
{
1151
-
/*
1152
-
* The capability, status, and control fields in Device 0,
1153
-
* Function 0 DVSEC control the CXL functionality of the
1154
-
* entire device (CXL 3.0, 8.1.3).
1155
-
*/
1156
-
if (dev->devfn != PCI_DEVFN(0, 0))
1157
-
return false;
1146
+
/*
1147
+
* Internal errors are too device-specific to enable generally, however for CXL
1148
+
* their behavior is standardized for conveying CXL protocol errors.
1149
+
*/
1150
+
EXPORT_SYMBOL_FOR_MODULES(pci_aer_unmask_internal_errors, "cxl_core");
1158
1151
1159
-
/*
1160
-
* CXL Memory Devices must have the 502h class code set (CXL
1161
-
* 3.0, 8.1.12.1).
1162
-
*/
1163
-
if ((dev->class >> 8) != PCI_CLASS_MEMORY_CXL)
1164
-
return false;
1165
-
1166
-
return true;
1167
-
}
1168
-
1169
-
static bool cxl_error_is_native(struct pci_dev *dev)
1170
-
{
1171
-
struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
1172
-
1173
-
return (pcie_ports_native || host->native_aer);
1174
-
}
1175
-
1176
-
static bool is_internal_error(struct aer_err_info *info)
1152
+
#ifdef CONFIG_CXL_RAS
1153
+
bool is_aer_internal_error(struct aer_err_info *info)
1177
1154
{
1178
1155
if (info->severity == AER_CORRECTABLE)
1179
1156
return info->status & PCI_ERR_COR_INTERNAL;
1180
1157
1181
1158
return info->status & PCI_ERR_UNC_INTN;
1182
1159
}
1183
-
1184
-
static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
1185
-
{
1186
-
struct aer_err_info *info = (struct aer_err_info *)data;
1187
-
const struct pci_error_handlers *err_handler;
1188
-
1189
-
if (!is_cxl_mem_dev(dev) || !cxl_error_is_native(dev))
1190
-
return 0;
1191
-
1192
-
/* Protect dev->driver */
1193
-
device_lock(&dev->dev);
1194
-
1195
-
err_handler = dev->driver ? dev->driver->err_handler : NULL;
1196
-
if (!err_handler)
1197
-
goto out;
1198
-
1199
-
if (info->severity == AER_CORRECTABLE) {
1200
-
if (err_handler->cor_error_detected)
1201
-
err_handler->cor_error_detected(dev);
1202
-
} else if (err_handler->error_detected) {
1203
-
if (info->severity == AER_NONFATAL)
1204
-
err_handler->error_detected(dev, pci_channel_io_normal);
1205
-
else if (info->severity == AER_FATAL)
1206
-
err_handler->error_detected(dev, pci_channel_io_frozen);
1207
-
}
1208
-
out:
1209
-
device_unlock(&dev->dev);
1210
-
return 0;
1211
-
}
1212
-
1213
-
static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
1214
-
{
1215
-
/*
1216
-
* Internal errors of an RCEC indicate an AER error in an
1217
-
* RCH's downstream port. Check and handle them in the CXL.mem
1218
-
* device driver.
1219
-
*/
1220
-
if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
1221
-
is_internal_error(info))
1222
-
pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
1223
-
}
1224
-
1225
-
static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
1226
-
{
1227
-
bool *handles_cxl = data;
1228
-
1229
-
if (!*handles_cxl)
1230
-
*handles_cxl = is_cxl_mem_dev(dev) && cxl_error_is_native(dev);
1231
-
1232
-
/* Non-zero terminates iteration */
1233
-
return *handles_cxl;
1234
-
}
1235
-
1236
-
static bool handles_cxl_errors(struct pci_dev *rcec)
1237
-
{
1238
-
bool handles_cxl = false;
1239
-
1240
-
if (pci_pcie_type(rcec) == PCI_EXP_TYPE_RC_EC &&
1241
-
pcie_aer_is_native(rcec))
1242
-
pcie_walk_rcec(rcec, handles_cxl_error_iter, &handles_cxl);
1243
-
1244
-
return handles_cxl;
1245
-
}
1246
-
1247
-
static void cxl_rch_enable_rcec(struct pci_dev *rcec)
1248
-
{
1249
-
if (!handles_cxl_errors(rcec))
1250
-
return;
1251
-
1252
-
pci_aer_unmask_internal_errors(rcec);
1253
-
pci_info(rcec, "CXL: Internal errors unmasked");
1254
-
}
1255
-
1256
-
#else
1257
-
static inline void cxl_rch_enable_rcec(struct pci_dev *dev) { }
1258
-
static inline void cxl_rch_handle_error(struct pci_dev *dev,
1259
-
struct aer_err_info *info) { }
1260
1160
#endif
1261
1161
1262
1162
/**
···
1308
1402
/* Must reset in this function */
1309
1403
info->status = 0;
1310
1404
info->tlp_header_valid = 0;
1405
+
info->is_cxl = pcie_is_cxl(dev);
1311
1406
1312
1407
/* The device might not support AER */
1313
1408
if (!aer)
+104
drivers/pci/pcie/aer_cxl_rch.c
+104
drivers/pci/pcie/aer_cxl_rch.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
/* Copyright(c) 2023 AMD Corporation. All rights reserved. */
3
+
4
+
#include <linux/pci.h>
5
+
#include <linux/aer.h>
6
+
#include <linux/bitfield.h>
7
+
#include "../pci.h"
8
+
#include "portdrv.h"
9
+
10
+
static bool is_cxl_mem_dev(struct pci_dev *dev)
11
+
{
12
+
/*
13
+
* The capability, status, and control fields in Device 0,
14
+
* Function 0 DVSEC control the CXL functionality of the
15
+
* entire device (CXL 3.0, 8.1.3).
16
+
*/
17
+
if (dev->devfn != PCI_DEVFN(0, 0))
18
+
return false;
19
+
20
+
/*
21
+
* CXL Memory Devices must have the 502h class code set (CXL
22
+
* 3.0, 8.1.12.1).
23
+
*/
24
+
if ((dev->class >> 8) != PCI_CLASS_MEMORY_CXL)
25
+
return false;
26
+
27
+
return true;
28
+
}
29
+
30
+
static bool cxl_error_is_native(struct pci_dev *dev)
31
+
{
32
+
struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
33
+
34
+
return (pcie_ports_native || host->native_aer);
35
+
}
36
+
37
+
static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
38
+
{
39
+
struct aer_err_info *info = (struct aer_err_info *)data;
40
+
const struct pci_error_handlers *err_handler;
41
+
42
+
if (!is_cxl_mem_dev(dev) || !cxl_error_is_native(dev))
43
+
return 0;
44
+
45
+
guard(device)(&dev->dev);
46
+
47
+
err_handler = dev->driver ? dev->driver->err_handler : NULL;
48
+
if (!err_handler)
49
+
return 0;
50
+
51
+
if (info->severity == AER_CORRECTABLE) {
52
+
if (err_handler->cor_error_detected)
53
+
err_handler->cor_error_detected(dev);
54
+
} else if (err_handler->error_detected) {
55
+
if (info->severity == AER_NONFATAL)
56
+
err_handler->error_detected(dev, pci_channel_io_normal);
57
+
else if (info->severity == AER_FATAL)
58
+
err_handler->error_detected(dev, pci_channel_io_frozen);
59
+
}
60
+
return 0;
61
+
}
62
+
63
+
void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
64
+
{
65
+
/*
66
+
* Internal errors of an RCEC indicate an AER error in an
67
+
* RCH's downstream port. Check and handle them in the CXL.mem
68
+
* device driver.
69
+
*/
70
+
if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
71
+
is_aer_internal_error(info))
72
+
pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
73
+
}
74
+
75
+
static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
76
+
{
77
+
bool *handles_cxl = data;
78
+
79
+
if (!*handles_cxl)
80
+
*handles_cxl = is_cxl_mem_dev(dev) && cxl_error_is_native(dev);
81
+
82
+
/* Non-zero terminates iteration */
83
+
return *handles_cxl;
84
+
}
85
+
86
+
static bool handles_cxl_errors(struct pci_dev *rcec)
87
+
{
88
+
bool handles_cxl = false;
89
+
90
+
if (pci_pcie_type(rcec) == PCI_EXP_TYPE_RC_EC &&
91
+
pcie_aer_is_native(rcec))
92
+
pcie_walk_rcec(rcec, handles_cxl_error_iter, &handles_cxl);
93
+
94
+
return handles_cxl;
95
+
}
96
+
97
+
void cxl_rch_enable_rcec(struct pci_dev *rcec)
98
+
{
99
+
if (!handles_cxl_errors(rcec))
100
+
return;
101
+
102
+
pci_aer_unmask_internal_errors(rcec);
103
+
pci_info(rcec, "CXL: Internal errors unmasked");
104
+
}
+12
drivers/pci/pcie/portdrv.h
+12
drivers/pci/pcie/portdrv.h
···
123
123
#endif /* !CONFIG_PCIE_PME */
124
124
125
125
struct device *pcie_port_find_device(struct pci_dev *dev, u32 service);
126
+
127
+
struct aer_err_info;
128
+
129
+
#ifdef CONFIG_CXL_RAS
130
+
bool is_aer_internal_error(struct aer_err_info *info);
131
+
void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info);
132
+
void cxl_rch_enable_rcec(struct pci_dev *rcec);
133
+
#else
134
+
static inline bool is_aer_internal_error(struct aer_err_info *info) { return false; }
135
+
static inline void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info) { }
136
+
static inline void cxl_rch_enable_rcec(struct pci_dev *rcec) { }
137
+
#endif /* CONFIG_CXL_RAS */
126
138
#endif /* _PORTDRV_H_ */
+31
drivers/pci/probe.c
+31
drivers/pci/probe.c
···
1704
1704
dev->is_thunderbolt = 1;
1705
1705
}
1706
1706
1707
+
static void set_pcie_cxl(struct pci_dev *dev)
1708
+
{
1709
+
struct pci_dev *bridge;
1710
+
u16 dvsec, cap;
1711
+
1712
+
if (!pci_is_pcie(dev))
1713
+
return;
1714
+
1715
+
/*
1716
+
* Update parent's CXL state because alternate protocol training
1717
+
* may have changed
1718
+
*/
1719
+
bridge = pci_upstream_bridge(dev);
1720
+
if (bridge)
1721
+
set_pcie_cxl(bridge);
1722
+
1723
+
dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
1724
+
PCI_DVSEC_CXL_FLEXBUS_PORT);
1725
+
if (!dvsec)
1726
+
return;
1727
+
1728
+
pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS,
1729
+
&cap);
1730
+
1731
+
dev->is_cxl = FIELD_GET(PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS_CACHE, cap) ||
1732
+
FIELD_GET(PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS_MEM, cap);
1733
+
1734
+
}
1735
+
1707
1736
static void set_pcie_untrusted(struct pci_dev *dev)
1708
1737
{
1709
1738
struct pci_dev *parent = pci_upstream_bridge(dev);
···
2069
2040
2070
2041
/* Need to have dev->cfg_size ready */
2071
2042
set_pcie_thunderbolt(dev);
2043
+
2044
+
set_pcie_cxl(dev);
2072
2045
2073
2046
set_pcie_untrusted(dev);
2074
2047
+2
include/linux/aer.h
+2
include/linux/aer.h
···
56
56
#if defined(CONFIG_PCIEAER)
57
57
int pci_aer_clear_nonfatal_status(struct pci_dev *dev);
58
58
int pcie_aer_is_native(struct pci_dev *dev);
59
+
void pci_aer_unmask_internal_errors(struct pci_dev *dev);
59
60
#else
60
61
static inline int pci_aer_clear_nonfatal_status(struct pci_dev *dev)
61
62
{
62
63
return -EINVAL;
63
64
}
64
65
static inline int pcie_aer_is_native(struct pci_dev *dev) { return 0; }
66
+
static inline void pci_aer_unmask_internal_errors(struct pci_dev *dev) { }
65
67
#endif
66
68
67
69
void pci_print_aer(struct pci_dev *dev, int aer_severity,
+5
include/linux/ioport.h
+5
include/linux/ioport.h
···
237
237
/* PC/ISA/whatever - the normal PC address spaces: IO and memory */
238
238
extern struct resource ioport_resource;
239
239
extern struct resource iomem_resource;
240
+
extern struct resource soft_reserve_resource;
240
241
241
242
extern struct resource *request_resource_conflict(struct resource *root, struct resource *new);
242
243
extern int request_resource(struct resource *root, struct resource *new);
···
424
423
extern int
425
424
walk_iomem_res_desc(unsigned long desc, unsigned long flags, u64 start, u64 end,
426
425
void *arg, int (*func)(struct resource *, void *));
426
+
extern int walk_soft_reserve_res(u64 start, u64 end, void *arg,
427
+
int (*func)(struct resource *, void *));
428
+
extern int
429
+
region_intersects_soft_reserve(resource_size_t start, size_t size);
427
430
428
431
struct resource *devm_request_free_mem_region(struct device *dev,
429
432
struct resource *base, unsigned long size);
+6
include/linux/pci.h
+6
include/linux/pci.h
···
475
475
unsigned int is_pciehp:1;
476
476
unsigned int shpc_managed:1; /* SHPC owned by shpchp */
477
477
unsigned int is_thunderbolt:1; /* Thunderbolt controller */
478
+
unsigned int is_cxl:1; /* Compute Express Link (CXL) */
478
479
/*
479
480
* Devices marked being untrusted are the ones that can potentially
480
481
* execute DMA attacks and similar. They are typically connected
···
803
802
static inline bool pci_is_display(struct pci_dev *pdev)
804
803
{
805
804
return (pdev->class >> 16) == PCI_BASE_CLASS_DISPLAY;
805
+
}
806
+
807
+
static inline bool pcie_is_cxl(struct pci_dev *pci_dev)
808
+
{
809
+
return pci_dev->is_cxl;
806
810
}
807
811
808
812
#define for_each_pci_bridge(dev, bus) \
+8
-4
include/ras/ras_event.h
+8
-4
include/ras/ras_event.h
···
339
339
const u32 status,
340
340
const u8 severity,
341
341
const u8 tlp_header_valid,
342
-
struct pcie_tlp_log *tlp),
342
+
struct pcie_tlp_log *tlp,
343
+
const char *bus_type),
343
344
344
-
TP_ARGS(dev_name, status, severity, tlp_header_valid, tlp),
345
+
346
+
TP_ARGS(dev_name, status, severity, tlp_header_valid, tlp, bus_type),
345
347
346
348
TP_STRUCT__entry(
347
349
__string( dev_name, dev_name )
···
351
349
__field( u8, severity )
352
350
__field( u8, tlp_header_valid)
353
351
__array( u32, tlp_header, PCIE_STD_MAX_TLP_HEADERLOG)
352
+
__string( bus_type, bus_type )
354
353
),
355
354
356
355
TP_fast_assign(
357
356
__assign_str(dev_name);
357
+
__assign_str(bus_type);
358
358
__entry->status = status;
359
359
__entry->severity = severity;
360
360
__entry->tlp_header_valid = tlp_header_valid;
···
368
364
}
369
365
),
370
366
371
-
TP_printk("%s PCIe Bus Error: severity=%s, %s, TLP Header=%s\n",
372
-
__get_str(dev_name),
367
+
TP_printk("%s %s Bus Error: severity=%s, %s, TLP Header=%s\n",
368
+
__get_str(dev_name), __get_str(bus_type),
373
369
__entry->severity == AER_CORRECTABLE ? "Corrected" :
374
370
__entry->severity == AER_FATAL ?
375
371
"Fatal" : "Uncorrected, non-fatal",
+59
-5
include/uapi/linux/pci_regs.h
+59
-5
include/uapi/linux/pci_regs.h
···
1258
1258
#define PCI_DEV3_STA 0x0c /* Device 3 Status Register */
1259
1259
#define PCI_DEV3_STA_SEGMENT 0x8 /* Segment Captured (end-to-end flit-mode detected) */
1260
1260
1261
-
/* Compute Express Link (CXL r3.1, sec 8.1.5) */
1262
-
#define PCI_DVSEC_CXL_PORT 3
1263
-
#define PCI_DVSEC_CXL_PORT_CTL 0x0c
1264
-
#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR 0x00000001
1265
-
1266
1261
/* Integrity and Data Encryption Extended Capability */
1267
1262
#define PCI_IDE_CAP 0x04
1268
1263
#define PCI_IDE_CAP_LINK 0x1 /* Link IDE Stream Supported */
···
1337
1342
/* IDE Address Association Register 3 is "Memory Base Upper" */
1338
1343
#define PCI_IDE_SEL_ADDR_3(x) (28 + (x) * PCI_IDE_SEL_ADDR_BLOCK_SIZE)
1339
1344
#define PCI_IDE_SEL_BLOCK_SIZE(nr_assoc) (20 + PCI_IDE_SEL_ADDR_BLOCK_SIZE * (nr_assoc))
1345
+
1346
+
/*
1347
+
* Compute Express Link (CXL r4.0, sec 8.1)
1348
+
*
1349
+
* Note that CXL DVSEC id 3 and 7 to be ignored when the CXL link state
1350
+
* is "disconnected" (CXL r4.0, sec 9.12.3). Re-enumerate these
1351
+
* registers on downstream link-up events.
1352
+
*/
1353
+
1354
+
/* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */
1355
+
#define PCI_DVSEC_CXL_DEVICE 0
1356
+
#define PCI_DVSEC_CXL_CAP 0xA
1357
+
#define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2)
1358
+
#define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4)
1359
+
#define PCI_DVSEC_CXL_CTRL 0xC
1360
+
#define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2)
1361
+
#define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10))
1362
+
#define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10))
1363
+
#define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0)
1364
+
#define PCI_DVSEC_CXL_MEM_ACTIVE _BITUL(1)
1365
+
#define PCI_DVSEC_CXL_MEM_SIZE_LOW __GENMASK(31, 28)
1366
+
#define PCI_DVSEC_CXL_RANGE_BASE_HIGH(i) (0x20 + (i * 0x10))
1367
+
#define PCI_DVSEC_CXL_RANGE_BASE_LOW(i) (0x24 + (i * 0x10))
1368
+
#define PCI_DVSEC_CXL_MEM_BASE_LOW __GENMASK(31, 28)
1369
+
1370
+
#define CXL_DVSEC_RANGE_MAX 2
1371
+
1372
+
/* CXL r4.0, 8.1.4: Non-CXL Function Map DVSEC */
1373
+
#define PCI_DVSEC_CXL_FUNCTION_MAP 2
1374
+
1375
+
/* CXL r4.0, 8.1.5: Extensions DVSEC for Ports */
1376
+
#define PCI_DVSEC_CXL_PORT 3
1377
+
#define PCI_DVSEC_CXL_PORT_CTL 0x0c
1378
+
#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR 0x00000001
1379
+
1380
+
/* CXL r4.0, 8.1.6: GPF DVSEC for CXL Port */
1381
+
#define PCI_DVSEC_CXL_PORT_GPF 4
1382
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_1_CONTROL 0x0C
1383
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_1_TMO_BASE __GENMASK(3, 0)
1384
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_1_TMO_SCALE __GENMASK(11, 8)
1385
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_2_CONTROL 0xE
1386
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_2_TMO_BASE __GENMASK(3, 0)
1387
+
#define PCI_DVSEC_CXL_PORT_GPF_PHASE_2_TMO_SCALE __GENMASK(11, 8)
1388
+
1389
+
/* CXL r4.0, 8.1.7: GPF DVSEC for CXL Device */
1390
+
#define PCI_DVSEC_CXL_DEVICE_GPF 5
1391
+
1392
+
/* CXL r4.0, 8.1.8: Flex Bus DVSEC */
1393
+
#define PCI_DVSEC_CXL_FLEXBUS_PORT 7
1394
+
#define PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS 0xE
1395
+
#define PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS_CACHE _BITUL(0)
1396
+
#define PCI_DVSEC_CXL_FLEXBUS_PORT_STATUS_MEM _BITUL(2)
1397
+
1398
+
/* CXL r4.0, 8.1.9: Register Locator DVSEC */
1399
+
#define PCI_DVSEC_CXL_REG_LOCATOR 8
1400
+
#define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1 0xC
1401
+
#define PCI_DVSEC_CXL_REG_LOCATOR_BIR __GENMASK(2, 0)
1402
+
#define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_ID __GENMASK(15, 8)
1403
+
#define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW __GENMASK(31, 16)
1340
1404
1341
1405
#endif /* LINUX_PCI_REGS_H */
+60
-11
kernel/resource.c
+60
-11
kernel/resource.c
···
48
48
};
49
49
EXPORT_SYMBOL(iomem_resource);
50
50
51
+
struct resource soft_reserve_resource = {
52
+
.name = "Soft Reserved",
53
+
.start = 0,
54
+
.end = -1,
55
+
.desc = IORES_DESC_SOFT_RESERVED,
56
+
.flags = IORESOURCE_MEM,
57
+
};
58
+
51
59
static DEFINE_RWLOCK(resource_lock);
52
60
53
61
/*
···
329
321
}
330
322
331
323
/**
332
-
* find_next_iomem_res - Finds the lowest iomem resource that covers part of
333
-
* [@start..@end].
324
+
* find_next_res - Finds the lowest resource that covers part of
325
+
* [@start..@end].
334
326
*
335
327
* If a resource is found, returns 0 and @*res is overwritten with the part
336
328
* of the resource that's within [@start..@end]; if none is found, returns
337
329
* -ENODEV. Returns -EINVAL for invalid parameters.
338
330
*
331
+
* @parent: resource tree root to search
339
332
* @start: start address of the resource searched for
340
333
* @end: end address of same resource
341
334
* @flags: flags which the resource must have
···
346
337
* The caller must specify @start, @end, @flags, and @desc
347
338
* (which may be IORES_DESC_NONE).
348
339
*/
349
-
static int find_next_iomem_res(resource_size_t start, resource_size_t end,
350
-
unsigned long flags, unsigned long desc,
351
-
struct resource *res)
340
+
static int find_next_res(struct resource *parent, resource_size_t start,
341
+
resource_size_t end, unsigned long flags,
342
+
unsigned long desc, struct resource *res)
352
343
{
353
344
/* Skip children until we find a top level range that matches */
354
345
bool skip_children = true;
···
362
353
363
354
read_lock(&resource_lock);
364
355
365
-
for_each_resource(&iomem_resource, p, skip_children) {
356
+
for_each_resource(parent, p, skip_children) {
366
357
/* If we passed the resource we are looking for, stop */
367
358
if (p->start > end) {
368
359
p = NULL;
···
399
390
return p ? 0 : -ENODEV;
400
391
}
401
392
402
-
static int __walk_iomem_res_desc(resource_size_t start, resource_size_t end,
403
-
unsigned long flags, unsigned long desc,
404
-
void *arg,
405
-
int (*func)(struct resource *, void *))
393
+
static int find_next_iomem_res(resource_size_t start, resource_size_t end,
394
+
unsigned long flags, unsigned long desc,
395
+
struct resource *res)
396
+
{
397
+
return find_next_res(&iomem_resource, start, end, flags, desc, res);
398
+
}
399
+
400
+
static int walk_res_desc(struct resource *parent, resource_size_t start,
401
+
resource_size_t end, unsigned long flags,
402
+
unsigned long desc, void *arg,
403
+
int (*func)(struct resource *, void *))
406
404
{
407
405
struct resource res;
408
406
int ret = -EINVAL;
409
407
410
408
while (start < end &&
411
-
!find_next_iomem_res(start, end, flags, desc, &res)) {
409
+
!find_next_res(parent, start, end, flags, desc, &res)) {
412
410
ret = (*func)(&res, arg);
413
411
if (ret)
414
412
break;
···
425
409
426
410
return ret;
427
411
}
412
+
413
+
static int __walk_iomem_res_desc(resource_size_t start, resource_size_t end,
414
+
unsigned long flags, unsigned long desc,
415
+
void *arg,
416
+
int (*func)(struct resource *, void *))
417
+
{
418
+
return walk_res_desc(&iomem_resource, start, end, flags, desc, arg, func);
419
+
}
420
+
428
421
429
422
/**
430
423
* walk_iomem_res_desc - Walks through iomem resources and calls func()
···
458
433
return __walk_iomem_res_desc(start, end, flags, desc, arg, func);
459
434
}
460
435
EXPORT_SYMBOL_GPL(walk_iomem_res_desc);
436
+
437
+
/*
438
+
* In support of device drivers claiming Soft Reserved resources, walk the Soft
439
+
* Reserved resource deferral tree.
440
+
*/
441
+
int walk_soft_reserve_res(u64 start, u64 end, void *arg,
442
+
int (*func)(struct resource *, void *))
443
+
{
444
+
return walk_res_desc(&soft_reserve_resource, start, end, IORESOURCE_MEM,
445
+
IORES_DESC_SOFT_RESERVED, arg, func);
446
+
}
447
+
EXPORT_SYMBOL_GPL(walk_soft_reserve_res);
461
448
462
449
/*
463
450
* This function calls the @func callback against all memory ranges of type
···
692
655
return ret;
693
656
}
694
657
EXPORT_SYMBOL_GPL(region_intersects);
658
+
659
+
/*
660
+
* Check if the provided range is registered in the Soft Reserved resource
661
+
* deferral tree for driver consideration.
662
+
*/
663
+
int region_intersects_soft_reserve(resource_size_t start, size_t size)
664
+
{
665
+
guard(read_lock)(&resource_lock);
666
+
return __region_intersects(&soft_reserve_resource, start, size,
667
+
IORESOURCE_MEM, IORES_DESC_SOFT_RESERVED);
668
+
}
669
+
EXPORT_SYMBOL_GPL(region_intersects_soft_reserve);
695
670
696
671
void __weak arch_remove_reservations(struct resource *avail)
697
672
{
+5
-2
tools/testing/cxl/Kbuild
+5
-2
tools/testing/cxl/Kbuild
···
7
7
ldflags-y += --wrap=cxl_await_media_ready
8
8
ldflags-y += --wrap=devm_cxl_add_rch_dport
9
9
ldflags-y += --wrap=cxl_endpoint_parse_cdat
10
-
ldflags-y += --wrap=cxl_dport_init_ras_reporting
11
10
ldflags-y += --wrap=devm_cxl_endpoint_decoders_setup
12
11
ldflags-y += --wrap=hmat_get_extended_linear_cache_size
12
+
ldflags-y += --wrap=devm_cxl_add_dport_by_dev
13
+
ldflags-y += --wrap=devm_cxl_switch_port_decoders_setup
13
14
14
15
DRIVERS := ../../../drivers
15
16
CXL_SRC := $(DRIVERS)/cxl
···
58
57
cxl_core-y += $(CXL_CORE_SRC)/hdm.o
59
58
cxl_core-y += $(CXL_CORE_SRC)/pmu.o
60
59
cxl_core-y += $(CXL_CORE_SRC)/cdat.o
61
-
cxl_core-y += $(CXL_CORE_SRC)/ras.o
62
60
cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o
63
61
cxl_core-$(CONFIG_CXL_REGION) += $(CXL_CORE_SRC)/region.o
64
62
cxl_core-$(CONFIG_CXL_MCE) += $(CXL_CORE_SRC)/mce.o
65
63
cxl_core-$(CONFIG_CXL_FEATURES) += $(CXL_CORE_SRC)/features.o
66
64
cxl_core-$(CONFIG_CXL_EDAC_MEM_FEATURES) += $(CXL_CORE_SRC)/edac.o
65
+
cxl_core-$(CONFIG_CXL_RAS) += $(CXL_CORE_SRC)/ras.o
66
+
cxl_core-$(CONFIG_CXL_RAS) += $(CXL_CORE_SRC)/ras_rch.o
67
+
cxl_core-$(CONFIG_CXL_ATL) += $(CXL_CORE_SRC)/atl.o
67
68
cxl_core-y += config_check.o
68
69
cxl_core-y += cxl_core_test.o
69
70
cxl_core-y += cxl_core_exports.o
-22
tools/testing/cxl/cxl_core_exports.c
-22
tools/testing/cxl/cxl_core_exports.c
···
2
2
/* Copyright(c) 2022 Intel Corporation. All rights reserved. */
3
3
4
4
#include "cxl.h"
5
-
#include "exports.h"
6
5
7
6
/* Exporting of cxl_core symbols that are only used by cxl_test */
8
7
EXPORT_SYMBOL_NS_GPL(cxl_num_decoders_committed, "CXL");
9
-
10
-
cxl_add_dport_by_dev_fn _devm_cxl_add_dport_by_dev =
11
-
__devm_cxl_add_dport_by_dev;
12
-
EXPORT_SYMBOL_NS_GPL(_devm_cxl_add_dport_by_dev, "CXL");
13
-
14
-
struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
15
-
struct device *dport_dev)
16
-
{
17
-
return _devm_cxl_add_dport_by_dev(port, dport_dev);
18
-
}
19
-
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport_by_dev, "CXL");
20
-
21
-
cxl_switch_decoders_setup_fn _devm_cxl_switch_port_decoders_setup =
22
-
__devm_cxl_switch_port_decoders_setup;
23
-
EXPORT_SYMBOL_NS_GPL(_devm_cxl_switch_port_decoders_setup, "CXL");
24
-
25
-
int devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
26
-
{
27
-
return _devm_cxl_switch_port_decoders_setup(port);
28
-
}
29
-
EXPORT_SYMBOL_NS_GPL(devm_cxl_switch_port_decoders_setup, "CXL");
-13
tools/testing/cxl/exports.h
-13
tools/testing/cxl/exports.h
···
1
-
/* SPDX-License-Identifier: GPL-2.0 */
2
-
/* Copyright(c) 2025 Intel Corporation */
3
-
#ifndef __MOCK_CXL_EXPORTS_H_
4
-
#define __MOCK_CXL_EXPORTS_H_
5
-
6
-
typedef struct cxl_dport *(*cxl_add_dport_by_dev_fn)(struct cxl_port *port,
7
-
struct device *dport_dev);
8
-
extern cxl_add_dport_by_dev_fn _devm_cxl_add_dport_by_dev;
9
-
10
-
typedef int(*cxl_switch_decoders_setup_fn)(struct cxl_port *port);
11
-
extern cxl_switch_decoders_setup_fn _devm_cxl_switch_port_decoders_setup;
12
-
13
-
#endif
+1
-1
tools/testing/cxl/test/mem.c
+1
-1
tools/testing/cxl/test/mem.c
+7
-29
tools/testing/cxl/test/mock.c
+7
-29
tools/testing/cxl/test/mock.c
···
10
10
#include <cxlmem.h>
11
11
#include <cxlpci.h>
12
12
#include "mock.h"
13
-
#include "../exports.h"
14
13
15
14
static LIST_HEAD(mock);
16
-
17
-
static struct cxl_dport *
18
-
redirect_devm_cxl_add_dport_by_dev(struct cxl_port *port,
19
-
struct device *dport_dev);
20
-
static int redirect_devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
21
15
22
16
void register_cxl_mock_ops(struct cxl_mock_ops *ops)
23
17
{
24
18
list_add_rcu(&ops->list, &mock);
25
-
_devm_cxl_add_dport_by_dev = redirect_devm_cxl_add_dport_by_dev;
26
-
_devm_cxl_switch_port_decoders_setup =
27
-
redirect_devm_cxl_switch_port_decoders_setup;
28
19
}
29
20
EXPORT_SYMBOL_GPL(register_cxl_mock_ops);
30
21
···
23
32
24
33
void unregister_cxl_mock_ops(struct cxl_mock_ops *ops)
25
34
{
26
-
_devm_cxl_switch_port_decoders_setup =
27
-
__devm_cxl_switch_port_decoders_setup;
28
-
_devm_cxl_add_dport_by_dev = __devm_cxl_add_dport_by_dev;
29
35
list_del_rcu(&ops->list);
30
36
synchronize_srcu(&cxl_mock_srcu);
31
37
}
···
151
163
}
152
164
EXPORT_SYMBOL_GPL(__wrap_nvdimm_bus_register);
153
165
154
-
int redirect_devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
166
+
int __wrap_devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
155
167
{
156
168
int rc, index;
157
169
struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
···
159
171
if (ops && ops->is_mock_port(port->uport_dev))
160
172
rc = ops->devm_cxl_switch_port_decoders_setup(port);
161
173
else
162
-
rc = __devm_cxl_switch_port_decoders_setup(port);
174
+
rc = devm_cxl_switch_port_decoders_setup(port);
163
175
put_cxl_mock_ops(index);
164
176
165
177
return rc;
166
178
}
179
+
EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_switch_port_decoders_setup, "CXL");
167
180
168
181
int __wrap_devm_cxl_endpoint_decoders_setup(struct cxl_port *port)
169
182
{
···
234
245
}
235
246
EXPORT_SYMBOL_NS_GPL(__wrap_cxl_endpoint_parse_cdat, "CXL");
236
247
237
-
void __wrap_cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host)
238
-
{
239
-
int index;
240
-
struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
241
-
242
-
if (!ops || !ops->is_mock_port(dport->dport_dev))
243
-
cxl_dport_init_ras_reporting(dport, host);
244
-
245
-
put_cxl_mock_ops(index);
246
-
}
247
-
EXPORT_SYMBOL_NS_GPL(__wrap_cxl_dport_init_ras_reporting, "CXL");
248
-
249
-
struct cxl_dport *redirect_devm_cxl_add_dport_by_dev(struct cxl_port *port,
250
-
struct device *dport_dev)
248
+
struct cxl_dport *__wrap_devm_cxl_add_dport_by_dev(struct cxl_port *port,
249
+
struct device *dport_dev)
251
250
{
252
251
int index;
253
252
struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
···
244
267
if (ops && ops->is_mock_port(port->uport_dev))
245
268
dport = ops->devm_cxl_add_dport_by_dev(port, dport_dev);
246
269
else
247
-
dport = __devm_cxl_add_dport_by_dev(port, dport_dev);
270
+
dport = devm_cxl_add_dport_by_dev(port, dport_dev);
248
271
put_cxl_mock_ops(index);
249
272
250
273
return dport;
251
274
}
275
+
EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_add_dport_by_dev, "CXL");
252
276
253
277
MODULE_LICENSE("GPL v2");
254
278
MODULE_DESCRIPTION("cxl_test: emulation module");