Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'drm-xe-next-2026-03-12' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next
UAPI Changes:
- add VM_BIND DECOMPRESS support and on-demand decompression (Nitin)
- Allow per queue programming of COMMON_SLICE_CHICKEN3 bit13 (Lionel)
Cross-subsystem Changes:
- Introduce the DRM RAS infrastructure over generic netlink (Riana, Rodrigo)
Core Changes:
- Two-pass MMU interval notifiers (Thomas)
Driver Changes:
- Merge drm/drm-next into drm-xe-next (Brost)
- Fix overflow in guc_ct_snapshot_capture (Mika, Fixes)
- Extract gt_pta_entry (Gustavo)
- Extra enabling patches for NVL-P (Gustavo)
- Add Wa_14026578760 (Varun)
- Add type-specific GT loop iterator (Roper)
- Refactor xe_migrate_prepare_vm (Raag)
- Don't disable GuCRC in suspend path (Vinay, Fixes)
- Add missing kernel docs in xe_exec_queue.c (Niranjana)
- Change TEST_VRAM to work with 32-bit resource_size_t (Wajdeczko)
- Fix memory leak in xe_vm_madvise_ioctl (Varun, Fixes)
- Skip access counter queue init for unsupported platforms (Himal)
Signed-off-by: Dave Airlie <airlied@redhat.com>
From: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/abLUVfSHu8EHRF9q@lstrano-desk.jf.intel.com
+2653
-266
+103
Documentation/gpu/drm-ras.rst
+103
Documentation/gpu/drm-ras.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0+
2
+
3
+
============================
4
+
DRM RAS over Generic Netlink
5
+
============================
6
+
7
+
The DRM RAS (Reliability, Availability, Serviceability) interface provides a
8
+
standardized way for GPU/accelerator drivers to expose error counters and
9
+
other reliability nodes to user space via Generic Netlink. This allows
10
+
diagnostic tools, monitoring daemons, or test infrastructure to query hardware
11
+
health in a uniform way across different DRM drivers.
12
+
13
+
Key Goals:
14
+
15
+
* Provide a standardized RAS solution for GPU and accelerator drivers, enabling
16
+
data center monitoring and reliability operations.
17
+
* Implement a single drm-ras Generic Netlink family to meet modern Netlink YAML
18
+
specifications and centralize all RAS-related communication in one namespace.
19
+
* Support a basic error counter interface, addressing the immediate, essential
20
+
monitoring needs.
21
+
* Offer a flexible, future-proof interface that can be extended to support
22
+
additional types of RAS data in the future.
23
+
* Allow multiple nodes per driver, enabling drivers to register separate
24
+
nodes for different IP blocks, sub-blocks, or other logical subdivisions
25
+
as applicable.
26
+
27
+
Nodes
28
+
=====
29
+
30
+
Nodes are logical abstractions representing an error type or error source within
31
+
the device. Currently, only error counter nodes is supported.
32
+
33
+
Drivers are responsible for registering and unregistering nodes via the
34
+
`drm_ras_node_register()` and `drm_ras_node_unregister()` APIs.
35
+
36
+
Node Management
37
+
-------------------
38
+
39
+
.. kernel-doc:: drivers/gpu/drm/drm_ras.c
40
+
:doc: DRM RAS Node Management
41
+
.. kernel-doc:: drivers/gpu/drm/drm_ras.c
42
+
:internal:
43
+
44
+
Generic Netlink Usage
45
+
=====================
46
+
47
+
The interface is implemented as a Generic Netlink family named ``drm-ras``.
48
+
User space tools can:
49
+
50
+
* List registered nodes with the ``list-nodes`` command.
51
+
* List all error counters in an node with the ``get-error-counter`` command with ``node-id``
52
+
as a parameter.
53
+
* Query specific error counter values with the ``get-error-counter`` command, using both
54
+
``node-id`` and ``error-id`` as parameters.
55
+
56
+
YAML-based Interface
57
+
--------------------
58
+
59
+
The interface is described in a YAML specification ``Documentation/netlink/specs/drm_ras.yaml``
60
+
61
+
This YAML is used to auto-generate user space bindings via
62
+
``tools/net/ynl/pyynl/ynl_gen_c.py``, and drives the structure of netlink
63
+
attributes and operations.
64
+
65
+
Usage Notes
66
+
-----------
67
+
68
+
* User space must first enumerate nodes to obtain their IDs.
69
+
* Node IDs or Node names can be used for all further queries, such as error counters.
70
+
* Error counters can be queried by either the Error ID or Error name.
71
+
* Query Parameters should be defined as part of the uAPI to ensure user interface stability.
72
+
* The interface supports future extension by adding new node types and
73
+
additional attributes.
74
+
75
+
Example: List nodes using ynl
76
+
77
+
.. code-block:: bash
78
+
79
+
sudo ynl --family drm_ras --dump list-nodes
80
+
[{'device-name': '0000:03:00.0',
81
+
'node-id': 0,
82
+
'node-name': 'correctable-errors',
83
+
'node-type': 'error-counter'},
84
+
{'device-name': '0000:03:00.0',
85
+
'node-id': 1,
86
+
'node-name': 'uncorrectable-errors',
87
+
'node-type': 'error-counter'}]
88
+
89
+
Example: List all error counters using ynl
90
+
91
+
.. code-block:: bash
92
+
93
+
sudo ynl --family drm_ras --dump get-error-counter --json '{"node-id":0}'
94
+
[{'error-id': 1, 'error-name': 'error_name1', 'error-value': 0},
95
+
{'error-id': 2, 'error-name': 'error_name2', 'error-value': 0}]
96
+
97
+
Example: Query an error counter for a given node
98
+
99
+
.. code-block:: bash
100
+
101
+
sudo ynl --family drm_ras --do get-error-counter --json '{"node-id":0, "error-id":1}'
102
+
{'error-id': 1, 'error-name': 'error_name1', 'error-value': 0}
103
+
+1
Documentation/gpu/index.rst
+1
Documentation/gpu/index.rst
+115
Documentation/netlink/specs/drm_ras.yaml
+115
Documentation/netlink/specs/drm_ras.yaml
···
1
+
# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
2
+
---
3
+
name: drm-ras
4
+
protocol: genetlink
5
+
uapi-header: drm/drm_ras.h
6
+
7
+
doc: >-
8
+
DRM RAS (Reliability, Availability, Serviceability) over Generic Netlink.
9
+
Provides a standardized mechanism for DRM drivers to register "nodes"
10
+
representing hardware/software components capable of reporting error counters.
11
+
Userspace tools can query the list of nodes or individual error counters
12
+
via the Generic Netlink interface.
13
+
14
+
definitions:
15
+
-
16
+
type: enum
17
+
name: node-type
18
+
value-start: 1
19
+
entries: [error-counter]
20
+
doc: >-
21
+
Type of the node. Currently, only error-counter nodes are
22
+
supported, which expose reliability counters for a hardware/software
23
+
component.
24
+
25
+
attribute-sets:
26
+
-
27
+
name: node-attrs
28
+
attributes:
29
+
-
30
+
name: node-id
31
+
type: u32
32
+
doc: >-
33
+
Unique identifier for the node.
34
+
Assigned dynamically by the DRM RAS core upon registration.
35
+
-
36
+
name: device-name
37
+
type: string
38
+
doc: >-
39
+
Device name chosen by the driver at registration.
40
+
Can be a PCI BDF, UUID, or module name if unique.
41
+
-
42
+
name: node-name
43
+
type: string
44
+
doc: >-
45
+
Node name chosen by the driver at registration.
46
+
Can be an IP block name, or any name that identifies the
47
+
RAS node inside the device.
48
+
-
49
+
name: node-type
50
+
type: u32
51
+
doc: Type of this node, identifying its function.
52
+
enum: node-type
53
+
-
54
+
name: error-counter-attrs
55
+
attributes:
56
+
-
57
+
name: node-id
58
+
type: u32
59
+
doc: Node ID targeted by this error counter operation.
60
+
-
61
+
name: error-id
62
+
type: u32
63
+
doc: Unique identifier for a specific error counter within an node.
64
+
-
65
+
name: error-name
66
+
type: string
67
+
doc: Name of the error.
68
+
-
69
+
name: error-value
70
+
type: u32
71
+
doc: Current value of the requested error counter.
72
+
73
+
operations:
74
+
list:
75
+
-
76
+
name: list-nodes
77
+
doc: >-
78
+
Retrieve the full list of currently registered DRM RAS nodes.
79
+
Each node includes its dynamically assigned ID, name, and type.
80
+
**Important:** User space must call this operation first to obtain
81
+
the node IDs. These IDs are required for all subsequent
82
+
operations on nodes, such as querying error counters.
83
+
attribute-set: node-attrs
84
+
flags: [admin-perm]
85
+
dump:
86
+
reply:
87
+
attributes:
88
+
- node-id
89
+
- device-name
90
+
- node-name
91
+
- node-type
92
+
-
93
+
name: get-error-counter
94
+
doc: >-
95
+
Retrieve error counter for a given node.
96
+
The response includes the id, the name, and even the current
97
+
value of each counter.
98
+
attribute-set: error-counter-attrs
99
+
flags: [admin-perm]
100
+
do:
101
+
request:
102
+
attributes:
103
+
- node-id
104
+
- error-id
105
+
reply:
106
+
attributes: &errorinfo
107
+
- error-id
108
+
- error-name
109
+
- error-value
110
+
dump:
111
+
request:
112
+
attributes:
113
+
- node-id
114
+
reply:
115
+
attributes: *errorinfo
+10
drivers/gpu/drm/Kconfig
+10
drivers/gpu/drm/Kconfig
···
130
130
Smaller QR code are easier to read, but will contain less debugging
131
131
data. Default is 40.
132
132
133
+
config DRM_RAS
134
+
bool "DRM RAS support"
135
+
depends on DRM
136
+
depends on NET
137
+
help
138
+
Enables the DRM RAS (Reliability, Availability and Serviceability)
139
+
support for DRM drivers. This provides a Generic Netlink interface
140
+
for error reporting and queries.
141
+
If in doubt, say "N".
142
+
133
143
config DRM_DEBUG_DP_MST_TOPOLOGY_REFS
134
144
bool "Enable refcount backtrace history in the DP MST helpers"
135
145
depends on STACKTRACE_SUPPORT
+1
drivers/gpu/drm/Makefile
+1
drivers/gpu/drm/Makefile
···
93
93
drm-$(CONFIG_DRM_PANIC) += drm_panic.o
94
94
drm-$(CONFIG_DRM_DRAW) += drm_draw.o
95
95
drm-$(CONFIG_DRM_PANIC_SCREEN_QR_CODE) += drm_panic_qr.o
96
+
drm-$(CONFIG_DRM_RAS) += drm_ras.o drm_ras_nl.o drm_ras_genl_family.o
96
97
obj-$(CONFIG_DRM) += drm.o
97
98
98
99
obj-$(CONFIG_DRM_PANEL_ORIENTATION_QUIRKS) += drm_panel_orientation_quirks.o
+6
drivers/gpu/drm/drm_drv.c
+6
drivers/gpu/drm/drm_drv.c
···
53
53
#include <drm/drm_panic.h>
54
54
#include <drm/drm_print.h>
55
55
#include <drm/drm_privacy_screen_machine.h>
56
+
#include <drm/drm_ras_genl_family.h>
56
57
57
58
#include "drm_crtc_internal.h"
58
59
#include "drm_internal.h"
···
1224
1223
1225
1224
static void drm_core_exit(void)
1226
1225
{
1226
+
drm_ras_genl_family_unregister();
1227
1227
drm_privacy_screen_lookup_exit();
1228
1228
drm_panic_exit();
1229
1229
accel_core_exit();
···
1262
1260
drm_panic_init();
1263
1261
1264
1262
drm_privacy_screen_lookup_init();
1263
+
1264
+
ret = drm_ras_genl_family_register();
1265
+
if (ret < 0)
1266
+
goto error;
1265
1267
1266
1268
drm_core_init_complete = true;
1267
1269
+354
drivers/gpu/drm/drm_ras.c
+354
drivers/gpu/drm/drm_ras.c
···
1
+
// SPDX-License-Identifier: MIT
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#include <linux/module.h>
7
+
#include <linux/kernel.h>
8
+
#include <linux/netdevice.h>
9
+
#include <linux/xarray.h>
10
+
#include <net/genetlink.h>
11
+
12
+
#include <drm/drm_ras.h>
13
+
14
+
#include "drm_ras_nl.h"
15
+
16
+
/**
17
+
* DOC: DRM RAS Node Management
18
+
*
19
+
* This module provides the infrastructure to manage RAS (Reliability,
20
+
* Availability, and Serviceability) nodes for DRM drivers. Each
21
+
* DRM driver may register one or more RAS nodes, which represent
22
+
* logical components capable of reporting error counters and other
23
+
* reliability metrics.
24
+
*
25
+
* The nodes are stored in a global xarray `drm_ras_xa` to allow
26
+
* efficient lookup by ID. Nodes can be registered or unregistered
27
+
* dynamically at runtime.
28
+
*
29
+
* A Generic Netlink family `drm_ras` exposes two main operations to
30
+
* userspace:
31
+
*
32
+
* 1. LIST_NODES: Dump all currently registered RAS nodes.
33
+
* The user receives an array of node IDs, names, and types.
34
+
*
35
+
* 2. GET_ERROR_COUNTER: Get error counters of a given node.
36
+
* Userspace must provide Node ID, Error ID (Optional for specific counter).
37
+
* Returns all counters of a node if only Node ID is provided or specific
38
+
* error counters.
39
+
*
40
+
* Node registration:
41
+
*
42
+
* - drm_ras_node_register(): Registers a new node and assigns
43
+
* it a unique ID in the xarray.
44
+
* - drm_ras_node_unregister(): Removes a previously registered
45
+
* node from the xarray.
46
+
*
47
+
* Node type:
48
+
*
49
+
* - ERROR_COUNTER:
50
+
* + Currently, only error counters are supported.
51
+
* + The driver must implement the query_error_counter() callback to provide
52
+
* the name and the value of the error counter.
53
+
* + The driver must provide a error_counter_range.last value informing the
54
+
* last valid error ID.
55
+
* + The driver can provide a error_counter_range.first value informing the
56
+
* first valid error ID.
57
+
* + The error counters in the driver doesn't need to be contiguous, but the
58
+
* driver must return -ENOENT to the query_error_counter as an indication
59
+
* that the ID should be skipped and not listed in the netlink API.
60
+
*
61
+
* Netlink handlers:
62
+
*
63
+
* - drm_ras_nl_list_nodes_dumpit(): Implements the LIST_NODES
64
+
* operation, iterating over the xarray.
65
+
* - drm_ras_nl_get_error_counter_dumpit(): Implements the GET_ERROR_COUNTER dumpit
66
+
* operation, fetching all counters from a specific node.
67
+
* - drm_ras_nl_get_error_counter_doit(): Implements the GET_ERROR_COUNTER doit
68
+
* operation, fetching a counter value from a specific node.
69
+
*/
70
+
71
+
static DEFINE_XARRAY_ALLOC(drm_ras_xa);
72
+
73
+
/*
74
+
* The netlink callback context carries dump state across multiple dumpit calls
75
+
*/
76
+
struct drm_ras_ctx {
77
+
/* Which xarray id to restart the dump from */
78
+
unsigned long restart;
79
+
};
80
+
81
+
/**
82
+
* drm_ras_nl_list_nodes_dumpit() - Dump all registered RAS nodes
83
+
* @skb: Netlink message buffer
84
+
* @cb: Callback context for multi-part dumps
85
+
*
86
+
* Iterates over all registered RAS nodes in the global xarray and appends
87
+
* their attributes (ID, name, type) to the given netlink message buffer.
88
+
* Uses @cb->ctx to track progress in case the message buffer fills up, allowing
89
+
* multi-part dump support. On buffer overflow, updates the context to resume
90
+
* from the last node on the next invocation.
91
+
*
92
+
* Return: 0 if all nodes fit in @skb, number of bytes added to @skb if
93
+
* the buffer filled up (requires multi-part continuation), or
94
+
* a negative error code on failure.
95
+
*/
96
+
int drm_ras_nl_list_nodes_dumpit(struct sk_buff *skb,
97
+
struct netlink_callback *cb)
98
+
{
99
+
const struct genl_info *info = genl_info_dump(cb);
100
+
struct drm_ras_ctx *ctx = (void *)cb->ctx;
101
+
struct drm_ras_node *node;
102
+
struct nlattr *hdr;
103
+
unsigned long id;
104
+
int ret;
105
+
106
+
xa_for_each_start(&drm_ras_xa, id, node, ctx->restart) {
107
+
hdr = genlmsg_iput(skb, info);
108
+
if (!hdr) {
109
+
ret = -EMSGSIZE;
110
+
break;
111
+
}
112
+
113
+
ret = nla_put_u32(skb, DRM_RAS_A_NODE_ATTRS_NODE_ID, node->id);
114
+
if (ret) {
115
+
genlmsg_cancel(skb, hdr);
116
+
break;
117
+
}
118
+
119
+
ret = nla_put_string(skb, DRM_RAS_A_NODE_ATTRS_DEVICE_NAME,
120
+
node->device_name);
121
+
if (ret) {
122
+
genlmsg_cancel(skb, hdr);
123
+
break;
124
+
}
125
+
126
+
ret = nla_put_string(skb, DRM_RAS_A_NODE_ATTRS_NODE_NAME,
127
+
node->node_name);
128
+
if (ret) {
129
+
genlmsg_cancel(skb, hdr);
130
+
break;
131
+
}
132
+
133
+
ret = nla_put_u32(skb, DRM_RAS_A_NODE_ATTRS_NODE_TYPE,
134
+
node->type);
135
+
if (ret) {
136
+
genlmsg_cancel(skb, hdr);
137
+
break;
138
+
}
139
+
140
+
genlmsg_end(skb, hdr);
141
+
}
142
+
143
+
if (ret == -EMSGSIZE)
144
+
ctx->restart = id;
145
+
146
+
return ret;
147
+
}
148
+
149
+
static int get_node_error_counter(u32 node_id, u32 error_id,
150
+
const char **name, u32 *value)
151
+
{
152
+
struct drm_ras_node *node;
153
+
154
+
node = xa_load(&drm_ras_xa, node_id);
155
+
if (!node || !node->query_error_counter)
156
+
return -ENOENT;
157
+
158
+
if (error_id < node->error_counter_range.first ||
159
+
error_id > node->error_counter_range.last)
160
+
return -EINVAL;
161
+
162
+
return node->query_error_counter(node, error_id, name, value);
163
+
}
164
+
165
+
static int msg_reply_value(struct sk_buff *msg, u32 error_id,
166
+
const char *error_name, u32 value)
167
+
{
168
+
int ret;
169
+
170
+
ret = nla_put_u32(msg, DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID, error_id);
171
+
if (ret)
172
+
return ret;
173
+
174
+
ret = nla_put_string(msg, DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_NAME,
175
+
error_name);
176
+
if (ret)
177
+
return ret;
178
+
179
+
return nla_put_u32(msg, DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_VALUE,
180
+
value);
181
+
}
182
+
183
+
static int doit_reply_value(struct genl_info *info, u32 node_id,
184
+
u32 error_id)
185
+
{
186
+
struct sk_buff *msg;
187
+
struct nlattr *hdr;
188
+
const char *error_name;
189
+
u32 value;
190
+
int ret;
191
+
192
+
msg = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
193
+
if (!msg)
194
+
return -ENOMEM;
195
+
196
+
hdr = genlmsg_iput(msg, info);
197
+
if (!hdr) {
198
+
nlmsg_free(msg);
199
+
return -EMSGSIZE;
200
+
}
201
+
202
+
ret = get_node_error_counter(node_id, error_id,
203
+
&error_name, &value);
204
+
if (ret)
205
+
return ret;
206
+
207
+
ret = msg_reply_value(msg, error_id, error_name, value);
208
+
if (ret) {
209
+
genlmsg_cancel(msg, hdr);
210
+
nlmsg_free(msg);
211
+
return ret;
212
+
}
213
+
214
+
genlmsg_end(msg, hdr);
215
+
216
+
return genlmsg_reply(msg, info);
217
+
}
218
+
219
+
/**
220
+
* drm_ras_nl_get_error_counter_dumpit() - Dump all Error Counters
221
+
* @skb: Netlink message buffer
222
+
* @cb: Callback context for multi-part dumps
223
+
*
224
+
* Iterates over all error counters in a given Node and appends
225
+
* their attributes (ID, name, value) to the given netlink message buffer.
226
+
* Uses @cb->ctx to track progress in case the message buffer fills up, allowing
227
+
* multi-part dump support. On buffer overflow, updates the context to resume
228
+
* from the last node on the next invocation.
229
+
*
230
+
* Return: 0 if all errors fit in @skb, number of bytes added to @skb if
231
+
* the buffer filled up (requires multi-part continuation), or
232
+
* a negative error code on failure.
233
+
*/
234
+
int drm_ras_nl_get_error_counter_dumpit(struct sk_buff *skb,
235
+
struct netlink_callback *cb)
236
+
{
237
+
const struct genl_info *info = genl_info_dump(cb);
238
+
struct drm_ras_ctx *ctx = (void *)cb->ctx;
239
+
struct drm_ras_node *node;
240
+
struct nlattr *hdr;
241
+
const char *error_name;
242
+
u32 node_id, error_id, value;
243
+
int ret;
244
+
245
+
if (!info->attrs || GENL_REQ_ATTR_CHECK(info, DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID))
246
+
return -EINVAL;
247
+
248
+
node_id = nla_get_u32(info->attrs[DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID]);
249
+
250
+
node = xa_load(&drm_ras_xa, node_id);
251
+
if (!node)
252
+
return -ENOENT;
253
+
254
+
for (error_id = max(node->error_counter_range.first, ctx->restart);
255
+
error_id <= node->error_counter_range.last;
256
+
error_id++) {
257
+
ret = get_node_error_counter(node_id, error_id,
258
+
&error_name, &value);
259
+
/*
260
+
* For non-contiguous range, driver return -ENOENT as indication
261
+
* to skip this ID when listing all errors.
262
+
*/
263
+
if (ret == -ENOENT)
264
+
continue;
265
+
if (ret)
266
+
return ret;
267
+
268
+
hdr = genlmsg_iput(skb, info);
269
+
270
+
if (!hdr) {
271
+
ret = -EMSGSIZE;
272
+
break;
273
+
}
274
+
275
+
ret = msg_reply_value(skb, error_id, error_name, value);
276
+
if (ret) {
277
+
genlmsg_cancel(skb, hdr);
278
+
break;
279
+
}
280
+
281
+
genlmsg_end(skb, hdr);
282
+
}
283
+
284
+
if (ret == -EMSGSIZE)
285
+
ctx->restart = error_id;
286
+
287
+
return ret;
288
+
}
289
+
290
+
/**
291
+
* drm_ras_nl_get_error_counter_doit() - Query an error counter of an node
292
+
* @skb: Netlink message buffer
293
+
* @info: Generic Netlink info containing attributes of the request
294
+
*
295
+
* Extracts the node ID and error ID from the netlink attributes and
296
+
* retrieves the current value of the corresponding error counter. Sends the
297
+
* result back to the requesting user via the standard Genl reply.
298
+
*
299
+
* Return: 0 on success, or negative errno on failure.
300
+
*/
301
+
int drm_ras_nl_get_error_counter_doit(struct sk_buff *skb,
302
+
struct genl_info *info)
303
+
{
304
+
u32 node_id, error_id;
305
+
306
+
if (!info->attrs ||
307
+
GENL_REQ_ATTR_CHECK(info, DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID) ||
308
+
GENL_REQ_ATTR_CHECK(info, DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID))
309
+
return -EINVAL;
310
+
311
+
node_id = nla_get_u32(info->attrs[DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID]);
312
+
error_id = nla_get_u32(info->attrs[DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID]);
313
+
314
+
return doit_reply_value(info, node_id, error_id);
315
+
}
316
+
317
+
/**
318
+
* drm_ras_node_register() - Register a new RAS node
319
+
* @node: Node structure to register
320
+
*
321
+
* Adds the given RAS node to the global node xarray and assigns it
322
+
* a unique ID. Both @node->name and @node->type must be valid.
323
+
*
324
+
* Return: 0 on success, or negative errno on failure:
325
+
*/
326
+
int drm_ras_node_register(struct drm_ras_node *node)
327
+
{
328
+
if (!node->device_name || !node->node_name)
329
+
return -EINVAL;
330
+
331
+
/* Currently, only Error Counter Endpoints are supported */
332
+
if (node->type != DRM_RAS_NODE_TYPE_ERROR_COUNTER)
333
+
return -EINVAL;
334
+
335
+
/* Mandatory entries for Error Counter Node */
336
+
if (node->type == DRM_RAS_NODE_TYPE_ERROR_COUNTER &&
337
+
(!node->error_counter_range.last || !node->query_error_counter))
338
+
return -EINVAL;
339
+
340
+
return xa_alloc(&drm_ras_xa, &node->id, node, xa_limit_32b, GFP_KERNEL);
341
+
}
342
+
EXPORT_SYMBOL(drm_ras_node_register);
343
+
344
+
/**
345
+
* drm_ras_node_unregister() - Unregister a previously registered node
346
+
* @node: Node structure to unregister
347
+
*
348
+
* Removes the given node from the global node xarray using its ID.
349
+
*/
350
+
void drm_ras_node_unregister(struct drm_ras_node *node)
351
+
{
352
+
xa_erase(&drm_ras_xa, node->id);
353
+
}
354
+
EXPORT_SYMBOL(drm_ras_node_unregister);
+42
drivers/gpu/drm/drm_ras_genl_family.c
+42
drivers/gpu/drm/drm_ras_genl_family.c
···
1
+
// SPDX-License-Identifier: MIT
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#include <drm/drm_ras_genl_family.h>
7
+
#include "drm_ras_nl.h"
8
+
9
+
/* Track family registration so the drm_exit can be called at any time */
10
+
static bool registered;
11
+
12
+
/**
13
+
* drm_ras_genl_family_register() - Register drm-ras genl family
14
+
*
15
+
* Only to be called one at drm_drv_init()
16
+
*/
17
+
int drm_ras_genl_family_register(void)
18
+
{
19
+
int ret;
20
+
21
+
registered = false;
22
+
23
+
ret = genl_register_family(&drm_ras_nl_family);
24
+
if (ret)
25
+
return ret;
26
+
27
+
registered = true;
28
+
return 0;
29
+
}
30
+
31
+
/**
32
+
* drm_ras_genl_family_unregister() - Unregister drm-ras genl family
33
+
*
34
+
* To be called one at drm_drv_exit() at any moment, but only once.
35
+
*/
36
+
void drm_ras_genl_family_unregister(void)
37
+
{
38
+
if (registered) {
39
+
genl_unregister_family(&drm_ras_nl_family);
40
+
registered = false;
41
+
}
42
+
}
+56
drivers/gpu/drm/drm_ras_nl.c
+56
drivers/gpu/drm/drm_ras_nl.c
···
1
+
// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
2
+
/* Do not edit directly, auto-generated from: */
3
+
/* Documentation/netlink/specs/drm_ras.yaml */
4
+
/* YNL-GEN kernel source */
5
+
/* To regenerate run: tools/net/ynl/ynl-regen.sh */
6
+
7
+
#include <net/netlink.h>
8
+
#include <net/genetlink.h>
9
+
10
+
#include "drm_ras_nl.h"
11
+
12
+
#include <uapi/drm/drm_ras.h>
13
+
14
+
/* DRM_RAS_CMD_GET_ERROR_COUNTER - do */
15
+
static const struct nla_policy drm_ras_get_error_counter_do_nl_policy[DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID + 1] = {
16
+
[DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID] = { .type = NLA_U32, },
17
+
[DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID] = { .type = NLA_U32, },
18
+
};
19
+
20
+
/* DRM_RAS_CMD_GET_ERROR_COUNTER - dump */
21
+
static const struct nla_policy drm_ras_get_error_counter_dump_nl_policy[DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID + 1] = {
22
+
[DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID] = { .type = NLA_U32, },
23
+
};
24
+
25
+
/* Ops table for drm_ras */
26
+
static const struct genl_split_ops drm_ras_nl_ops[] = {
27
+
{
28
+
.cmd = DRM_RAS_CMD_LIST_NODES,
29
+
.dumpit = drm_ras_nl_list_nodes_dumpit,
30
+
.flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DUMP,
31
+
},
32
+
{
33
+
.cmd = DRM_RAS_CMD_GET_ERROR_COUNTER,
34
+
.doit = drm_ras_nl_get_error_counter_doit,
35
+
.policy = drm_ras_get_error_counter_do_nl_policy,
36
+
.maxattr = DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID,
37
+
.flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
38
+
},
39
+
{
40
+
.cmd = DRM_RAS_CMD_GET_ERROR_COUNTER,
41
+
.dumpit = drm_ras_nl_get_error_counter_dumpit,
42
+
.policy = drm_ras_get_error_counter_dump_nl_policy,
43
+
.maxattr = DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID,
44
+
.flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DUMP,
45
+
},
46
+
};
47
+
48
+
struct genl_family drm_ras_nl_family __ro_after_init = {
49
+
.name = DRM_RAS_FAMILY_NAME,
50
+
.version = DRM_RAS_FAMILY_VERSION,
51
+
.netnsok = true,
52
+
.parallel_ops = true,
53
+
.module = THIS_MODULE,
54
+
.split_ops = drm_ras_nl_ops,
55
+
.n_split_ops = ARRAY_SIZE(drm_ras_nl_ops),
56
+
};
+24
drivers/gpu/drm/drm_ras_nl.h
+24
drivers/gpu/drm/drm_ras_nl.h
···
1
+
/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
2
+
/* Do not edit directly, auto-generated from: */
3
+
/* Documentation/netlink/specs/drm_ras.yaml */
4
+
/* YNL-GEN kernel header */
5
+
/* To regenerate run: tools/net/ynl/ynl-regen.sh */
6
+
7
+
#ifndef _LINUX_DRM_RAS_GEN_H
8
+
#define _LINUX_DRM_RAS_GEN_H
9
+
10
+
#include <net/netlink.h>
11
+
#include <net/genetlink.h>
12
+
13
+
#include <uapi/drm/drm_ras.h>
14
+
15
+
int drm_ras_nl_list_nodes_dumpit(struct sk_buff *skb,
16
+
struct netlink_callback *cb);
17
+
int drm_ras_nl_get_error_counter_doit(struct sk_buff *skb,
18
+
struct genl_info *info);
19
+
int drm_ras_nl_get_error_counter_dumpit(struct sk_buff *skb,
20
+
struct netlink_callback *cb);
21
+
22
+
extern struct genl_family drm_ras_nl_family;
23
+
24
+
#endif /* _LINUX_DRM_RAS_GEN_H */
+1
drivers/gpu/drm/xe/Makefile
+1
drivers/gpu/drm/xe/Makefile
+9
-3
drivers/gpu/drm/xe/regs/xe_gt_regs.h
+9
-3
drivers/gpu/drm/xe/regs/xe_gt_regs.h
···
20
20
#define MTL_MIRROR_TARGET_WP1 XE_REG(0xc60)
21
21
#define MTL_CAGF_MASK REG_GENMASK(8, 0)
22
22
#define MTL_CC_MASK REG_GENMASK(12, 9)
23
+
#define MTL_CRST 0xf
23
24
24
25
/* RPM unit config (Gen8+) */
25
26
#define RPM_CONFIG0 XE_REG(0xd00)
···
100
99
101
100
#define VE1_AUX_INV XE_REG(0x42b8)
102
101
#define AUX_INV REG_BIT(0)
102
+
103
+
#define GAMSTLB_CTRL XE_REG_MCR(0x477c)
104
+
#define DIS_PEND_GPA_LINK REG_BIT(13)
103
105
104
106
#define GAMSTLB_CTRL2 XE_REG_MCR(0x4788)
105
107
#define STLB_SINGLE_BANK_MODE REG_BIT(11)
···
184
180
185
181
#define COMMON_SLICE_CHICKEN3 XE_REG(0x7304, XE_REG_OPTION_MASKED)
186
182
#define XEHP_COMMON_SLICE_CHICKEN3 XE_REG_MCR(0x7304, XE_REG_OPTION_MASKED)
183
+
#define DISABLE_STATE_CACHE_PERF_FIX REG_BIT(13)
187
184
#define DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN REG_BIT(12)
188
185
#define XEHP_DUAL_SIMD8_SEQ_MERGE_DISABLE REG_BIT(12)
189
186
#define BLEND_EMB_FIX_DISABLE_IN_RCC REG_BIT(11)
···
222
217
#define XE2_FLAT_CCS_BASE_UPPER_ADDR_MASK REG_GENMASK(7, 0)
223
218
224
219
#define GSCPSMI_BASE XE_REG(0x880c)
225
-
226
-
#define CCCHKNREG2 XE_REG_MCR(0x881c)
227
-
#define LOCALITYDIS REG_BIT(7)
228
220
229
221
#define CCCHKNREG1 XE_REG_MCR(0x8828)
230
222
#define L3CMPCTRL REG_BIT(23)
···
453
451
#define BLEND_FILL_CACHING_OPT_DIS REG_BIT(3)
454
452
455
453
#define XEHPC_L3CLOS_MASK(i) XE_REG_MCR(0xb194 + (i) * 8)
454
+
455
+
#define L2COMPUTESIDECTRL XE_REG_MCR(0xb1c0)
456
+
#define CECTRL REG_GENMASK(2, 1)
457
+
#define CECTRL_CENODATA_ALWAYS REG_FIELD_PREP(CECTRL, 0x0)
456
458
457
459
#define XE2_GLOBAL_INVAL XE_REG(0xb404)
458
460
+77
-9
drivers/gpu/drm/xe/regs/xe_hw_error_regs.h
+77
-9
drivers/gpu/drm/xe/regs/xe_hw_error_regs.h
···
6
6
#ifndef _XE_HW_ERROR_REGS_H_
7
7
#define _XE_HW_ERROR_REGS_H_
8
8
9
-
#define HEC_UNCORR_ERR_STATUS(base) XE_REG((base) + 0x118)
10
-
#define UNCORR_FW_REPORTED_ERR BIT(6)
9
+
#define HEC_UNCORR_ERR_STATUS(base) XE_REG((base) + 0x118)
10
+
#define UNCORR_FW_REPORTED_ERR REG_BIT(6)
11
11
12
-
#define HEC_UNCORR_FW_ERR_DW0(base) XE_REG((base) + 0x124)
12
+
#define HEC_UNCORR_FW_ERR_DW0(base) XE_REG((base) + 0x124)
13
13
14
-
#define DEV_ERR_STAT_NONFATAL 0x100178
15
-
#define DEV_ERR_STAT_CORRECTABLE 0x10017c
16
-
#define DEV_ERR_STAT_REG(x) XE_REG(_PICK_EVEN((x), \
17
-
DEV_ERR_STAT_CORRECTABLE, \
18
-
DEV_ERR_STAT_NONFATAL))
19
-
#define XE_CSC_ERROR BIT(17)
14
+
#define ERR_STAT_GT_COR 0x100160
15
+
#define EU_GRF_COR_ERR REG_BIT(15)
16
+
#define EU_IC_COR_ERR REG_BIT(14)
17
+
#define SLM_COR_ERR REG_BIT(13)
18
+
#define GUC_COR_ERR REG_BIT(1)
19
+
20
+
#define ERR_STAT_GT_NONFATAL 0x100164
21
+
#define ERR_STAT_GT_FATAL 0x100168
22
+
#define EU_GRF_FAT_ERR REG_BIT(15)
23
+
#define SLM_FAT_ERR REG_BIT(13)
24
+
#define GUC_FAT_ERR REG_BIT(6)
25
+
#define FPU_FAT_ERR REG_BIT(3)
26
+
27
+
#define ERR_STAT_GT_REG(x) XE_REG(_PICK_EVEN((x), \
28
+
ERR_STAT_GT_COR, \
29
+
ERR_STAT_GT_NONFATAL))
30
+
31
+
#define PVC_COR_ERR_MASK (GUC_COR_ERR | SLM_COR_ERR | \
32
+
EU_IC_COR_ERR | EU_GRF_COR_ERR)
33
+
34
+
#define PVC_FAT_ERR_MASK (FPU_FAT_ERR | GUC_FAT_ERR | \
35
+
EU_GRF_FAT_ERR | SLM_FAT_ERR)
36
+
37
+
#define DEV_ERR_STAT_NONFATAL 0x100178
38
+
#define DEV_ERR_STAT_CORRECTABLE 0x10017c
39
+
#define DEV_ERR_STAT_REG(x) XE_REG(_PICK_EVEN((x), \
40
+
DEV_ERR_STAT_CORRECTABLE, \
41
+
DEV_ERR_STAT_NONFATAL))
42
+
43
+
#define XE_CSC_ERROR 17
44
+
#define XE_SOC_ERROR 16
45
+
#define XE_GT_ERROR 0
46
+
47
+
#define ERR_STAT_GT_FATAL_VECTOR_0 0x100260
48
+
#define ERR_STAT_GT_FATAL_VECTOR_1 0x100264
49
+
50
+
#define ERR_STAT_GT_FATAL_VECTOR_REG(x) XE_REG(_PICK_EVEN((x), \
51
+
ERR_STAT_GT_FATAL_VECTOR_0, \
52
+
ERR_STAT_GT_FATAL_VECTOR_1))
53
+
54
+
#define ERR_STAT_GT_COR_VECTOR_0 0x1002a0
55
+
#define ERR_STAT_GT_COR_VECTOR_1 0x1002a4
56
+
57
+
#define ERR_STAT_GT_COR_VECTOR_REG(x) XE_REG(_PICK_EVEN((x), \
58
+
ERR_STAT_GT_COR_VECTOR_0, \
59
+
ERR_STAT_GT_COR_VECTOR_1))
60
+
61
+
#define ERR_STAT_GT_VECTOR_REG(hw_err, x) (hw_err == HARDWARE_ERROR_CORRECTABLE ? \
62
+
ERR_STAT_GT_COR_VECTOR_REG(x) : \
63
+
ERR_STAT_GT_FATAL_VECTOR_REG(x))
64
+
65
+
#define SOC_PVC_MASTER_BASE 0x282000
66
+
#define SOC_PVC_SLAVE_BASE 0x283000
67
+
68
+
#define SOC_GCOERRSTS 0x200
69
+
#define SOC_GNFERRSTS 0x210
70
+
#define SOC_GLOBAL_ERR_STAT_REG(base, x) XE_REG(_PICK_EVEN((x), \
71
+
(base) + SOC_GCOERRSTS, \
72
+
(base) + SOC_GNFERRSTS))
73
+
#define SOC_SLAVE_IEH REG_BIT(1)
74
+
#define SOC_IEH0_LOCAL_ERR_STATUS REG_BIT(0)
75
+
#define SOC_IEH1_LOCAL_ERR_STATUS REG_BIT(0)
76
+
77
+
#define SOC_GSYSEVTCTL 0x264
78
+
#define SOC_GSYSEVTCTL_REG(master, slave, x) XE_REG(_PICK_EVEN((x), \
79
+
(master) + SOC_GSYSEVTCTL, \
80
+
(slave) + SOC_GSYSEVTCTL))
81
+
82
+
#define SOC_LERRUNCSTS 0x280
83
+
#define SOC_LERRCORSTS 0x294
84
+
#define SOC_LOCAL_ERR_STAT_REG(base, hw_err) XE_REG(hw_err == HARDWARE_ERROR_CORRECTABLE ? \
85
+
(base) + SOC_LERRCORSTS : \
86
+
(base) + SOC_LERRUNCSTS)
87
+
20
88
#endif
+1
-1
drivers/gpu/drm/xe/tests/xe_gt_sriov_pf_config_kunit.c
+1
-1
drivers/gpu/drm/xe/tests/xe_gt_sriov_pf_config_kunit.c
+55
drivers/gpu/drm/xe/xe_bo.c
+55
drivers/gpu/drm/xe/xe_bo.c
···
3331
3331
}
3332
3332
3333
3333
/**
3334
+
* xe_bo_decompress - schedule in-place decompress and install fence
3335
+
* @bo: buffer object (caller should hold drm_exec reservations for VM+BO)
3336
+
*
3337
+
* Schedules an in-place resolve via the migrate layer and installs the
3338
+
* returned dma_fence into the BO kernel reservation slot (DMA_RESV_USAGE_KERNEL).
3339
+
* In preempt fence mode, this operation interrupts hardware execution
3340
+
* which is expensive. Page fault mode is recommended for better performance.
3341
+
*
3342
+
* The resolve path only runs for VRAM-backed buffers (currently dGPU-only);
3343
+
* iGPU/system-memory objects fail the resource check and bypass the resolve.
3344
+
*
3345
+
* Returns 0 on success, negative errno on error.
3346
+
*/
3347
+
int xe_bo_decompress(struct xe_bo *bo)
3348
+
{
3349
+
struct xe_device *xe = xe_bo_device(bo);
3350
+
struct xe_tile *tile = xe_device_get_root_tile(xe);
3351
+
struct dma_fence *decomp_fence = NULL;
3352
+
struct ttm_operation_ctx op_ctx = {
3353
+
.interruptible = true,
3354
+
.no_wait_gpu = false,
3355
+
.gfp_retry_mayfail = false,
3356
+
};
3357
+
int err = 0;
3358
+
3359
+
/* Silently skip decompression for non-VRAM buffers */
3360
+
if (!bo->ttm.resource || !mem_type_is_vram(bo->ttm.resource->mem_type))
3361
+
return 0;
3362
+
3363
+
/* Notify before scheduling resolve */
3364
+
err = xe_bo_move_notify(bo, &op_ctx);
3365
+
if (err)
3366
+
return err;
3367
+
3368
+
/* Reserve fence slot before scheduling */
3369
+
err = dma_resv_reserve_fences(bo->ttm.base.resv, 1);
3370
+
if (err)
3371
+
return err;
3372
+
3373
+
/* Schedule the in-place decompression */
3374
+
decomp_fence = xe_migrate_resolve(tile->migrate,
3375
+
bo,
3376
+
bo->ttm.resource);
3377
+
3378
+
if (IS_ERR(decomp_fence))
3379
+
return PTR_ERR(decomp_fence);
3380
+
3381
+
/* Install kernel-usage fence */
3382
+
dma_resv_add_fence(bo->ttm.base.resv, decomp_fence, DMA_RESV_USAGE_KERNEL);
3383
+
dma_fence_put(decomp_fence);
3384
+
3385
+
return 0;
3386
+
}
3387
+
3388
+
/**
3334
3389
* xe_bo_lock() - Lock the buffer object's dma_resv object
3335
3390
* @bo: The struct xe_bo whose lock is to be taken
3336
3391
* @intr: Whether to perform any wait interruptible
+2
drivers/gpu/drm/xe/xe_bo.h
+2
drivers/gpu/drm/xe/xe_bo.h
+1
-4
drivers/gpu/drm/xe/xe_device.c
+1
-4
drivers/gpu/drm/xe/xe_device.c
···
1074
1074
struct xe_gt *gt;
1075
1075
u8 id;
1076
1076
1077
-
for_each_gt(gt, xe, id) {
1078
-
if (xe_gt_is_media_type(gt))
1079
-
continue;
1080
-
1077
+
for_each_gt_with_type(gt, xe, id, BIT(XE_GT_TYPE_MAIN)) {
1081
1078
CLASS(xe_force_wake, fw_ref)(gt_to_fw(gt), XE_FW_GT);
1082
1079
if (!fw_ref.domains)
1083
1080
return;
+4
drivers/gpu/drm/xe/xe_device.h
+4
drivers/gpu/drm/xe/xe_device.h
···
131
131
for ((id__) = 0; (id__) < (xe__)->info.tile_count * (xe__)->info.max_gt_per_tile; (id__)++) \
132
132
for_each_if((gt__) = xe_device_get_gt((xe__), (id__)))
133
133
134
+
#define for_each_gt_with_type(gt__, xe__, id__, typemask__) \
135
+
for_each_gt((gt__), (xe__), (id__)) \
136
+
for_each_if((typemask__) & BIT((gt__)->info.type))
137
+
134
138
#define for_each_gt_on_tile(gt__, tile__, id__) \
135
139
for_each_gt((gt__), (tile__)->xe, (id__)) \
136
140
for_each_if((gt__)->tile == (tile__))
+6
-10
drivers/gpu/drm/xe/xe_device_types.h
+6
-10
drivers/gpu/drm/xe/xe_device_types.h
···
13
13
#include <drm/ttm/ttm_device.h>
14
14
15
15
#include "xe_devcoredump_types.h"
16
+
#include "xe_drm_ras_types.h"
16
17
#include "xe_heci_gsc.h"
17
18
#include "xe_late_bind_fw_types.h"
18
19
#include "xe_oa_types.h"
···
82
81
83
82
#define XE_MAX_ASID (BIT(20))
84
83
85
-
#define IS_PLATFORM_STEP(_xe, _platform, min_step, max_step) \
86
-
((_xe)->info.platform == (_platform) && \
87
-
(_xe)->info.step.graphics >= (min_step) && \
88
-
(_xe)->info.step.graphics < (max_step))
89
-
#define IS_SUBPLATFORM_STEP(_xe, _platform, sub, min_step, max_step) \
90
-
((_xe)->info.platform == (_platform) && \
91
-
(_xe)->info.subplatform == (sub) && \
92
-
(_xe)->info.step.graphics >= (min_step) && \
93
-
(_xe)->info.step.graphics < (max_step))
94
-
95
84
/**
96
85
* struct xe_device - Top level struct of Xe device
97
86
*/
···
144
153
145
154
/** @info.force_execlist: Forced execlist submission */
146
155
u8 force_execlist:1;
156
+
/** @info.has_access_counter: Device supports access counter */
157
+
u8 has_access_counter:1;
147
158
/** @info.has_asid: Has address space ID */
148
159
u8 has_asid:1;
149
160
/** @info.has_atomic_enable_pte_bit: Device has atomic enable PTE bit */
···
501
508
502
509
/** @pmu: performance monitoring unit */
503
510
struct xe_pmu pmu;
511
+
512
+
/** @ras: RAS structure for device */
513
+
struct xe_drm_ras ras;
504
514
505
515
/** @i2c: I2C host controller */
506
516
struct xe_i2c *i2c;
+1
drivers/gpu/drm/xe/xe_device_wa_oob.rules
+1
drivers/gpu/drm/xe/xe_device_wa_oob.rules
+186
drivers/gpu/drm/xe/xe_drm_ras.c
+186
drivers/gpu/drm/xe/xe_drm_ras.c
···
1
+
// SPDX-License-Identifier: MIT
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#include <linux/bitmap.h>
7
+
8
+
#include <drm/drm_managed.h>
9
+
#include <drm/drm_print.h>
10
+
#include <drm/drm_ras.h>
11
+
12
+
#include "xe_device_types.h"
13
+
#include "xe_drm_ras.h"
14
+
15
+
static const char * const error_components[] = DRM_XE_RAS_ERROR_COMPONENT_NAMES;
16
+
static const char * const error_severity[] = DRM_XE_RAS_ERROR_SEVERITY_NAMES;
17
+
18
+
static int hw_query_error_counter(struct xe_drm_ras_counter *info,
19
+
u32 error_id, const char **name, u32 *val)
20
+
{
21
+
if (!info || !info[error_id].name)
22
+
return -ENOENT;
23
+
24
+
*name = info[error_id].name;
25
+
*val = atomic_read(&info[error_id].counter);
26
+
27
+
return 0;
28
+
}
29
+
30
+
static int query_uncorrectable_error_counter(struct drm_ras_node *ep, u32 error_id,
31
+
const char **name, u32 *val)
32
+
{
33
+
struct xe_device *xe = ep->priv;
34
+
struct xe_drm_ras *ras = &xe->ras;
35
+
struct xe_drm_ras_counter *info = ras->info[DRM_XE_RAS_ERR_SEV_UNCORRECTABLE];
36
+
37
+
return hw_query_error_counter(info, error_id, name, val);
38
+
}
39
+
40
+
static int query_correctable_error_counter(struct drm_ras_node *ep, u32 error_id,
41
+
const char **name, u32 *val)
42
+
{
43
+
struct xe_device *xe = ep->priv;
44
+
struct xe_drm_ras *ras = &xe->ras;
45
+
struct xe_drm_ras_counter *info = ras->info[DRM_XE_RAS_ERR_SEV_CORRECTABLE];
46
+
47
+
return hw_query_error_counter(info, error_id, name, val);
48
+
}
49
+
50
+
static struct xe_drm_ras_counter *allocate_and_copy_counters(struct xe_device *xe)
51
+
{
52
+
struct xe_drm_ras_counter *counter;
53
+
int i;
54
+
55
+
counter = kcalloc(DRM_XE_RAS_ERR_COMP_MAX, sizeof(*counter), GFP_KERNEL);
56
+
if (!counter)
57
+
return ERR_PTR(-ENOMEM);
58
+
59
+
for (i = DRM_XE_RAS_ERR_COMP_CORE_COMPUTE; i < DRM_XE_RAS_ERR_COMP_MAX; i++) {
60
+
if (!error_components[i])
61
+
continue;
62
+
63
+
counter[i].name = error_components[i];
64
+
atomic_set(&counter[i].counter, 0);
65
+
}
66
+
67
+
return counter;
68
+
}
69
+
70
+
static int assign_node_params(struct xe_device *xe, struct drm_ras_node *node,
71
+
const enum drm_xe_ras_error_severity severity)
72
+
{
73
+
struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
74
+
struct xe_drm_ras *ras = &xe->ras;
75
+
const char *device_name;
76
+
77
+
device_name = kasprintf(GFP_KERNEL, "%04x:%02x:%02x.%d",
78
+
pci_domain_nr(pdev->bus), pdev->bus->number,
79
+
PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
80
+
81
+
if (!device_name)
82
+
return -ENOMEM;
83
+
84
+
node->device_name = device_name;
85
+
node->node_name = error_severity[severity];
86
+
node->type = DRM_RAS_NODE_TYPE_ERROR_COUNTER;
87
+
node->error_counter_range.first = DRM_XE_RAS_ERR_COMP_CORE_COMPUTE;
88
+
node->error_counter_range.last = DRM_XE_RAS_ERR_COMP_MAX - 1;
89
+
node->priv = xe;
90
+
91
+
ras->info[severity] = allocate_and_copy_counters(xe);
92
+
if (IS_ERR(ras->info[severity]))
93
+
return PTR_ERR(ras->info[severity]);
94
+
95
+
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
96
+
node->query_error_counter = query_correctable_error_counter;
97
+
else
98
+
node->query_error_counter = query_uncorrectable_error_counter;
99
+
100
+
return 0;
101
+
}
102
+
103
+
static void cleanup_node_param(struct xe_drm_ras *ras, const enum drm_xe_ras_error_severity severity)
104
+
{
105
+
struct drm_ras_node *node = &ras->node[severity];
106
+
107
+
kfree(ras->info[severity]);
108
+
ras->info[severity] = NULL;
109
+
110
+
kfree(node->device_name);
111
+
node->device_name = NULL;
112
+
}
113
+
114
+
static int register_nodes(struct xe_device *xe)
115
+
{
116
+
struct xe_drm_ras *ras = &xe->ras;
117
+
int i;
118
+
119
+
for_each_error_severity(i) {
120
+
struct drm_ras_node *node = &ras->node[i];
121
+
int ret;
122
+
123
+
ret = assign_node_params(xe, node, i);
124
+
if (ret) {
125
+
cleanup_node_param(ras, i);
126
+
return ret;
127
+
}
128
+
129
+
ret = drm_ras_node_register(node);
130
+
if (ret) {
131
+
cleanup_node_param(ras, i);
132
+
return ret;
133
+
}
134
+
}
135
+
136
+
return 0;
137
+
}
138
+
139
+
static void xe_drm_ras_unregister_nodes(struct drm_device *device, void *arg)
140
+
{
141
+
struct xe_device *xe = arg;
142
+
struct xe_drm_ras *ras = &xe->ras;
143
+
int i;
144
+
145
+
for_each_error_severity(i) {
146
+
struct drm_ras_node *node = &ras->node[i];
147
+
148
+
drm_ras_node_unregister(node);
149
+
cleanup_node_param(ras, i);
150
+
}
151
+
}
152
+
153
+
/**
154
+
* xe_drm_ras_init() - Initialize DRM RAS
155
+
* @xe: xe device instance
156
+
*
157
+
* Allocate and register DRM RAS nodes per device
158
+
*
159
+
* Return: 0 on success, negative error code otherwise.
160
+
*/
161
+
int xe_drm_ras_init(struct xe_device *xe)
162
+
{
163
+
struct xe_drm_ras *ras = &xe->ras;
164
+
struct drm_ras_node *node;
165
+
int err;
166
+
167
+
node = drmm_kcalloc(&xe->drm, DRM_XE_RAS_ERR_SEV_MAX, sizeof(*node), GFP_KERNEL);
168
+
if (!node)
169
+
return -ENOMEM;
170
+
171
+
ras->node = node;
172
+
173
+
err = register_nodes(xe);
174
+
if (err) {
175
+
drm_err(&xe->drm, "Failed to register DRM RAS nodes (%pe)\n", ERR_PTR(err));
176
+
return err;
177
+
}
178
+
179
+
err = drmm_add_action_or_reset(&xe->drm, xe_drm_ras_unregister_nodes, xe);
180
+
if (err) {
181
+
drm_err(&xe->drm, "Failed to add action for Xe DRM RAS (%pe)\n", ERR_PTR(err));
182
+
return err;
183
+
}
184
+
185
+
return 0;
186
+
}
+15
drivers/gpu/drm/xe/xe_drm_ras.h
+15
drivers/gpu/drm/xe/xe_drm_ras.h
···
1
+
/* SPDX-License-Identifier: MIT */
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
#ifndef XE_DRM_RAS_H_
6
+
#define XE_DRM_RAS_H_
7
+
8
+
struct xe_device;
9
+
10
+
#define for_each_error_severity(i) \
11
+
for (i = 0; i < DRM_XE_RAS_ERR_SEV_MAX; i++)
12
+
13
+
int xe_drm_ras_init(struct xe_device *xe);
14
+
15
+
#endif
+48
drivers/gpu/drm/xe/xe_drm_ras_types.h
+48
drivers/gpu/drm/xe/xe_drm_ras_types.h
···
1
+
/* SPDX-License-Identifier: MIT */
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#ifndef _XE_DRM_RAS_TYPES_H_
7
+
#define _XE_DRM_RAS_TYPES_H_
8
+
9
+
#include <linux/atomic.h>
10
+
#include <drm/xe_drm.h>
11
+
12
+
struct drm_ras_node;
13
+
14
+
/* Error categories reported by hardware */
15
+
enum hardware_error {
16
+
HARDWARE_ERROR_CORRECTABLE = 0,
17
+
HARDWARE_ERROR_NONFATAL,
18
+
HARDWARE_ERROR_FATAL,
19
+
HARDWARE_ERROR_MAX
20
+
};
21
+
22
+
/**
23
+
* struct xe_drm_ras_counter - XE RAS counter
24
+
*
25
+
* This structure contains error component and counter information
26
+
*/
27
+
struct xe_drm_ras_counter {
28
+
/** @name: error component name */
29
+
const char *name;
30
+
31
+
/** @counter: count of error */
32
+
atomic_t counter;
33
+
};
34
+
35
+
/**
36
+
* struct xe_drm_ras - XE DRM RAS structure
37
+
*
38
+
* This structure has details of error counters
39
+
*/
40
+
struct xe_drm_ras {
41
+
/** @node: DRM RAS node */
42
+
struct drm_ras_node *node;
43
+
44
+
/** @info: info array for all types of errors */
45
+
struct xe_drm_ras_counter *info[DRM_XE_RAS_ERR_SEV_MAX];
46
+
};
47
+
48
+
#endif
+130
-1
drivers/gpu/drm/xe/xe_exec_queue.c
+130
-1
drivers/gpu/drm/xe/xe_exec_queue.c
···
353
353
if (!(exec_queue_flags & EXEC_QUEUE_FLAG_KERNEL))
354
354
flags |= XE_LRC_CREATE_USER_CTX;
355
355
356
+
if (q->flags & EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX)
357
+
flags |= XE_LRC_DISABLE_STATE_CACHE_PERF_FIX;
358
+
356
359
err = q->ops->init(q);
357
360
if (err)
358
361
return err;
···
402
399
return err;
403
400
}
404
401
402
+
/**
403
+
* xe_exec_queue_create() - Create an exec queue
404
+
* @xe: Xe device
405
+
* @vm: VM for the exec queue
406
+
* @logical_mask: Logical mask of HW engines
407
+
* @width: Width of the exec queue (number of LRCs)
408
+
* @hwe: Hardware engine
409
+
* @flags: Exec queue creation flags
410
+
* @extensions: Extensions for exec queue creation
411
+
*
412
+
* Create an exec queue (allocate and initialize) with the specified parameters
413
+
*
414
+
* Return: Pointer to the created exec queue on success, ERR_PTR on failure
415
+
*/
405
416
struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *vm,
406
417
u32 logical_mask, u16 width,
407
418
struct xe_hw_engine *hwe, u32 flags,
···
459
442
}
460
443
ALLOW_ERROR_INJECTION(xe_exec_queue_create, ERRNO);
461
444
445
+
/**
446
+
* xe_exec_queue_create_class() - Create an exec queue for a specific engine class
447
+
* @xe: Xe device
448
+
* @gt: GT for the exec queue
449
+
* @vm: VM for the exec queue
450
+
* @class: Engine class
451
+
* @flags: Exec queue creation flags
452
+
* @extensions: Extensions for exec queue creation
453
+
*
454
+
* Create an exec queue for the specified engine class.
455
+
*
456
+
* Return: Pointer to the created exec queue on success, ERR_PTR on failure
457
+
*/
462
458
struct xe_exec_queue *xe_exec_queue_create_class(struct xe_device *xe, struct xe_gt *gt,
463
459
struct xe_vm *vm,
464
460
enum xe_engine_class class,
···
563
533
}
564
534
ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
565
535
536
+
/**
537
+
* xe_exec_queue_destroy() - Destroy an exec queue
538
+
* @ref: Reference count of the exec queue
539
+
*
540
+
* Called when the last reference to the exec queue is dropped.
541
+
* Cleans up all resources associated with the exec queue.
542
+
* This function should not be called directly; use xe_exec_queue_put() instead.
543
+
*/
566
544
void xe_exec_queue_destroy(struct kref *ref)
567
545
{
568
546
struct xe_exec_queue *q = container_of(ref, struct xe_exec_queue, refcount);
···
603
565
q->ops->destroy(q);
604
566
}
605
567
568
+
/**
569
+
* xe_exec_queue_fini() - Finalize an exec queue
570
+
* @q: The exec queue
571
+
*
572
+
* Finalizes the exec queue by updating run ticks, releasing LRC references,
573
+
* and freeing the queue structure. This is called after the queue has been
574
+
* destroyed and all references have been dropped.
575
+
*/
606
576
void xe_exec_queue_fini(struct xe_exec_queue *q)
607
577
{
608
578
/*
···
625
579
__xe_exec_queue_free(q);
626
580
}
627
581
582
+
/**
583
+
* xe_exec_queue_assign_name() - Assign a name to an exec queue
584
+
* @q: The exec queue
585
+
* @instance: Instance number for the engine
586
+
*
587
+
* Assigns a human-readable name to the exec queue based on its engine class
588
+
* and instance number (e.g., "rcs0", "vcs1", "bcs2").
589
+
*/
628
590
void xe_exec_queue_assign_name(struct xe_exec_queue *q, u32 instance)
629
591
{
630
592
switch (q->class) {
···
659
605
}
660
606
}
661
607
608
+
/**
609
+
* xe_exec_queue_lookup() - Look up an exec queue by ID
610
+
* @xef: Xe file private data
611
+
* @id: Exec queue ID
612
+
*
613
+
* Looks up an exec queue by its ID and increments its reference count.
614
+
*
615
+
* Return: Pointer to the exec queue if found, NULL otherwise
616
+
*/
662
617
struct xe_exec_queue *xe_exec_queue_lookup(struct xe_file *xef, u32 id)
663
618
{
664
619
struct xe_exec_queue *q;
···
681
618
return q;
682
619
}
683
620
621
+
/**
622
+
* xe_exec_queue_device_get_max_priority() - Get maximum priority for an exec queues
623
+
* @xe: Xe device
624
+
*
625
+
* Returns the maximum priority level that can be assigned to an exec queues.
626
+
*
627
+
* Return: Maximum priority level (HIGH if CAP_SYS_NICE, NORMAL otherwise)
628
+
*/
684
629
enum xe_exec_queue_priority
685
630
xe_exec_queue_device_get_max_priority(struct xe_device *xe)
686
631
{
···
981
910
return q->ops->set_multi_queue_priority(q, value);
982
911
}
983
912
913
+
static int exec_queue_set_state_cache_perf_fix(struct xe_device *xe, struct xe_exec_queue *q,
914
+
u64 value)
915
+
{
916
+
if (XE_IOCTL_DBG(xe, q->class != XE_ENGINE_CLASS_RENDER))
917
+
return -EOPNOTSUPP;
918
+
919
+
q->flags |= value != 0 ? EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX : 0;
920
+
921
+
return 0;
922
+
}
923
+
984
924
typedef int (*xe_exec_queue_set_property_fn)(struct xe_device *xe,
985
925
struct xe_exec_queue *q,
986
926
u64 value);
···
1004
922
[DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_GROUP] = exec_queue_set_multi_group,
1005
923
[DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY] =
1006
924
exec_queue_set_multi_queue_priority,
925
+
[DRM_XE_EXEC_QUEUE_SET_DISABLE_STATE_CACHE_PERF_FIX] =
926
+
exec_queue_set_state_cache_perf_fix,
1007
927
};
1008
928
929
+
/**
930
+
* xe_exec_queue_set_property_ioctl() - Set a property on an exec queue
931
+
* @dev: DRM device
932
+
* @data: IOCTL data
933
+
* @file: DRM file
934
+
*
935
+
* Allows setting properties on an existing exec queue. Currently only
936
+
* supports setting multi-queue priority.
937
+
*
938
+
* Return: 0 on success, negative error code on failure
939
+
*/
1009
940
int xe_exec_queue_set_property_ioctl(struct drm_device *dev, void *data,
1010
941
struct drm_file *file)
1011
942
{
···
1101
1006
ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE &&
1102
1007
ext.property != DRM_XE_EXEC_QUEUE_SET_HANG_REPLAY_STATE &&
1103
1008
ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_GROUP &&
1104
-
ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY))
1009
+
ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY &&
1010
+
ext.property != DRM_XE_EXEC_QUEUE_SET_DISABLE_STATE_CACHE_PERF_FIX))
1105
1011
return -EINVAL;
1106
1012
1107
1013
idx = array_index_nospec(ext.property, ARRAY_SIZE(exec_queue_set_property_funcs));
···
1244
1148
return false;
1245
1149
}
1246
1150
1151
+
/**
1152
+
* xe_exec_queue_create_ioctl() - Create an exec queue via IOCTL
1153
+
* @dev: DRM device
1154
+
* @data: IOCTL data
1155
+
* @file: DRM file
1156
+
*
1157
+
* Creates a new exec queue based on user-provided parameters. Supports
1158
+
* creating VM bind queues, regular exec queues, multi-lrc exec queues
1159
+
* and multi-queue groups.
1160
+
*
1161
+
* Return: 0 on success with exec_queue_id filled in, negative error code on failure
1162
+
*/
1247
1163
int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data,
1248
1164
struct drm_file *file)
1249
1165
{
···
1432
1324
return err;
1433
1325
}
1434
1326
1327
+
/**
1328
+
* xe_exec_queue_get_property_ioctl() - Get a property from an exec queue
1329
+
* @dev: DRM device
1330
+
* @data: IOCTL data
1331
+
* @file: DRM file
1332
+
*
1333
+
* Retrieves property values from an existing exec queue. Currently supports
1334
+
* getting the ban/reset status.
1335
+
*
1336
+
* Return: 0 on success with value filled in, negative error code on failure
1337
+
*/
1435
1338
int xe_exec_queue_get_property_ioctl(struct drm_device *dev, void *data,
1436
1339
struct drm_file *file)
1437
1340
{
···
1580
1461
xe_vm_remove_compute_exec_queue(q->vm, q);
1581
1462
}
1582
1463
1464
+
/**
1465
+
* xe_exec_queue_destroy_ioctl() - Destroy an exec queue via IOCTL
1466
+
* @dev: DRM device
1467
+
* @data: IOCTL data
1468
+
* @file: DRM file
1469
+
*
1470
+
* Destroys an existing exec queue and releases its reference.
1471
+
*
1472
+
* Return: 0 on success, negative error code on failure
1473
+
*/
1583
1474
int xe_exec_queue_destroy_ioctl(struct drm_device *dev, void *data,
1584
1475
struct drm_file *file)
1585
1476
{
+2
drivers/gpu/drm/xe/xe_exec_queue_types.h
+2
drivers/gpu/drm/xe/xe_exec_queue_types.h
···
134
134
#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(5)
135
135
/* for migration (kernel copy, clear, bind) jobs */
136
136
#define EXEC_QUEUE_FLAG_MIGRATE BIT(6)
137
+
/* for programming COMMON_SLICE_CHICKEN3 on first submission */
138
+
#define EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX BIT(7)
137
139
138
140
/**
139
141
* @flags: flags for this exec queue, should statically setup aside from ban
+41
drivers/gpu/drm/xe/xe_gt.c
+41
drivers/gpu/drm/xe/xe_gt.c
···
10
10
#include <drm/drm_managed.h>
11
11
#include <uapi/drm/xe_drm.h>
12
12
13
+
#include <generated/xe_device_wa_oob.h>
13
14
#include <generated/xe_wa_oob.h>
14
15
15
16
#include "instructions/xe_alu_commands.h"
···
39
38
#include "xe_gt_topology.h"
40
39
#include "xe_guc_exec_queue_types.h"
41
40
#include "xe_guc_pc.h"
41
+
#include "xe_guc_rc.h"
42
42
#include "xe_guc_submit.h"
43
43
#include "xe_hw_fence.h"
44
44
#include "xe_hw_engine_class_sysfs.h"
···
452
450
return err;
453
451
}
454
452
453
+
static void wa_14026539277(struct xe_gt *gt)
454
+
{
455
+
struct xe_device *xe = gt_to_xe(gt);
456
+
u32 val;
457
+
458
+
/*
459
+
* FIXME: We currently can't use FUNC(xe_rtp_match_not_sriov_vf) in the
460
+
* rules for Wa_14026539277 due to xe_wa_process_device_oob() being
461
+
* called before xe_sriov_probe_early(); and we can't move the call to
462
+
* the former to happen after the latter because MMIO read functions
463
+
* already depend on a device OOB workaround. This needs to be fixed by
464
+
* allowing workaround checks to happen at different stages of driver
465
+
* initialization.
466
+
*/
467
+
if (IS_SRIOV_VF(xe))
468
+
return;
469
+
470
+
if (!XE_DEVICE_WA(xe, 14026539277))
471
+
return;
472
+
473
+
if (!xe_gt_is_main_type(gt))
474
+
return;
475
+
476
+
val = xe_gt_mcr_unicast_read_any(gt, L2COMPUTESIDECTRL);
477
+
val &= ~CECTRL;
478
+
val |= CECTRL_CENODATA_ALWAYS;
479
+
xe_gt_mcr_multicast_write(gt, L2COMPUTESIDECTRL, val);
480
+
}
481
+
455
482
int xe_gt_init_early(struct xe_gt *gt)
456
483
{
457
484
int err;
···
605
574
* on pre-MTL platforms, reading it there will (correctly) return 0.
606
575
*/
607
576
gt->info.gmdid = xe_mmio_read32(>->mmio, GMD_ID);
577
+
578
+
/*
579
+
* Wa_14026539277 can't be implemented as a regular GT workaround (i.e.
580
+
* as an entry in gt_was[]) for two reasons: it is actually a device
581
+
* workaround that happens to involve programming a GT register; and it
582
+
* needs to be applied early to avoid getting the hardware in a bad
583
+
* state before we have a chance to do the necessary programming.
584
+
*/
585
+
wa_14026539277(gt);
608
586
609
587
return 0;
610
588
}
···
936
896
if (IS_SRIOV_PF(gt_to_xe(gt)))
937
897
xe_gt_sriov_pf_stop_prepare(gt);
938
898
899
+
xe_guc_rc_disable(>->uc.guc);
939
900
xe_uc_stop_prepare(>->uc);
940
901
xe_pagefault_reset(gt_to_xe(gt), gt);
941
902
+1
-4
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c
+1
-4
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c
···
1579
1579
goto fail;
1580
1580
1581
1581
offset = 0;
1582
-
for_each_gt(gt, xe, gtid) {
1583
-
if (xe_gt_is_media_type(gt))
1584
-
continue;
1585
-
1582
+
for_each_gt_with_type(gt, xe, gtid, BIT(XE_GT_TYPE_MAIN)) {
1586
1583
config = pf_pick_vf_config(gt, vfid);
1587
1584
bo = config->lmem_obj;
1588
1585
if (!bo)
-1
drivers/gpu/drm/xe/xe_guc.c
-1
drivers/gpu/drm/xe/xe_guc.c
+11
drivers/gpu/drm/xe/xe_guc_ads.c
+11
drivers/gpu/drm/xe/xe_guc_ads.c
···
819
819
{
820
820
u32 um_queue_offset = guc_ads_um_queues_offset(ads);
821
821
struct xe_guc *guc = ads_to_guc(ads);
822
+
struct xe_device *xe = ads_to_xe(ads);
822
823
u64 base_dpa;
823
824
u32 base_ggtt;
824
825
bool with_dpa;
···
831
830
base_dpa = xe_bo_main_addr(ads->bo, PAGE_SIZE) + um_queue_offset;
832
831
833
832
for (i = 0; i < GUC_UM_HW_QUEUE_MAX; ++i) {
833
+
/*
834
+
* Some platforms support USM but not access counters.
835
+
* Skip ACCESS_COUNTER queue initialization for such
836
+
* platforms, leaving queue_params[2] zero-initialized
837
+
* to signal unavailability to the GuC.
838
+
*/
839
+
if (i == GUC_UM_HW_QUEUE_ACCESS_COUNTER &&
840
+
!xe->info.has_access_counter)
841
+
continue;
842
+
834
843
ads_blob_write(ads, um_init_params.queue_params[i].base_dpa,
835
844
with_dpa ? (base_dpa + (i * GUC_UM_QUEUE_SIZE)) : 0);
836
845
ads_blob_write(ads, um_init_params.queue_params[i].base_ggtt_address,
+1
-1
drivers/gpu/drm/xe/xe_guc_ct_types.h
+1
-1
drivers/gpu/drm/xe/xe_guc_ct_types.h
+8
drivers/gpu/drm/xe/xe_guc_pc.c
+8
drivers/gpu/drm/xe/xe_guc_pc.c
···
756
756
if (GRAPHICS_VERx100(gt_to_xe(gt)) >= 1270) {
757
757
reg = xe_mmio_read32(>->mmio, MTL_MIRROR_TARGET_WP1);
758
758
gt_c_state = REG_FIELD_GET(MTL_CC_MASK, reg);
759
+
760
+
/*
761
+
* There are higher level sleep states that will cause this
762
+
* field to read out as its reset state, and those are only
763
+
* possible after the GT is already in C6.
764
+
*/
765
+
if (gt_c_state == MTL_CRST)
766
+
gt_c_state = GT_C6;
759
767
} else {
760
768
reg = xe_mmio_read32(>->mmio, GT_CORE_STATUS);
761
769
gt_c_state = REG_FIELD_GET(RCN_MASK, reg);
+418
-33
drivers/gpu/drm/xe/xe_hw_error.c
+418
-33
drivers/gpu/drm/xe/xe_hw_error.c
···
3
3
* Copyright © 2025 Intel Corporation
4
4
*/
5
5
6
+
#include <linux/bitmap.h>
6
7
#include <linux/fault-inject.h>
7
8
8
9
#include "regs/xe_gsc_regs.h"
···
11
10
#include "regs/xe_irq_regs.h"
12
11
13
12
#include "xe_device.h"
13
+
#include "xe_drm_ras.h"
14
14
#include "xe_hw_error.h"
15
15
#include "xe_mmio.h"
16
16
#include "xe_survivability_mode.h"
17
17
18
-
#define HEC_UNCORR_FW_ERR_BITS 4
18
+
#define GT_HW_ERROR_MAX_ERR_BITS 16
19
+
#define HEC_UNCORR_FW_ERR_BITS 4
20
+
#define XE_RAS_REG_SIZE 32
21
+
#define XE_SOC_NUM_IEH 2
22
+
23
+
#define PVC_ERROR_MASK_SET(hw_err, err_bit) ((hw_err == HARDWARE_ERROR_CORRECTABLE) ? \
24
+
(PVC_COR_ERR_MASK & REG_BIT(err_bit)) : \
25
+
(PVC_FAT_ERR_MASK & REG_BIT(err_bit)))
26
+
19
27
extern struct fault_attr inject_csc_hw_error;
20
28
21
-
/* Error categories reported by hardware */
22
-
enum hardware_error {
23
-
HARDWARE_ERROR_CORRECTABLE = 0,
24
-
HARDWARE_ERROR_NONFATAL = 1,
25
-
HARDWARE_ERROR_FATAL = 2,
26
-
HARDWARE_ERROR_MAX,
27
-
};
29
+
static const char * const error_severity[] = DRM_XE_RAS_ERROR_SEVERITY_NAMES;
28
30
29
31
static const char * const hec_uncorrected_fw_errors[] = {
30
32
"Fatal",
···
36
32
"Data Corruption"
37
33
};
38
34
39
-
static const char *hw_error_to_str(const enum hardware_error hw_err)
35
+
static const unsigned long xe_hw_error_map[] = {
36
+
[XE_GT_ERROR] = DRM_XE_RAS_ERR_COMP_CORE_COMPUTE,
37
+
[XE_SOC_ERROR] = DRM_XE_RAS_ERR_COMP_SOC_INTERNAL,
38
+
};
39
+
40
+
enum gt_vector_regs {
41
+
ERR_STAT_GT_VECTOR0 = 0,
42
+
ERR_STAT_GT_VECTOR1,
43
+
ERR_STAT_GT_VECTOR2,
44
+
ERR_STAT_GT_VECTOR3,
45
+
ERR_STAT_GT_VECTOR4,
46
+
ERR_STAT_GT_VECTOR5,
47
+
ERR_STAT_GT_VECTOR6,
48
+
ERR_STAT_GT_VECTOR7,
49
+
ERR_STAT_GT_VECTOR_MAX
50
+
};
51
+
52
+
#define PVC_GT_VECTOR_LEN(hw_err) ((hw_err == HARDWARE_ERROR_CORRECTABLE) ? \
53
+
ERR_STAT_GT_VECTOR4 : ERR_STAT_GT_VECTOR_MAX)
54
+
55
+
static enum drm_xe_ras_error_severity hw_err_to_severity(const enum hardware_error hw_err)
40
56
{
41
-
switch (hw_err) {
42
-
case HARDWARE_ERROR_CORRECTABLE:
43
-
return "CORRECTABLE";
44
-
case HARDWARE_ERROR_NONFATAL:
45
-
return "NONFATAL";
46
-
case HARDWARE_ERROR_FATAL:
47
-
return "FATAL";
48
-
default:
49
-
return "UNKNOWN";
50
-
}
57
+
if (hw_err == HARDWARE_ERROR_CORRECTABLE)
58
+
return DRM_XE_RAS_ERR_SEV_CORRECTABLE;
59
+
60
+
/* Uncorrectable errors comprise of both fatal and non-fatal errors */
61
+
return DRM_XE_RAS_ERR_SEV_UNCORRECTABLE;
51
62
}
63
+
64
+
static const char * const pvc_master_global_err_reg[] = {
65
+
[0 ... 1] = "Undefined",
66
+
[2] = "HBM SS0: Channel0",
67
+
[3] = "HBM SS0: Channel1",
68
+
[4] = "HBM SS0: Channel2",
69
+
[5] = "HBM SS0: Channel3",
70
+
[6] = "HBM SS0: Channel4",
71
+
[7] = "HBM SS0: Channel5",
72
+
[8] = "HBM SS0: Channel6",
73
+
[9] = "HBM SS0: Channel7",
74
+
[10] = "HBM SS1: Channel0",
75
+
[11] = "HBM SS1: Channel1",
76
+
[12] = "HBM SS1: Channel2",
77
+
[13] = "HBM SS1: Channel3",
78
+
[14] = "HBM SS1: Channel4",
79
+
[15] = "HBM SS1: Channel5",
80
+
[16] = "HBM SS1: Channel6",
81
+
[17] = "HBM SS1: Channel7",
82
+
[18 ... 31] = "Undefined",
83
+
};
84
+
static_assert(ARRAY_SIZE(pvc_master_global_err_reg) == XE_RAS_REG_SIZE);
85
+
86
+
static const char * const pvc_slave_global_err_reg[] = {
87
+
[0] = "Undefined",
88
+
[1] = "HBM SS2: Channel0",
89
+
[2] = "HBM SS2: Channel1",
90
+
[3] = "HBM SS2: Channel2",
91
+
[4] = "HBM SS2: Channel3",
92
+
[5] = "HBM SS2: Channel4",
93
+
[6] = "HBM SS2: Channel5",
94
+
[7] = "HBM SS2: Channel6",
95
+
[8] = "HBM SS2: Channel7",
96
+
[9] = "HBM SS3: Channel0",
97
+
[10] = "HBM SS3: Channel1",
98
+
[11] = "HBM SS3: Channel2",
99
+
[12] = "HBM SS3: Channel3",
100
+
[13] = "HBM SS3: Channel4",
101
+
[14] = "HBM SS3: Channel5",
102
+
[15] = "HBM SS3: Channel6",
103
+
[16] = "HBM SS3: Channel7",
104
+
[17] = "Undefined",
105
+
[18] = "ANR MDFI",
106
+
[19 ... 31] = "Undefined",
107
+
};
108
+
static_assert(ARRAY_SIZE(pvc_slave_global_err_reg) == XE_RAS_REG_SIZE);
109
+
110
+
static const char * const pvc_slave_local_fatal_err_reg[] = {
111
+
[0] = "Local IEH: Malformed PCIe AER",
112
+
[1] = "Local IEH: Malformed PCIe ERR",
113
+
[2] = "Local IEH: UR conditions in IEH",
114
+
[3] = "Local IEH: From SERR Sources",
115
+
[4 ... 19] = "Undefined",
116
+
[20] = "Malformed MCA error packet (HBM/Punit)",
117
+
[21 ... 31] = "Undefined",
118
+
};
119
+
static_assert(ARRAY_SIZE(pvc_slave_local_fatal_err_reg) == XE_RAS_REG_SIZE);
120
+
121
+
static const char * const pvc_master_local_fatal_err_reg[] = {
122
+
[0] = "Local IEH: Malformed IOSF PCIe AER",
123
+
[1] = "Local IEH: Malformed IOSF PCIe ERR",
124
+
[2] = "Local IEH: UR RESPONSE",
125
+
[3] = "Local IEH: From SERR SPI controller",
126
+
[4] = "Base Die MDFI T2T",
127
+
[5] = "Undefined",
128
+
[6] = "Base Die MDFI T2C",
129
+
[7] = "Undefined",
130
+
[8] = "Invalid CSC PSF Command Parity",
131
+
[9] = "Invalid CSC PSF Unexpected Completion",
132
+
[10] = "Invalid CSC PSF Unsupported Request",
133
+
[11] = "Invalid PCIe PSF Command Parity",
134
+
[12] = "PCIe PSF Unexpected Completion",
135
+
[13] = "PCIe PSF Unsupported Request",
136
+
[14 ... 19] = "Undefined",
137
+
[20] = "Malformed MCA error packet (HBM/Punit)",
138
+
[21 ... 31] = "Undefined",
139
+
};
140
+
static_assert(ARRAY_SIZE(pvc_master_local_fatal_err_reg) == XE_RAS_REG_SIZE);
141
+
142
+
static const char * const pvc_master_local_nonfatal_err_reg[] = {
143
+
[0 ... 3] = "Undefined",
144
+
[4] = "Base Die MDFI T2T",
145
+
[5] = "Undefined",
146
+
[6] = "Base Die MDFI T2C",
147
+
[7] = "Undefined",
148
+
[8] = "Invalid CSC PSF Command Parity",
149
+
[9] = "Invalid CSC PSF Unexpected Completion",
150
+
[10] = "Invalid PCIe PSF Command Parity",
151
+
[11 ... 31] = "Undefined",
152
+
};
153
+
static_assert(ARRAY_SIZE(pvc_master_local_nonfatal_err_reg) == XE_RAS_REG_SIZE);
154
+
155
+
#define PVC_MASTER_LOCAL_REG_INFO(hw_err) ((hw_err == HARDWARE_ERROR_FATAL) ? \
156
+
pvc_master_local_fatal_err_reg : \
157
+
pvc_master_local_nonfatal_err_reg)
52
158
53
159
static bool fault_inject_csc_hw_error(void)
54
160
{
···
178
64
179
65
static void csc_hw_error_handler(struct xe_tile *tile, const enum hardware_error hw_err)
180
66
{
181
-
const char *hw_err_str = hw_error_to_str(hw_err);
67
+
const enum drm_xe_ras_error_severity severity = hw_err_to_severity(hw_err);
68
+
const char *severity_str = error_severity[severity];
182
69
struct xe_device *xe = tile_to_xe(tile);
183
70
struct xe_mmio *mmio = &tile->mmio;
184
71
u32 base, err_bit, err_src;
···
192
77
lockdep_assert_held(&xe->irq.lock);
193
78
err_src = xe_mmio_read32(mmio, HEC_UNCORR_ERR_STATUS(base));
194
79
if (!err_src) {
195
-
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported HEC_ERR_STATUS_%s blank\n",
196
-
tile->id, hw_err_str);
80
+
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported %s HEC_ERR_STATUS register blank\n",
81
+
tile->id, severity_str);
197
82
return;
198
83
}
199
84
···
201
86
fw_err = xe_mmio_read32(mmio, HEC_UNCORR_FW_ERR_DW0(base));
202
87
for_each_set_bit(err_bit, &fw_err, HEC_UNCORR_FW_ERR_BITS) {
203
88
drm_err_ratelimited(&xe->drm, HW_ERR
204
-
"%s: HEC Uncorrected FW %s error reported, bit[%d] is set\n",
205
-
hw_err_str, hec_uncorrected_fw_errors[err_bit],
89
+
"HEC FW %s %s reported, bit[%d] is set\n",
90
+
hec_uncorrected_fw_errors[err_bit], severity_str,
206
91
err_bit);
207
92
208
93
schedule_work(&tile->csc_hw_error_work);
···
212
97
xe_mmio_write32(mmio, HEC_UNCORR_ERR_STATUS(base), err_src);
213
98
}
214
99
100
+
static void log_hw_error(struct xe_tile *tile, const char *name,
101
+
const enum drm_xe_ras_error_severity severity)
102
+
{
103
+
const char *severity_str = error_severity[severity];
104
+
struct xe_device *xe = tile_to_xe(tile);
105
+
106
+
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
107
+
drm_warn(&xe->drm, "%s %s detected\n", name, severity_str);
108
+
else
109
+
drm_err_ratelimited(&xe->drm, "%s %s detected\n", name, severity_str);
110
+
}
111
+
112
+
static void log_gt_err(struct xe_tile *tile, const char *name, int i, u32 err,
113
+
const enum drm_xe_ras_error_severity severity)
114
+
{
115
+
const char *severity_str = error_severity[severity];
116
+
struct xe_device *xe = tile_to_xe(tile);
117
+
118
+
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
119
+
drm_warn(&xe->drm, "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
120
+
name, severity_str, i, err);
121
+
else
122
+
drm_err_ratelimited(&xe->drm, "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
123
+
name, severity_str, i, err);
124
+
}
125
+
126
+
static void log_soc_error(struct xe_tile *tile, const char * const *reg_info,
127
+
const enum drm_xe_ras_error_severity severity, u32 err_bit, u32 index)
128
+
{
129
+
const char *severity_str = error_severity[severity];
130
+
struct xe_device *xe = tile_to_xe(tile);
131
+
struct xe_drm_ras *ras = &xe->ras;
132
+
struct xe_drm_ras_counter *info = ras->info[severity];
133
+
const char *name;
134
+
135
+
name = reg_info[err_bit];
136
+
137
+
if (strcmp(name, "Undefined")) {
138
+
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
139
+
drm_warn(&xe->drm, "%s SOC %s detected", name, severity_str);
140
+
else
141
+
drm_err_ratelimited(&xe->drm, "%s SOC %s detected", name, severity_str);
142
+
atomic_inc(&info[index].counter);
143
+
}
144
+
}
145
+
146
+
static void gt_hw_error_handler(struct xe_tile *tile, const enum hardware_error hw_err,
147
+
u32 error_id)
148
+
{
149
+
const enum drm_xe_ras_error_severity severity = hw_err_to_severity(hw_err);
150
+
struct xe_device *xe = tile_to_xe(tile);
151
+
struct xe_drm_ras *ras = &xe->ras;
152
+
struct xe_drm_ras_counter *info = ras->info[severity];
153
+
struct xe_mmio *mmio = &tile->mmio;
154
+
unsigned long err_stat = 0;
155
+
int i;
156
+
157
+
if (xe->info.platform != XE_PVC)
158
+
return;
159
+
160
+
if (hw_err == HARDWARE_ERROR_NONFATAL) {
161
+
atomic_inc(&info[error_id].counter);
162
+
log_hw_error(tile, info[error_id].name, severity);
163
+
return;
164
+
}
165
+
166
+
for (i = 0; i < PVC_GT_VECTOR_LEN(hw_err); i++) {
167
+
u32 vector, val;
168
+
169
+
vector = xe_mmio_read32(mmio, ERR_STAT_GT_VECTOR_REG(hw_err, i));
170
+
if (!vector)
171
+
continue;
172
+
173
+
switch (i) {
174
+
case ERR_STAT_GT_VECTOR0:
175
+
case ERR_STAT_GT_VECTOR1: {
176
+
u32 errbit;
177
+
178
+
val = hweight32(vector);
179
+
atomic_add(val, &info[error_id].counter);
180
+
log_gt_err(tile, "Subslice", i, vector, severity);
181
+
182
+
/*
183
+
* Error status register is only populated once per error.
184
+
* Read the register and clear once.
185
+
*/
186
+
if (err_stat)
187
+
break;
188
+
189
+
err_stat = xe_mmio_read32(mmio, ERR_STAT_GT_REG(hw_err));
190
+
for_each_set_bit(errbit, &err_stat, GT_HW_ERROR_MAX_ERR_BITS) {
191
+
if (PVC_ERROR_MASK_SET(hw_err, errbit))
192
+
atomic_inc(&info[error_id].counter);
193
+
}
194
+
if (err_stat)
195
+
xe_mmio_write32(mmio, ERR_STAT_GT_REG(hw_err), err_stat);
196
+
break;
197
+
}
198
+
case ERR_STAT_GT_VECTOR2:
199
+
case ERR_STAT_GT_VECTOR3:
200
+
val = hweight32(vector);
201
+
atomic_add(val, &info[error_id].counter);
202
+
log_gt_err(tile, "L3 BANK", i, vector, severity);
203
+
break;
204
+
case ERR_STAT_GT_VECTOR6:
205
+
val = hweight32(vector);
206
+
atomic_add(val, &info[error_id].counter);
207
+
log_gt_err(tile, "TLB", i, vector, severity);
208
+
break;
209
+
case ERR_STAT_GT_VECTOR7:
210
+
val = hweight32(vector);
211
+
atomic_add(val, &info[error_id].counter);
212
+
log_gt_err(tile, "L3 Fabric", i, vector, severity);
213
+
break;
214
+
default:
215
+
log_gt_err(tile, "Undefined", i, vector, severity);
216
+
}
217
+
218
+
xe_mmio_write32(mmio, ERR_STAT_GT_VECTOR_REG(hw_err, i), vector);
219
+
}
220
+
}
221
+
222
+
static void soc_slave_ieh_handler(struct xe_tile *tile, const enum hardware_error hw_err, u32 error_id)
223
+
{
224
+
const enum drm_xe_ras_error_severity severity = hw_err_to_severity(hw_err);
225
+
unsigned long slave_global_errstat, slave_local_errstat;
226
+
struct xe_mmio *mmio = &tile->mmio;
227
+
u32 regbit, slave;
228
+
229
+
slave = SOC_PVC_SLAVE_BASE;
230
+
slave_global_errstat = xe_mmio_read32(mmio, SOC_GLOBAL_ERR_STAT_REG(slave, hw_err));
231
+
232
+
if (slave_global_errstat & SOC_IEH1_LOCAL_ERR_STATUS) {
233
+
slave_local_errstat = xe_mmio_read32(mmio, SOC_LOCAL_ERR_STAT_REG(slave, hw_err));
234
+
235
+
if (hw_err == HARDWARE_ERROR_FATAL) {
236
+
for_each_set_bit(regbit, &slave_local_errstat, XE_RAS_REG_SIZE)
237
+
log_soc_error(tile, pvc_slave_local_fatal_err_reg, severity,
238
+
regbit, error_id);
239
+
}
240
+
241
+
xe_mmio_write32(mmio, SOC_LOCAL_ERR_STAT_REG(slave, hw_err),
242
+
slave_local_errstat);
243
+
}
244
+
245
+
for_each_set_bit(regbit, &slave_global_errstat, XE_RAS_REG_SIZE)
246
+
log_soc_error(tile, pvc_slave_global_err_reg, severity, regbit, error_id);
247
+
248
+
xe_mmio_write32(mmio, SOC_GLOBAL_ERR_STAT_REG(slave, hw_err), slave_global_errstat);
249
+
}
250
+
251
+
static void soc_hw_error_handler(struct xe_tile *tile, const enum hardware_error hw_err,
252
+
u32 error_id)
253
+
{
254
+
const enum drm_xe_ras_error_severity severity = hw_err_to_severity(hw_err);
255
+
struct xe_device *xe = tile_to_xe(tile);
256
+
struct xe_mmio *mmio = &tile->mmio;
257
+
unsigned long master_global_errstat, master_local_errstat;
258
+
u32 master, slave, regbit;
259
+
int i;
260
+
261
+
if (xe->info.platform != XE_PVC)
262
+
return;
263
+
264
+
master = SOC_PVC_MASTER_BASE;
265
+
slave = SOC_PVC_SLAVE_BASE;
266
+
267
+
/* Mask error type in GSYSEVTCTL so that no new errors of the type will be reported */
268
+
for (i = 0; i < XE_SOC_NUM_IEH; i++)
269
+
xe_mmio_write32(mmio, SOC_GSYSEVTCTL_REG(master, slave, i), ~REG_BIT(hw_err));
270
+
271
+
if (hw_err == HARDWARE_ERROR_CORRECTABLE) {
272
+
xe_mmio_write32(mmio, SOC_GLOBAL_ERR_STAT_REG(master, hw_err), REG_GENMASK(31, 0));
273
+
xe_mmio_write32(mmio, SOC_LOCAL_ERR_STAT_REG(master, hw_err), REG_GENMASK(31, 0));
274
+
xe_mmio_write32(mmio, SOC_GLOBAL_ERR_STAT_REG(slave, hw_err), REG_GENMASK(31, 0));
275
+
xe_mmio_write32(mmio, SOC_LOCAL_ERR_STAT_REG(slave, hw_err), REG_GENMASK(31, 0));
276
+
goto unmask_gsysevtctl;
277
+
}
278
+
279
+
/*
280
+
* Read the master global IEH error register, if BIT(1) is set then process
281
+
* the slave IEH first. If BIT(0) in global error register is set then process
282
+
* the corresponding local error registers.
283
+
*/
284
+
master_global_errstat = xe_mmio_read32(mmio, SOC_GLOBAL_ERR_STAT_REG(master, hw_err));
285
+
if (master_global_errstat & SOC_SLAVE_IEH)
286
+
soc_slave_ieh_handler(tile, hw_err, error_id);
287
+
288
+
if (master_global_errstat & SOC_IEH0_LOCAL_ERR_STATUS) {
289
+
master_local_errstat = xe_mmio_read32(mmio, SOC_LOCAL_ERR_STAT_REG(master, hw_err));
290
+
291
+
for_each_set_bit(regbit, &master_local_errstat, XE_RAS_REG_SIZE)
292
+
log_soc_error(tile, PVC_MASTER_LOCAL_REG_INFO(hw_err), severity, regbit, error_id);
293
+
294
+
xe_mmio_write32(mmio, SOC_LOCAL_ERR_STAT_REG(master, hw_err), master_local_errstat);
295
+
}
296
+
297
+
for_each_set_bit(regbit, &master_global_errstat, XE_RAS_REG_SIZE)
298
+
log_soc_error(tile, pvc_master_global_err_reg, severity, regbit, error_id);
299
+
300
+
xe_mmio_write32(mmio, SOC_GLOBAL_ERR_STAT_REG(master, hw_err), master_global_errstat);
301
+
302
+
unmask_gsysevtctl:
303
+
for (i = 0; i < XE_SOC_NUM_IEH; i++)
304
+
xe_mmio_write32(mmio, SOC_GSYSEVTCTL_REG(master, slave, i),
305
+
(HARDWARE_ERROR_MAX << 1) + 1);
306
+
}
307
+
215
308
static void hw_error_source_handler(struct xe_tile *tile, const enum hardware_error hw_err)
216
309
{
217
-
const char *hw_err_str = hw_error_to_str(hw_err);
310
+
const enum drm_xe_ras_error_severity severity = hw_err_to_severity(hw_err);
311
+
const char *severity_str = error_severity[severity];
218
312
struct xe_device *xe = tile_to_xe(tile);
219
-
unsigned long flags;
220
-
u32 err_src;
313
+
struct xe_drm_ras *ras = &xe->ras;
314
+
struct xe_drm_ras_counter *info = ras->info[severity];
315
+
unsigned long flags, err_src;
316
+
u32 err_bit;
221
317
222
-
if (xe->info.platform != XE_BATTLEMAGE)
318
+
if (!IS_DGFX(xe))
223
319
return;
224
320
225
321
spin_lock_irqsave(&xe->irq.lock, flags);
226
322
err_src = xe_mmio_read32(&tile->mmio, DEV_ERR_STAT_REG(hw_err));
227
323
if (!err_src) {
228
-
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported DEV_ERR_STAT_%s blank!\n",
229
-
tile->id, hw_err_str);
324
+
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported %s DEV_ERR_STAT register blank!\n",
325
+
tile->id, severity_str);
230
326
goto unlock;
231
327
}
232
328
233
-
if (err_src & XE_CSC_ERROR)
329
+
/*
330
+
* On encountering CSC firmware errors, the graphics device becomes unrecoverable
331
+
* so return immediately on error. The only way to recover from these errors is
332
+
* firmware flash. The device will enter Runtime Survivability mode when such
333
+
* errors are detected.
334
+
*/
335
+
if (err_src & REG_BIT(XE_CSC_ERROR)) {
234
336
csc_hw_error_handler(tile, hw_err);
337
+
goto clear_reg;
338
+
}
235
339
340
+
if (!info)
341
+
goto clear_reg;
342
+
343
+
for_each_set_bit(err_bit, &err_src, XE_RAS_REG_SIZE) {
344
+
const char *name;
345
+
u32 error_id;
346
+
347
+
/* Check error bit is within bounds */
348
+
if (err_bit >= ARRAY_SIZE(xe_hw_error_map))
349
+
break;
350
+
351
+
error_id = xe_hw_error_map[err_bit];
352
+
353
+
/* Check error component is within max */
354
+
if (!error_id || error_id >= DRM_XE_RAS_ERR_COMP_MAX)
355
+
continue;
356
+
357
+
name = info[error_id].name;
358
+
if (!name)
359
+
continue;
360
+
361
+
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE) {
362
+
drm_warn(&xe->drm, HW_ERR
363
+
"TILE%d reported %s %s, bit[%d] is set\n",
364
+
tile->id, name, severity_str, err_bit);
365
+
} else {
366
+
drm_err_ratelimited(&xe->drm, HW_ERR
367
+
"TILE%d reported %s %s, bit[%d] is set\n",
368
+
tile->id, name, severity_str, err_bit);
369
+
}
370
+
371
+
if (err_bit == XE_GT_ERROR)
372
+
gt_hw_error_handler(tile, hw_err, error_id);
373
+
if (err_bit == XE_SOC_ERROR)
374
+
soc_hw_error_handler(tile, hw_err, error_id);
375
+
}
376
+
377
+
clear_reg:
236
378
xe_mmio_write32(&tile->mmio, DEV_ERR_STAT_REG(hw_err), err_src);
237
-
238
379
unlock:
239
380
spin_unlock_irqrestore(&xe->irq.lock, flags);
240
381
}
···
512
141
if (fault_inject_csc_hw_error())
513
142
schedule_work(&tile->csc_hw_error_work);
514
143
515
-
for (hw_err = 0; hw_err < HARDWARE_ERROR_MAX; hw_err++)
144
+
for (hw_err = 0; hw_err < HARDWARE_ERROR_MAX; hw_err++) {
516
145
if (master_ctl & ERROR_IRQ(hw_err))
517
146
hw_error_source_handler(tile, hw_err);
147
+
}
148
+
}
149
+
150
+
static int hw_error_info_init(struct xe_device *xe)
151
+
{
152
+
if (xe->info.platform != XE_PVC)
153
+
return 0;
154
+
155
+
return xe_drm_ras_init(xe);
518
156
}
519
157
520
158
/*
···
552
172
void xe_hw_error_init(struct xe_device *xe)
553
173
{
554
174
struct xe_tile *tile = xe_device_get_root_tile(xe);
175
+
int ret;
555
176
556
177
if (!IS_DGFX(xe) || IS_SRIOV_VF(xe))
557
178
return;
558
179
559
180
INIT_WORK(&tile->csc_hw_error_work, csc_hw_error_work);
181
+
182
+
ret = hw_error_info_init(xe);
183
+
if (ret)
184
+
drm_err(&xe->drm, "Failed to initialize XE DRM RAS (%pe)\n", ERR_PTR(ret));
560
185
561
186
process_hw_errors(xe);
562
187
}
+9
drivers/gpu/drm/xe/xe_lrc.c
+9
drivers/gpu/drm/xe/xe_lrc.c
···
14
14
#include "instructions/xe_gfxpipe_commands.h"
15
15
#include "instructions/xe_gfx_state_commands.h"
16
16
#include "regs/xe_engine_regs.h"
17
+
#include "regs/xe_gt_regs.h"
17
18
#include "regs/xe_lrc_layout.h"
18
19
#include "xe_bb.h"
19
20
#include "xe_bo.h"
···
1447
1446
struct xe_device *xe = gt_to_xe(gt);
1448
1447
struct iosys_map map;
1449
1448
u32 arb_enable;
1449
+
u32 state_cache_perf_fix[3];
1450
1450
int err;
1451
1451
1452
1452
/*
···
1547
1545
1548
1546
arb_enable = MI_ARB_ON_OFF | MI_ARB_ENABLE;
1549
1547
xe_lrc_write_ring(lrc, &arb_enable, sizeof(arb_enable));
1548
+
1549
+
if (init_flags & XE_LRC_DISABLE_STATE_CACHE_PERF_FIX) {
1550
+
state_cache_perf_fix[0] = MI_LOAD_REGISTER_IMM | MI_LRI_NUM_REGS(1);
1551
+
state_cache_perf_fix[1] = COMMON_SLICE_CHICKEN3.addr;
1552
+
state_cache_perf_fix[2] = _MASKED_BIT_ENABLE(DISABLE_STATE_CACHE_PERF_FIX);
1553
+
xe_lrc_write_ring(lrc, state_cache_perf_fix, sizeof(state_cache_perf_fix));
1554
+
}
1550
1555
1551
1556
map = __xe_lrc_seqno_map(lrc);
1552
1557
xe_map_write32(lrc_to_xe(lrc), &map, lrc->fence_ctx.next_seqno - 1);
+1
drivers/gpu/drm/xe/xe_lrc.h
+1
drivers/gpu/drm/xe/xe_lrc.h
···
49
49
#define XE_LRC_CREATE_RUNALONE BIT(0)
50
50
#define XE_LRC_CREATE_PXP BIT(1)
51
51
#define XE_LRC_CREATE_USER_CTX BIT(2)
52
+
#define XE_LRC_DISABLE_STATE_CACHE_PERF_FIX BIT(3)
52
53
53
54
struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
54
55
void *replay_state, u32 ring_size, u16 msix_vec, u32 flags);
+101
-42
drivers/gpu/drm/xe/xe_migrate.c
+101
-42
drivers/gpu/drm/xe/xe_migrate.c
···
184
184
xe_assert(xe, pos == vram_limit);
185
185
}
186
186
187
-
static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
188
-
struct xe_vm *vm, struct drm_exec *exec)
187
+
static int xe_migrate_pt_bo_alloc(struct xe_tile *tile, struct xe_migrate *m,
188
+
struct xe_vm *vm, struct drm_exec *exec)
189
189
{
190
-
struct xe_device *xe = tile_to_xe(tile);
191
-
u16 pat_index = xe->pat.idx[XE_CACHE_WB];
192
-
u8 id = tile->id;
193
-
u32 num_entries = NUM_PT_SLOTS, num_level = vm->pt_root[id]->level;
194
-
#define VRAM_IDENTITY_MAP_COUNT 2
195
-
u32 num_setup = num_level + VRAM_IDENTITY_MAP_COUNT;
196
-
#undef VRAM_IDENTITY_MAP_COUNT
197
-
u32 map_ofs, level, i;
198
190
struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo;
199
-
u64 entry, pt29_ofs;
191
+
u32 num_entries = NUM_PT_SLOTS;
200
192
201
193
/* Can't bump NUM_PT_SLOTS too high */
202
194
BUILD_BUG_ON(NUM_PT_SLOTS > SZ_2M/XE_PAGE_SIZE);
···
207
215
XE_BO_FLAG_PAGETABLE, exec);
208
216
if (IS_ERR(bo))
209
217
return PTR_ERR(bo);
218
+
219
+
m->pt_bo = bo;
220
+
return 0;
221
+
}
222
+
223
+
static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
224
+
struct xe_vm *vm, u32 *ofs)
225
+
{
226
+
struct xe_device *xe = tile_to_xe(tile);
227
+
u16 pat_index = xe->pat.idx[XE_CACHE_WB];
228
+
u8 id = tile->id;
229
+
u32 num_entries = NUM_PT_SLOTS, num_level = vm->pt_root[id]->level;
230
+
#define VRAM_IDENTITY_MAP_COUNT 2
231
+
u32 num_setup = num_level + VRAM_IDENTITY_MAP_COUNT;
232
+
#undef VRAM_IDENTITY_MAP_COUNT
233
+
u32 map_ofs, level, i;
234
+
struct xe_bo *bo = m->pt_bo, *batch = tile->mem.kernel_bb_pool->bo;
235
+
u64 entry, pt29_ofs;
210
236
211
237
/* PT30 & PT31 reserved for 2M identity map */
212
238
pt29_ofs = xe_bo_size(bo) - 3 * XE_PAGE_SIZE;
···
348
338
}
349
339
}
350
340
341
+
if (ofs)
342
+
*ofs = map_ofs;
343
+
}
344
+
345
+
static void xe_migrate_suballoc_manager_init(struct xe_migrate *m, u32 map_ofs)
346
+
{
351
347
/*
352
348
* Example layout created above, with root level = 3:
353
349
* [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's
···
379
363
drm_suballoc_manager_init(&m->vm_update_sa,
380
364
(size_t)(map_ofs / XE_PAGE_SIZE - NUM_KERNEL_PDE) *
381
365
NUM_VMUSA_UNIT_PER_PAGE, 0);
382
-
383
-
m->pt_bo = bo;
384
-
return 0;
385
366
}
386
367
387
368
/*
···
429
416
struct xe_device *xe = tile_to_xe(tile);
430
417
struct xe_validation_ctx ctx;
431
418
struct drm_exec exec;
419
+
u32 map_ofs;
432
420
int err = 0;
433
421
434
422
xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) {
435
423
err = xe_vm_drm_exec_lock(vm, &exec);
424
+
if (err)
425
+
return err;
426
+
436
427
drm_exec_retry_on_contention(&exec);
437
-
err = xe_migrate_prepare_vm(tile, m, vm, &exec);
428
+
429
+
err = xe_migrate_pt_bo_alloc(tile, m, vm, &exec);
430
+
if (err)
431
+
return err;
432
+
433
+
xe_migrate_prepare_vm(tile, m, vm, &map_ofs);
434
+
xe_migrate_suballoc_manager_init(m, map_ofs);
438
435
drm_exec_retry_on_contention(&exec);
439
436
xe_validation_retry_on_oom(&ctx, &err);
440
437
}
···
860
837
return flush_flags;
861
838
}
862
839
863
-
/**
864
-
* xe_migrate_copy() - Copy content of TTM resources.
865
-
* @m: The migration context.
866
-
* @src_bo: The buffer object @src is currently bound to.
867
-
* @dst_bo: If copying between resources created for the same bo, set this to
868
-
* the same value as @src_bo. If copying between buffer objects, set it to
869
-
* the buffer object @dst is currently bound to.
870
-
* @src: The source TTM resource.
871
-
* @dst: The dst TTM resource.
872
-
* @copy_only_ccs: If true copy only CCS metadata
873
-
*
874
-
* Copies the contents of @src to @dst: On flat CCS devices,
875
-
* the CCS metadata is copied as well if needed, or if not present,
876
-
* the CCS metadata of @dst is cleared for security reasons.
877
-
*
878
-
* Return: Pointer to a dma_fence representing the last copy batch, or
879
-
* an error pointer on failure. If there is a failure, any copy operation
880
-
* started by the function call has been synced.
881
-
*/
882
-
struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
883
-
struct xe_bo *src_bo,
884
-
struct xe_bo *dst_bo,
885
-
struct ttm_resource *src,
886
-
struct ttm_resource *dst,
887
-
bool copy_only_ccs)
840
+
static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
841
+
struct xe_bo *src_bo,
842
+
struct xe_bo *dst_bo,
843
+
struct ttm_resource *src,
844
+
struct ttm_resource *dst,
845
+
bool copy_only_ccs,
846
+
bool is_vram_resolve)
888
847
{
889
848
struct xe_gt *gt = m->tile->primary_gt;
890
849
struct xe_device *xe = gt_to_xe(gt);
···
887
882
bool copy_ccs = xe_device_has_flat_ccs(xe) &&
888
883
xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
889
884
bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
890
-
bool use_comp_pat = type_device && xe_device_has_flat_ccs(xe) &&
891
-
GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram;
885
+
886
+
/*
887
+
* For decompression operation, always use the compression PAT index.
888
+
* Otherwise, only use the compression PAT index for device memory
889
+
* when copying from VRAM to system memory.
890
+
*/
891
+
bool use_comp_pat = is_vram_resolve || (type_device &&
892
+
xe_device_has_flat_ccs(xe) &&
893
+
GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
892
894
893
895
/* Copying CCS between two different BOs is not supported yet. */
894
896
if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
···
1052
1040
}
1053
1041
1054
1042
return fence;
1043
+
}
1044
+
1045
+
/**
1046
+
* xe_migrate_copy() - Copy content of TTM resources.
1047
+
* @m: The migration context.
1048
+
* @src_bo: The buffer object @src is currently bound to.
1049
+
* @dst_bo: If copying between resources created for the same bo, set this to
1050
+
* the same value as @src_bo. If copying between buffer objects, set it to
1051
+
* the buffer object @dst is currently bound to.
1052
+
* @src: The source TTM resource.
1053
+
* @dst: The dst TTM resource.
1054
+
* @copy_only_ccs: If true copy only CCS metadata
1055
+
*
1056
+
* Copies the contents of @src to @dst: On flat CCS devices,
1057
+
* the CCS metadata is copied as well if needed, or if not present,
1058
+
* the CCS metadata of @dst is cleared for security reasons.
1059
+
*
1060
+
* Return: Pointer to a dma_fence representing the last copy batch, or
1061
+
* an error pointer on failure. If there is a failure, any copy operation
1062
+
* started by the function call has been synced.
1063
+
*/
1064
+
struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
1065
+
struct xe_bo *src_bo,
1066
+
struct xe_bo *dst_bo,
1067
+
struct ttm_resource *src,
1068
+
struct ttm_resource *dst,
1069
+
bool copy_only_ccs)
1070
+
{
1071
+
return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
1072
+
}
1073
+
1074
+
/**
1075
+
* xe_migrate_resolve() - Resolve and decompress a buffer object if required.
1076
+
* @m: The migrate context
1077
+
* @bo: The buffer object to resolve
1078
+
* @res: The reservation object
1079
+
*
1080
+
* Wrapper around __xe_migrate_copy() with is_vram_resolve set to true
1081
+
* to trigger decompression if needed.
1082
+
*
1083
+
* Return: A dma_fence that signals on completion, or an ERR_PTR on failure.
1084
+
*/
1085
+
struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
1086
+
struct xe_bo *bo,
1087
+
struct ttm_resource *res)
1088
+
{
1089
+
return __xe_migrate_copy(m, bo, bo, res, res, false, true);
1055
1090
}
1056
1091
1057
1092
/**
+4
drivers/gpu/drm/xe/xe_migrate.h
+4
drivers/gpu/drm/xe/xe_migrate.h
···
132
132
struct ttm_resource *dst,
133
133
bool copy_only_ccs);
134
134
135
+
struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
136
+
struct xe_bo *bo,
137
+
struct ttm_resource *res);
138
+
135
139
int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
136
140
struct xe_bo *src_bo,
137
141
enum xe_sriov_vf_ccs_rw_ctxs read_write);
+23
-10
drivers/gpu/drm/xe/xe_pat.c
+23
-10
drivers/gpu/drm/xe/xe_pat.c
···
311
311
return REG_FIELD_GET(XE2_L3_POLICY, xe->pat.table[pat_index].value);
312
312
}
313
313
314
+
static const struct xe_pat_table_entry *gt_pta_entry(struct xe_gt *gt)
315
+
{
316
+
struct xe_device *xe = gt_to_xe(gt);
317
+
318
+
if (xe_gt_is_main_type(gt))
319
+
return xe->pat.pat_primary_pta;
320
+
321
+
if (xe_gt_is_media_type(gt))
322
+
return xe->pat.pat_media_pta;
323
+
324
+
xe_assert(xe, false);
325
+
return NULL;
326
+
}
327
+
314
328
static void program_pat(struct xe_gt *gt, const struct xe_pat_table_entry table[],
315
329
int n_entries)
316
330
{
317
331
struct xe_device *xe = gt_to_xe(gt);
332
+
const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt);
318
333
319
334
for (int i = 0; i < n_entries; i++) {
320
335
struct xe_reg reg = XE_REG(_PAT_INDEX(i));
···
339
324
340
325
if (xe->pat.pat_ats)
341
326
xe_mmio_write32(>->mmio, XE_REG(_PAT_ATS), xe->pat.pat_ats->value);
342
-
if (xe->pat.pat_primary_pta && xe_gt_is_main_type(gt))
343
-
xe_mmio_write32(>->mmio, XE_REG(_PAT_PTA), xe->pat.pat_primary_pta->value);
344
-
if (xe->pat.pat_media_pta && xe_gt_is_media_type(gt))
345
-
xe_mmio_write32(>->mmio, XE_REG(_PAT_PTA), xe->pat.pat_media_pta->value);
327
+
328
+
if (pta_entry)
329
+
xe_mmio_write32(>->mmio, XE_REG(_PAT_PTA), pta_entry->value);
346
330
}
347
331
348
332
static void program_pat_mcr(struct xe_gt *gt, const struct xe_pat_table_entry table[],
349
333
int n_entries)
350
334
{
351
335
struct xe_device *xe = gt_to_xe(gt);
336
+
const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt);
352
337
353
338
for (int i = 0; i < n_entries; i++) {
354
339
struct xe_reg_mcr reg_mcr = XE_REG_MCR(_PAT_INDEX(i));
···
358
343
359
344
if (xe->pat.pat_ats)
360
345
xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_ATS), xe->pat.pat_ats->value);
361
-
if (xe->pat.pat_primary_pta && xe_gt_is_main_type(gt))
362
-
xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_PTA), xe->pat.pat_primary_pta->value);
363
-
if (xe->pat.pat_media_pta && xe_gt_is_media_type(gt))
364
-
xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_PTA), xe->pat.pat_media_pta->value);
346
+
347
+
if (pta_entry)
348
+
xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_PTA), pta_entry->value);
365
349
}
366
350
367
351
static int xelp_dump(struct xe_gt *gt, struct drm_printer *p)
···
691
677
int xe_pat_dump_sw_config(struct xe_gt *gt, struct drm_printer *p)
692
678
{
693
679
struct xe_device *xe = gt_to_xe(gt);
694
-
const struct xe_pat_table_entry *pta_entry = xe_gt_is_main_type(gt) ?
695
-
xe->pat.pat_primary_pta : xe->pat.pat_media_pta;
680
+
const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt);
696
681
char label[PAT_LABEL_LEN];
697
682
698
683
if (!xe->pat.table || !xe->pat.n_entries)
+8
-4
drivers/gpu/drm/xe/xe_pci.c
+8
-4
drivers/gpu/drm/xe/xe_pci.c
···
81
81
82
82
XE_HP_FEATURES,
83
83
84
+
.has_access_counter = 1,
84
85
.has_asid = 1,
85
86
.has_atomic_enable_pte_bit = 1,
86
87
.has_usm = 1,
···
99
98
};
100
99
101
100
#define XE2_GFX_FEATURES \
101
+
.has_access_counter = 1, \
102
102
.has_asid = 1, \
103
103
.has_atomic_enable_pte_bit = 1, \
104
104
.has_range_tlb_inval = 1, \
···
125
123
126
124
static const struct xe_graphics_desc graphics_xe3p_xpc = {
127
125
XE2_GFX_FEATURES,
126
+
.has_access_counter = 0,
128
127
.has_indirect_ring_state = 1,
129
128
.hw_engine_mask =
130
129
GENMASK(XE_HW_ENGINE_BCS8, XE_HW_ENGINE_BCS1) |
···
779
776
xe->info.max_gt_per_tile = desc->max_gt_per_tile;
780
777
xe->info.tile_count = 1 + desc->max_remote_tiles;
781
778
779
+
xe_step_platform_get(xe);
780
+
782
781
err = xe_tile_init_early(xe_device_get_root_tile(xe), xe, 0);
783
782
if (err)
784
783
return err;
···
915
910
if (desc->pre_gmdid_graphics_ip) {
916
911
graphics_ip = desc->pre_gmdid_graphics_ip;
917
912
media_ip = desc->pre_gmdid_media_ip;
918
-
xe->info.step = xe_step_pre_gmdid_get(xe);
913
+
xe_step_pre_gmdid_get(xe);
919
914
} else {
920
915
xe_assert(xe, !desc->pre_gmdid_media_ip);
921
916
ret = handle_gmdid(xe, &graphics_ip, &media_ip,
···
923
918
if (ret)
924
919
return ret;
925
920
926
-
xe->info.step = xe_step_gmdid_get(xe,
927
-
graphics_gmdid_revid,
928
-
media_gmdid_revid);
921
+
xe_step_gmdid_get(xe, graphics_gmdid_revid, media_gmdid_revid);
929
922
}
930
923
931
924
/*
···
947
944
media_desc = NULL;
948
945
}
949
946
947
+
xe->info.has_access_counter = graphics_desc->has_access_counter;
950
948
xe->info.has_asid = graphics_desc->has_asid;
951
949
xe->info.has_atomic_enable_pte_bit = graphics_desc->has_atomic_enable_pte_bit;
952
950
if (xe->info.platform != XE_PVC)
+1
drivers/gpu/drm/xe/xe_pci_types.h
+1
drivers/gpu/drm/xe/xe_pci_types.h
+2
drivers/gpu/drm/xe/xe_query.c
+2
drivers/gpu/drm/xe/xe_query.c
···
340
340
DRM_XE_QUERY_CONFIG_FLAG_HAS_NO_COMPRESSION_HINT;
341
341
config->info[DRM_XE_QUERY_CONFIG_FLAGS] |=
342
342
DRM_XE_QUERY_CONFIG_FLAG_HAS_LOW_LATENCY;
343
+
config->info[DRM_XE_QUERY_CONFIG_FLAGS] |=
344
+
DRM_XE_QUERY_CONFIG_FLAG_HAS_DISABLE_STATE_CACHE_PERF_FIX;
343
345
config->info[DRM_XE_QUERY_CONFIG_MIN_ALIGNMENT] =
344
346
xe->info.vram_flags & XE_VRAM_FLAGS_NEED64K ? SZ_64K : SZ_4K;
345
347
config->info[DRM_XE_QUERY_CONFIG_VA_BITS] = xe->info.va_bits;
+7
drivers/gpu/drm/xe/xe_rtp.c
+7
drivers/gpu/drm/xe/xe_rtp.c
···
55
55
match = xe->info.platform == r->platform &&
56
56
xe->info.subplatform == r->subplatform;
57
57
break;
58
+
case XE_RTP_MATCH_PLATFORM_STEP:
59
+
if (drm_WARN_ON(&xe->drm, xe->info.step.platform == STEP_NONE))
60
+
return false;
61
+
62
+
match = xe->info.step.platform >= r->step_start &&
63
+
xe->info.step.platform < r->step_end;
64
+
break;
58
65
case XE_RTP_MATCH_GRAPHICS_VERSION:
59
66
if (drm_WARN_ON(&xe->drm, !gt))
60
67
return false;
+20
drivers/gpu/drm/xe/xe_rtp.h
+20
drivers/gpu/drm/xe/xe_rtp.h
···
35
35
{ .match_type = XE_RTP_MATCH_SUBPLATFORM, \
36
36
.platform = plat__, .subplatform = sub__ }
37
37
38
+
#define _XE_RTP_RULE_PLATFORM_STEP(start__, end__) \
39
+
{ .match_type = XE_RTP_MATCH_PLATFORM_STEP, \
40
+
.step_start = start__, .step_end = end__ }
41
+
38
42
#define _XE_RTP_RULE_GRAPHICS_STEP(start__, end__) \
39
43
{ .match_type = XE_RTP_MATCH_GRAPHICS_STEP, \
40
44
.step_start = start__, .step_end = end__ }
···
69
65
*/
70
66
#define XE_RTP_RULE_SUBPLATFORM(plat_, sub_) \
71
67
_XE_RTP_RULE_SUBPLATFORM(XE_##plat_, XE_SUBPLATFORM_##plat_##_##sub_)
68
+
69
+
/**
70
+
* XE_RTP_RULE_PLATFORM_STEP - Create rule matching platform-level stepping
71
+
* @start_: First stepping matching the rule
72
+
* @end_: First stepping that does not match the rule
73
+
*
74
+
* Note that the range matching this rule is [ @start_, @end_ ), i.e. inclusive
75
+
* on the left, exclusive on the right.
76
+
*
77
+
* You need to make sure that proper support for reading platform-level stepping
78
+
* information is present for the target platform before using this rule.
79
+
*
80
+
* Refer to XE_RTP_RULES() for expected usage.
81
+
*/
82
+
#define XE_RTP_RULE_PLATFORM_STEP(start_, end_) \
83
+
_XE_RTP_RULE_PLATFORM_STEP(STEP_##start_, STEP_##end_)
72
84
73
85
/**
74
86
* XE_RTP_RULE_GRAPHICS_STEP - Create rule matching graphics stepping
+1
drivers/gpu/drm/xe/xe_rtp_types.h
+1
drivers/gpu/drm/xe/xe_rtp_types.h
+52
-22
drivers/gpu/drm/xe/xe_step.c
+52
-22
drivers/gpu/drm/xe/xe_step.c
···
109
109
__diag_pop();
110
110
111
111
/**
112
+
* xe_step_platform_get - Determine platform-level stepping from PCI revid
113
+
* @xe: Xe device
114
+
*
115
+
* Convert the PCI revid into a platform-level stepping value and store that
116
+
* in the device info.
117
+
*/
118
+
void xe_step_platform_get(struct xe_device *xe)
119
+
{
120
+
/*
121
+
* Not all platforms map PCI revid directly into our symbolic stepping
122
+
* enumeration. Some platforms will have a single PCI revid used for a
123
+
* range platform level steppings and some might even require specific
124
+
* mappings. So prefer to err on the side of caution and include only
125
+
* the platforms from which we need the stepping info for workaround
126
+
* checks.
127
+
*/
128
+
129
+
if (xe->info.platform == XE_NOVALAKE_P)
130
+
xe->info.step.platform = STEP_A0 + xe->info.revid;
131
+
}
132
+
133
+
/**
112
134
* xe_step_pre_gmdid_get - Determine IP steppings from PCI revid
113
135
* @xe: Xe device
114
136
*
115
137
* Convert the PCI revid into proper IP steppings. This should only be
116
138
* used on platforms that do not have GMD_ID support.
117
139
*/
118
-
struct xe_step_info xe_step_pre_gmdid_get(struct xe_device *xe)
140
+
void xe_step_pre_gmdid_get(struct xe_device *xe)
119
141
{
120
142
const struct xe_step_info *revids = NULL;
121
-
struct xe_step_info step = {};
122
143
u16 revid = xe->info.revid;
123
144
int size = 0;
124
145
const int *basedie_info = NULL;
125
146
int basedie_size = 0;
126
147
int baseid = 0;
148
+
u8 graphics = STEP_NONE;
149
+
u8 media = STEP_NONE;
150
+
u8 basedie = STEP_NONE;
127
151
128
152
if (xe->info.platform == XE_PVC) {
129
153
baseid = FIELD_GET(GENMASK(5, 3), xe->info.revid);
···
190
166
191
167
/* Not using the stepping scheme for the platform yet. */
192
168
if (!revids)
193
-
return step;
169
+
goto done;
194
170
195
171
if (revid < size && revids[revid].graphics != STEP_NONE) {
196
-
step = revids[revid];
172
+
graphics = revids[revid].graphics;
173
+
media = revids[revid].media;
174
+
basedie = revids[revid].basedie;
197
175
} else {
198
176
drm_warn(&xe->drm, "Unknown revid 0x%02x\n", revid);
199
177
···
213
187
if (revid < size) {
214
188
drm_dbg(&xe->drm, "Using steppings for revid 0x%02x\n",
215
189
revid);
216
-
step = revids[revid];
190
+
graphics = revids[revid].graphics;
191
+
media = revids[revid].media;
192
+
basedie = revids[revid].basedie;
217
193
} else {
218
194
drm_dbg(&xe->drm, "Using future steppings\n");
219
-
step.graphics = STEP_FUTURE;
195
+
graphics = STEP_FUTURE;
220
196
}
221
197
}
222
198
223
-
drm_WARN_ON(&xe->drm, step.graphics == STEP_NONE);
199
+
drm_WARN_ON(&xe->drm, graphics == STEP_NONE);
224
200
225
201
if (basedie_info && basedie_size) {
226
202
if (baseid < basedie_size && basedie_info[baseid] != STEP_NONE) {
227
-
step.basedie = basedie_info[baseid];
203
+
basedie = basedie_info[baseid];
228
204
} else {
229
205
drm_warn(&xe->drm, "Unknown baseid 0x%02x\n", baseid);
230
-
step.basedie = STEP_FUTURE;
206
+
basedie = STEP_FUTURE;
231
207
}
232
208
}
233
209
234
-
return step;
210
+
done:
211
+
xe->info.step.graphics = graphics;
212
+
xe->info.step.media = media;
213
+
xe->info.step.basedie = basedie;
235
214
}
236
215
237
216
/**
···
251
220
* all platforms: major steppings (A0, B0, etc.) are 4 apart, with minor
252
221
* steppings (A1, A2, etc.) taking the values in between.
253
222
*/
254
-
struct xe_step_info xe_step_gmdid_get(struct xe_device *xe,
255
-
u32 graphics_gmdid_revid,
256
-
u32 media_gmdid_revid)
223
+
void xe_step_gmdid_get(struct xe_device *xe,
224
+
u32 graphics_gmdid_revid,
225
+
u32 media_gmdid_revid)
257
226
{
258
-
struct xe_step_info step = {
259
-
.graphics = STEP_A0 + graphics_gmdid_revid,
260
-
.media = STEP_A0 + media_gmdid_revid,
261
-
};
227
+
u8 graphics = STEP_A0 + graphics_gmdid_revid;
228
+
u8 media = STEP_A0 + media_gmdid_revid;
262
229
263
-
if (step.graphics >= STEP_FUTURE) {
264
-
step.graphics = STEP_FUTURE;
230
+
if (graphics >= STEP_FUTURE) {
231
+
graphics = STEP_FUTURE;
265
232
drm_dbg(&xe->drm, "Graphics GMD_ID revid value %d treated as future stepping\n",
266
233
graphics_gmdid_revid);
267
234
}
268
235
269
-
if (step.media >= STEP_FUTURE) {
270
-
step.media = STEP_FUTURE;
236
+
if (media >= STEP_FUTURE) {
237
+
media = STEP_FUTURE;
271
238
drm_dbg(&xe->drm, "Media GMD_ID revid value %d treated as future stepping\n",
272
239
media_gmdid_revid);
273
240
}
274
241
275
-
return step;
242
+
xe->info.step.graphics = graphics;
243
+
xe->info.step.media = media;
276
244
}
277
245
278
246
#define STEP_NAME_CASE(name) \
+6
-4
drivers/gpu/drm/xe/xe_step.h
+6
-4
drivers/gpu/drm/xe/xe_step.h
···
12
12
13
13
struct xe_device;
14
14
15
-
struct xe_step_info xe_step_pre_gmdid_get(struct xe_device *xe);
16
-
struct xe_step_info xe_step_gmdid_get(struct xe_device *xe,
17
-
u32 graphics_gmdid_revid,
18
-
u32 media_gmdid_revid);
15
+
void xe_step_platform_get(struct xe_device *xe);
16
+
17
+
void xe_step_pre_gmdid_get(struct xe_device *xe);
18
+
void xe_step_gmdid_get(struct xe_device *xe,
19
+
u32 graphics_gmdid_revid,
20
+
u32 media_gmdid_revid);
19
21
static inline u32 xe_step_to_gmdid(enum xe_step step) { return step - STEP_A0; }
20
22
21
23
const char *xe_step_name(enum xe_step step);
+1
drivers/gpu/drm/xe/xe_step_types.h
+1
drivers/gpu/drm/xe/xe_step_types.h
+6
-2
drivers/gpu/drm/xe/xe_svm.c
+6
-2
drivers/gpu/drm/xe/xe_svm.c
···
19
19
#include "xe_pt.h"
20
20
#include "xe_svm.h"
21
21
#include "xe_tile.h"
22
+
#include "xe_tlb_inval.h"
22
23
#include "xe_ttm_vram_mgr.h"
23
24
#include "xe_vm.h"
24
25
#include "xe_vm_types.h"
···
226
225
const struct mmu_notifier_range *mmu_range)
227
226
{
228
227
struct xe_vm *vm = gpusvm_to_vm(gpusvm);
228
+
struct xe_tlb_inval_batch batch;
229
229
struct xe_device *xe = vm->xe;
230
230
struct drm_gpusvm_range *r, *first;
231
231
struct xe_tile *tile;
···
278
276
279
277
xe_device_wmb(xe);
280
278
281
-
err = xe_vm_range_tilemask_tlb_inval(vm, adj_start, adj_end, tile_mask);
282
-
WARN_ON_ONCE(err);
279
+
err = xe_tlb_inval_range_tilemask_submit(xe, vm->usm.asid, adj_start, adj_end,
280
+
tile_mask, &batch);
281
+
if (!WARN_ON_ONCE(err))
282
+
xe_tlb_inval_batch_wait(&batch);
283
283
284
284
range_notifier_event_end:
285
285
r = first;
+84
drivers/gpu/drm/xe/xe_tlb_inval.c
+84
drivers/gpu/drm/xe/xe_tlb_inval.c
···
486
486
guard(spinlock_irq)(&tlb_inval->pending_lock);
487
487
return list_is_singular(&tlb_inval->pending_fences);
488
488
}
489
+
490
+
/**
491
+
* xe_tlb_inval_batch_wait() - Wait for all fences in a TLB invalidation batch
492
+
* @batch: Batch of TLB invalidation fences to wait on
493
+
*
494
+
* Waits for every fence in @batch to signal, then resets @batch so it can be
495
+
* reused for a subsequent invalidation.
496
+
*/
497
+
void xe_tlb_inval_batch_wait(struct xe_tlb_inval_batch *batch)
498
+
{
499
+
struct xe_tlb_inval_fence *fence = &batch->fence[0];
500
+
unsigned int i;
501
+
502
+
for (i = 0; i < batch->num_fences; ++i)
503
+
xe_tlb_inval_fence_wait(fence++);
504
+
505
+
batch->num_fences = 0;
506
+
}
507
+
508
+
/**
509
+
* xe_tlb_inval_range_tilemask_submit() - Submit TLB invalidations for an
510
+
* address range on a tile mask
511
+
* @xe: The xe device
512
+
* @asid: Address space ID
513
+
* @start: start address
514
+
* @end: end address
515
+
* @tile_mask: mask for which gt's issue tlb invalidation
516
+
* @batch: Batch of tlb invalidate fences
517
+
*
518
+
* Issue a range based TLB invalidation for gt's in tilemask
519
+
* If the function returns an error, there is no need to call
520
+
* xe_tlb_inval_batch_wait() on @batch.
521
+
*
522
+
* Returns 0 for success, negative error code otherwise.
523
+
*/
524
+
int xe_tlb_inval_range_tilemask_submit(struct xe_device *xe, u32 asid,
525
+
u64 start, u64 end, u8 tile_mask,
526
+
struct xe_tlb_inval_batch *batch)
527
+
{
528
+
struct xe_tlb_inval_fence *fence = &batch->fence[0];
529
+
struct xe_tile *tile;
530
+
u32 fence_id = 0;
531
+
u8 id;
532
+
int err;
533
+
534
+
batch->num_fences = 0;
535
+
if (!tile_mask)
536
+
return 0;
537
+
538
+
for_each_tile(tile, xe, id) {
539
+
if (!(tile_mask & BIT(id)))
540
+
continue;
541
+
542
+
xe_tlb_inval_fence_init(&tile->primary_gt->tlb_inval,
543
+
&fence[fence_id], true);
544
+
545
+
err = xe_tlb_inval_range(&tile->primary_gt->tlb_inval,
546
+
&fence[fence_id], start, end,
547
+
asid, NULL);
548
+
if (err)
549
+
goto wait;
550
+
++fence_id;
551
+
552
+
if (!tile->media_gt)
553
+
continue;
554
+
555
+
xe_tlb_inval_fence_init(&tile->media_gt->tlb_inval,
556
+
&fence[fence_id], true);
557
+
558
+
err = xe_tlb_inval_range(&tile->media_gt->tlb_inval,
559
+
&fence[fence_id], start, end,
560
+
asid, NULL);
561
+
if (err)
562
+
goto wait;
563
+
++fence_id;
564
+
}
565
+
566
+
wait:
567
+
batch->num_fences = fence_id;
568
+
if (err)
569
+
xe_tlb_inval_batch_wait(batch);
570
+
571
+
return err;
572
+
}
+6
drivers/gpu/drm/xe/xe_tlb_inval.h
+6
drivers/gpu/drm/xe/xe_tlb_inval.h
···
45
45
46
46
bool xe_tlb_inval_idle(struct xe_tlb_inval *tlb_inval);
47
47
48
+
int xe_tlb_inval_range_tilemask_submit(struct xe_device *xe, u32 asid,
49
+
u64 start, u64 end, u8 tile_mask,
50
+
struct xe_tlb_inval_batch *batch);
51
+
52
+
void xe_tlb_inval_batch_wait(struct xe_tlb_inval_batch *batch);
53
+
48
54
#endif /* _XE_TLB_INVAL_ */
+14
drivers/gpu/drm/xe/xe_tlb_inval_types.h
+14
drivers/gpu/drm/xe/xe_tlb_inval_types.h
···
9
9
#include <linux/workqueue.h>
10
10
#include <linux/dma-fence.h>
11
11
12
+
#include "xe_device_types.h"
13
+
12
14
struct drm_suballoc;
13
15
struct xe_tlb_inval;
14
16
···
132
130
int seqno;
133
131
/** @inval_time: time of TLB invalidation */
134
132
ktime_t inval_time;
133
+
};
134
+
135
+
/**
136
+
* struct xe_tlb_inval_batch - Batch of TLB invalidation fences
137
+
*
138
+
* Holds one fence per GT covered by a TLB invalidation request.
139
+
*/
140
+
struct xe_tlb_inval_batch {
141
+
/** @fence: per-GT TLB invalidation fences */
142
+
struct xe_tlb_inval_fence fence[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE];
143
+
/** @num_fences: number of valid entries in @fence */
144
+
unsigned int num_fences;
135
145
};
136
146
137
147
#endif
+133
-22
drivers/gpu/drm/xe/xe_userptr.c
+133
-22
drivers/gpu/drm/xe/xe_userptr.c
···
8
8
9
9
#include <linux/mm.h>
10
10
11
+
#include "xe_tlb_inval.h"
11
12
#include "xe_trace_bo.h"
13
+
14
+
static void xe_userptr_assert_in_notifier(struct xe_vm *vm)
15
+
{
16
+
lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) ||
17
+
(lockdep_is_held(&vm->lock) &&
18
+
lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 1) &&
19
+
dma_resv_held(xe_vm_resv(vm))));
20
+
}
12
21
13
22
/**
14
23
* xe_vma_userptr_check_repin() - Advisory check for repin needed
···
82
73
&ctx);
83
74
}
84
75
85
-
static void __vma_userptr_invalidate(struct xe_vm *vm, struct xe_userptr_vma *uvma)
76
+
static struct mmu_interval_notifier_finish *
77
+
xe_vma_userptr_do_inval(struct xe_vm *vm, struct xe_userptr_vma *uvma, bool is_deferred)
86
78
{
87
79
struct xe_userptr *userptr = &uvma->userptr;
88
80
struct xe_vma *vma = &uvma->vma;
89
-
struct dma_resv_iter cursor;
90
-
struct dma_fence *fence;
91
81
struct drm_gpusvm_ctx ctx = {
92
82
.in_notifier = true,
93
83
.read_only = xe_vma_read_only(vma),
94
84
};
95
85
long err;
86
+
87
+
xe_userptr_assert_in_notifier(vm);
88
+
if (is_deferred)
89
+
xe_assert(vm->xe, userptr->finish_inuse && !userptr->tlb_inval_submitted);
90
+
91
+
err = dma_resv_wait_timeout(xe_vm_resv(vm),
92
+
DMA_RESV_USAGE_BOOKKEEP,
93
+
false, MAX_SCHEDULE_TIMEOUT);
94
+
XE_WARN_ON(err <= 0);
95
+
96
+
if (xe_vm_in_fault_mode(vm) && userptr->initial_bind) {
97
+
if (!userptr->finish_inuse) {
98
+
/*
99
+
* Defer the TLB wait to an extra pass so the caller
100
+
* can pipeline TLB flushes across GPUs before waiting
101
+
* on any of them.
102
+
*/
103
+
xe_assert(vm->xe, !userptr->tlb_inval_submitted);
104
+
userptr->finish_inuse = true;
105
+
userptr->tlb_inval_submitted = true;
106
+
err = xe_vm_invalidate_vma_submit(vma, &userptr->inval_batch);
107
+
XE_WARN_ON(err);
108
+
return &userptr->finish;
109
+
}
110
+
err = xe_vm_invalidate_vma(vma);
111
+
XE_WARN_ON(err);
112
+
}
113
+
114
+
if (is_deferred)
115
+
userptr->finish_inuse = false;
116
+
drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages,
117
+
xe_vma_size(vma) >> PAGE_SHIFT, &ctx);
118
+
return NULL;
119
+
}
120
+
121
+
static void
122
+
xe_vma_userptr_complete_tlb_inval(struct xe_vm *vm, struct xe_userptr_vma *uvma)
123
+
{
124
+
struct xe_userptr *userptr = &uvma->userptr;
125
+
struct xe_vma *vma = &uvma->vma;
126
+
struct drm_gpusvm_ctx ctx = {
127
+
.in_notifier = true,
128
+
.read_only = xe_vma_read_only(vma),
129
+
};
130
+
131
+
xe_userptr_assert_in_notifier(vm);
132
+
xe_assert(vm->xe, userptr->finish_inuse);
133
+
xe_assert(vm->xe, userptr->tlb_inval_submitted);
134
+
135
+
xe_tlb_inval_batch_wait(&userptr->inval_batch);
136
+
userptr->tlb_inval_submitted = false;
137
+
userptr->finish_inuse = false;
138
+
drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages,
139
+
xe_vma_size(vma) >> PAGE_SHIFT, &ctx);
140
+
}
141
+
142
+
static struct mmu_interval_notifier_finish *
143
+
xe_vma_userptr_invalidate_pass1(struct xe_vm *vm, struct xe_userptr_vma *uvma)
144
+
{
145
+
struct xe_userptr *userptr = &uvma->userptr;
146
+
struct xe_vma *vma = &uvma->vma;
147
+
struct dma_resv_iter cursor;
148
+
struct dma_fence *fence;
149
+
bool signaled = true;
150
+
151
+
xe_userptr_assert_in_notifier(vm);
96
152
97
153
/*
98
154
* Tell exec and rebind worker they need to repin and rebind this
···
179
105
*/
180
106
dma_resv_iter_begin(&cursor, xe_vm_resv(vm),
181
107
DMA_RESV_USAGE_BOOKKEEP);
182
-
dma_resv_for_each_fence_unlocked(&cursor, fence)
108
+
dma_resv_for_each_fence_unlocked(&cursor, fence) {
183
109
dma_fence_enable_sw_signaling(fence);
110
+
if (signaled && !dma_fence_is_signaled(fence))
111
+
signaled = false;
112
+
}
184
113
dma_resv_iter_end(&cursor);
185
114
186
-
err = dma_resv_wait_timeout(xe_vm_resv(vm),
187
-
DMA_RESV_USAGE_BOOKKEEP,
188
-
false, MAX_SCHEDULE_TIMEOUT);
189
-
XE_WARN_ON(err <= 0);
115
+
/*
116
+
* Only one caller at a time can use the multi-pass state.
117
+
* If it's already in use, or all fences are already signaled,
118
+
* proceed directly to invalidation without deferring.
119
+
*/
120
+
if (signaled || userptr->finish_inuse)
121
+
return xe_vma_userptr_do_inval(vm, uvma, false);
190
122
191
-
if (xe_vm_in_fault_mode(vm) && userptr->initial_bind) {
192
-
err = xe_vm_invalidate_vma(vma);
193
-
XE_WARN_ON(err);
194
-
}
123
+
/* Defer: the notifier core will call invalidate_finish once done. */
124
+
userptr->finish_inuse = true;
195
125
196
-
drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages,
197
-
xe_vma_size(vma) >> PAGE_SHIFT, &ctx);
126
+
return &userptr->finish;
198
127
}
199
128
200
-
static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni,
201
-
const struct mmu_notifier_range *range,
202
-
unsigned long cur_seq)
129
+
static bool xe_vma_userptr_invalidate_start(struct mmu_interval_notifier *mni,
130
+
const struct mmu_notifier_range *range,
131
+
unsigned long cur_seq,
132
+
struct mmu_interval_notifier_finish **p_finish)
203
133
{
204
134
struct xe_userptr_vma *uvma = container_of(mni, typeof(*uvma), userptr.notifier);
205
135
struct xe_vma *vma = &uvma->vma;
···
216
138
return false;
217
139
218
140
vm_dbg(&xe_vma_vm(vma)->xe->drm,
219
-
"NOTIFIER: addr=0x%016llx, range=0x%016llx",
141
+
"NOTIFIER PASS1: addr=0x%016llx, range=0x%016llx",
220
142
xe_vma_start(vma), xe_vma_size(vma));
221
143
222
144
down_write(&vm->svm.gpusvm.notifier_lock);
223
145
mmu_interval_set_seq(mni, cur_seq);
224
146
225
-
__vma_userptr_invalidate(vm, uvma);
147
+
*p_finish = xe_vma_userptr_invalidate_pass1(vm, uvma);
148
+
226
149
up_write(&vm->svm.gpusvm.notifier_lock);
227
-
trace_xe_vma_userptr_invalidate_complete(vma);
150
+
if (!*p_finish)
151
+
trace_xe_vma_userptr_invalidate_complete(vma);
228
152
229
153
return true;
230
154
}
231
155
156
+
static void xe_vma_userptr_invalidate_finish(struct mmu_interval_notifier_finish *finish)
157
+
{
158
+
struct xe_userptr_vma *uvma = container_of(finish, typeof(*uvma), userptr.finish);
159
+
struct xe_vma *vma = &uvma->vma;
160
+
struct xe_vm *vm = xe_vma_vm(vma);
161
+
162
+
vm_dbg(&xe_vma_vm(vma)->xe->drm,
163
+
"NOTIFIER PASS2: addr=0x%016llx, range=0x%016llx",
164
+
xe_vma_start(vma), xe_vma_size(vma));
165
+
166
+
down_write(&vm->svm.gpusvm.notifier_lock);
167
+
/*
168
+
* If a TLB invalidation was previously submitted (deferred from the
169
+
* synchronous pass1 fallback), wait for it and unmap pages.
170
+
* Otherwise, fences have now completed: invalidate the TLB and unmap.
171
+
*/
172
+
if (uvma->userptr.tlb_inval_submitted)
173
+
xe_vma_userptr_complete_tlb_inval(vm, uvma);
174
+
else
175
+
xe_vma_userptr_do_inval(vm, uvma, true);
176
+
up_write(&vm->svm.gpusvm.notifier_lock);
177
+
trace_xe_vma_userptr_invalidate_complete(vma);
178
+
}
179
+
232
180
static const struct mmu_interval_notifier_ops vma_userptr_notifier_ops = {
233
-
.invalidate = vma_userptr_invalidate,
181
+
.invalidate_start = xe_vma_userptr_invalidate_start,
182
+
.invalidate_finish = xe_vma_userptr_invalidate_finish,
234
183
};
235
184
236
185
#if IS_ENABLED(CONFIG_DRM_XE_USERPTR_INVAL_INJECT)
···
269
164
*/
270
165
void xe_vma_userptr_force_invalidate(struct xe_userptr_vma *uvma)
271
166
{
167
+
static struct mmu_interval_notifier_finish *finish;
272
168
struct xe_vm *vm = xe_vma_vm(&uvma->vma);
273
169
274
170
/* Protect against concurrent userptr pinning */
···
285
179
if (!mmu_interval_read_retry(&uvma->userptr.notifier,
286
180
uvma->userptr.pages.notifier_seq))
287
181
uvma->userptr.pages.notifier_seq -= 2;
288
-
__vma_userptr_invalidate(vm, uvma);
182
+
183
+
finish = xe_vma_userptr_invalidate_pass1(vm, uvma);
184
+
if (finish)
185
+
finish = xe_vma_userptr_do_inval(vm, uvma, true);
186
+
if (finish)
187
+
xe_vma_userptr_complete_tlb_inval(vm, uvma);
289
188
}
290
189
#endif
291
190
+30
-1
drivers/gpu/drm/xe/xe_userptr.h
+30
-1
drivers/gpu/drm/xe/xe_userptr.h
···
14
14
15
15
#include <drm/drm_gpusvm.h>
16
16
17
+
#include "xe_tlb_inval_types.h"
18
+
17
19
struct xe_vm;
18
20
struct xe_vma;
19
21
struct xe_userptr_vma;
···
58
56
* @notifier: MMU notifier for user pointer (invalidation call back)
59
57
*/
60
58
struct mmu_interval_notifier notifier;
61
-
59
+
/**
60
+
* @finish: MMU notifier finish structure for two-pass invalidation.
61
+
* Embedded here to avoid allocation in the notifier callback.
62
+
* Protected by struct xe_vm::svm.gpusvm.notifier_lock in write mode
63
+
* alternatively by the same lock in read mode *and* the vm resv held.
64
+
*/
65
+
struct mmu_interval_notifier_finish finish;
66
+
/**
67
+
* @inval_batch: TLB invalidation batch for deferred completion.
68
+
* Stores an in-flight TLB invalidation submitted during a two-pass
69
+
* notifier so the wait can be deferred to a subsequent pass, allowing
70
+
* multiple GPUs to be signalled before any of them are waited on.
71
+
* Protected using the same locking as @finish.
72
+
*/
73
+
struct xe_tlb_inval_batch inval_batch;
74
+
/**
75
+
* @finish_inuse: Whether @finish is currently in use by an in-progress
76
+
* two-pass invalidation.
77
+
* Protected using the same locking as @finish.
78
+
*/
79
+
bool finish_inuse;
80
+
/**
81
+
* @tlb_inval_submitted: Whether a TLB invalidation has been submitted
82
+
* via @inval_batch and is pending completion. When set, the next pass
83
+
* must call xe_tlb_inval_batch_wait() before reusing @inval_batch.
84
+
* Protected using the same locking as @finish.
85
+
*/
86
+
bool tlb_inval_submitted;
62
87
/**
63
88
* @initial_bind: user pointer has been bound at least once.
64
89
* write: vm->svm.gpusvm.notifier_lock in read mode and vm->resv held.
+61
-75
drivers/gpu/drm/xe/xe_vm.c
+61
-75
drivers/gpu/drm/xe/xe_vm.c
···
2362
2362
op->map.vma_flags |= XE_VMA_DUMPABLE;
2363
2363
if (flags & DRM_XE_VM_BIND_FLAG_MADVISE_AUTORESET)
2364
2364
op->map.vma_flags |= XE_VMA_MADV_AUTORESET;
2365
+
op->map.request_decompress = flags & DRM_XE_VM_BIND_FLAG_DECOMPRESS;
2365
2366
op->map.pat_index = pat_index;
2366
2367
op->map.invalidate_on_bind =
2367
2368
__xe_vm_needs_clear_scratch_pages(vm, flags);
···
2903
2902
}
2904
2903
2905
2904
static int vma_lock_and_validate(struct drm_exec *exec, struct xe_vma *vma,
2906
-
bool res_evict, bool validate)
2905
+
bool res_evict, bool validate, bool request_decompress)
2907
2906
{
2908
2907
struct xe_bo *bo = xe_vma_bo(vma);
2909
2908
struct xe_vm *vm = xe_vma_vm(vma);
···
2916
2915
err = xe_bo_validate(bo, vm,
2917
2916
xe_vm_allow_vm_eviction(vm) &&
2918
2917
res_evict, exec);
2918
+
2919
+
if (err)
2920
+
return err;
2921
+
2922
+
if (request_decompress)
2923
+
err = xe_bo_decompress(bo);
2919
2924
}
2920
2925
2921
2926
return err;
···
3016
3009
err = vma_lock_and_validate(exec, op->map.vma,
3017
3010
res_evict,
3018
3011
!xe_vm_in_fault_mode(vm) ||
3019
-
op->map.immediate);
3012
+
op->map.immediate,
3013
+
op->map.request_decompress);
3020
3014
break;
3021
3015
case DRM_GPUVA_OP_REMAP:
3022
3016
err = check_ufence(gpuva_to_vma(op->base.remap.unmap->va));
···
3026
3018
3027
3019
err = vma_lock_and_validate(exec,
3028
3020
gpuva_to_vma(op->base.remap.unmap->va),
3029
-
res_evict, false);
3021
+
res_evict, false, false);
3030
3022
if (!err && op->remap.prev)
3031
3023
err = vma_lock_and_validate(exec, op->remap.prev,
3032
-
res_evict, true);
3024
+
res_evict, true, false);
3033
3025
if (!err && op->remap.next)
3034
3026
err = vma_lock_and_validate(exec, op->remap.next,
3035
-
res_evict, true);
3027
+
res_evict, true, false);
3036
3028
break;
3037
3029
case DRM_GPUVA_OP_UNMAP:
3038
3030
err = check_ufence(gpuva_to_vma(op->base.unmap.va));
···
3041
3033
3042
3034
err = vma_lock_and_validate(exec,
3043
3035
gpuva_to_vma(op->base.unmap.va),
3044
-
res_evict, false);
3036
+
res_evict, false, false);
3045
3037
break;
3046
3038
case DRM_GPUVA_OP_PREFETCH:
3047
3039
{
···
3056
3048
3057
3049
err = vma_lock_and_validate(exec,
3058
3050
gpuva_to_vma(op->base.prefetch.va),
3059
-
res_evict, false);
3051
+
res_evict, false, false);
3060
3052
if (!err && !xe_vma_has_no_bo(vma))
3061
3053
err = xe_bo_migrate(xe_vma_bo(vma),
3062
3054
region_to_mem_type[region],
···
3378
3370
DRM_XE_VM_BIND_FLAG_DUMPABLE | \
3379
3371
DRM_XE_VM_BIND_FLAG_CHECK_PXP | \
3380
3372
DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR | \
3381
-
DRM_XE_VM_BIND_FLAG_MADVISE_AUTORESET)
3373
+
DRM_XE_VM_BIND_FLAG_MADVISE_AUTORESET | \
3374
+
DRM_XE_VM_BIND_FLAG_DECOMPRESS)
3382
3375
3383
3376
#ifdef TEST_VM_OPS_ERROR
3384
3377
#define SUPPORTED_FLAGS (SUPPORTED_FLAGS_STUB | FORCE_OP_ERROR)
···
3439
3430
bool is_null = flags & DRM_XE_VM_BIND_FLAG_NULL;
3440
3431
bool is_cpu_addr_mirror = flags &
3441
3432
DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR;
3433
+
bool is_decompress = flags & DRM_XE_VM_BIND_FLAG_DECOMPRESS;
3442
3434
u16 pat_index = (*bind_ops)[i].pat_index;
3443
3435
u16 coh_mode;
3444
3436
bool comp_en;
···
3476
3466
XE_IOCTL_DBG(xe, obj_offset && (is_null ||
3477
3467
is_cpu_addr_mirror)) ||
3478
3468
XE_IOCTL_DBG(xe, op != DRM_XE_VM_BIND_OP_MAP &&
3479
-
(is_null || is_cpu_addr_mirror)) ||
3469
+
(is_decompress || is_null || is_cpu_addr_mirror)) ||
3470
+
XE_IOCTL_DBG(xe, is_decompress &&
3471
+
xe_pat_index_get_comp_en(xe, pat_index)) ||
3480
3472
XE_IOCTL_DBG(xe, !obj &&
3481
3473
op == DRM_XE_VM_BIND_OP_MAP &&
3482
3474
!is_null && !is_cpu_addr_mirror) ||
···
3518
3506
XE_IOCTL_DBG(xe, !range &&
3519
3507
op != DRM_XE_VM_BIND_OP_UNMAP_ALL)) {
3520
3508
err = -EINVAL;
3509
+
goto free_bind_ops;
3510
+
}
3511
+
3512
+
if (is_decompress && (XE_IOCTL_DBG(xe, !xe_device_has_flat_ccs(xe)) ||
3513
+
XE_IOCTL_DBG(xe, GRAPHICS_VER(xe) < 20) ||
3514
+
XE_IOCTL_DBG(xe, !IS_DGFX(xe)))) {
3515
+
err = -EOPNOTSUPP;
3521
3516
goto free_bind_ops;
3522
3517
}
3523
3518
}
···
3986
3967
}
3987
3968
3988
3969
/**
3989
-
* xe_vm_range_tilemask_tlb_inval - Issue a TLB invalidation on this tilemask for an
3990
-
* address range
3991
-
* @vm: The VM
3992
-
* @start: start address
3993
-
* @end: end address
3994
-
* @tile_mask: mask for which gt's issue tlb invalidation
3995
-
*
3996
-
* Issue a range based TLB invalidation for gt's in tilemask
3997
-
*
3998
-
* Returns 0 for success, negative error code otherwise.
3999
-
*/
4000
-
int xe_vm_range_tilemask_tlb_inval(struct xe_vm *vm, u64 start,
4001
-
u64 end, u8 tile_mask)
4002
-
{
4003
-
struct xe_tlb_inval_fence
4004
-
fence[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE];
4005
-
struct xe_tile *tile;
4006
-
u32 fence_id = 0;
4007
-
u8 id;
4008
-
int err;
4009
-
4010
-
if (!tile_mask)
4011
-
return 0;
4012
-
4013
-
for_each_tile(tile, vm->xe, id) {
4014
-
if (!(tile_mask & BIT(id)))
4015
-
continue;
4016
-
4017
-
xe_tlb_inval_fence_init(&tile->primary_gt->tlb_inval,
4018
-
&fence[fence_id], true);
4019
-
4020
-
err = xe_tlb_inval_range(&tile->primary_gt->tlb_inval,
4021
-
&fence[fence_id], start, end,
4022
-
vm->usm.asid, NULL);
4023
-
if (err)
4024
-
goto wait;
4025
-
++fence_id;
4026
-
4027
-
if (!tile->media_gt)
4028
-
continue;
4029
-
4030
-
xe_tlb_inval_fence_init(&tile->media_gt->tlb_inval,
4031
-
&fence[fence_id], true);
4032
-
4033
-
err = xe_tlb_inval_range(&tile->media_gt->tlb_inval,
4034
-
&fence[fence_id], start, end,
4035
-
vm->usm.asid, NULL);
4036
-
if (err)
4037
-
goto wait;
4038
-
++fence_id;
4039
-
}
4040
-
4041
-
wait:
4042
-
for (id = 0; id < fence_id; ++id)
4043
-
xe_tlb_inval_fence_wait(&fence[id]);
4044
-
4045
-
return err;
4046
-
}
4047
-
4048
-
/**
4049
-
* xe_vm_invalidate_vma - invalidate GPU mappings for VMA without a lock
3970
+
* xe_vm_invalidate_vma_submit - Submit a job to invalidate GPU mappings for
3971
+
* VMA.
4050
3972
* @vma: VMA to invalidate
3973
+
* @batch: TLB invalidation batch to populate; caller must later call
3974
+
* xe_tlb_inval_batch_wait() on it to wait for completion
4051
3975
*
4052
3976
* Walks a list of page tables leaves which it memset the entries owned by this
4053
-
* VMA to zero, invalidates the TLBs, and block until TLBs invalidation is
4054
-
* complete.
3977
+
* VMA to zero, invalidates the TLBs, but doesn't block waiting for TLB flush
3978
+
* to complete, but instead populates @batch which can be waited on using
3979
+
* xe_tlb_inval_batch_wait().
4055
3980
*
4056
3981
* Returns 0 for success, negative error code otherwise.
4057
3982
*/
4058
-
int xe_vm_invalidate_vma(struct xe_vma *vma)
3983
+
int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch)
4059
3984
{
4060
3985
struct xe_device *xe = xe_vma_vm(vma)->xe;
4061
3986
struct xe_vm *vm = xe_vma_vm(vma);
···
4043
4080
4044
4081
xe_device_wmb(xe);
4045
4082
4046
-
ret = xe_vm_range_tilemask_tlb_inval(xe_vma_vm(vma), xe_vma_start(vma),
4047
-
xe_vma_end(vma), tile_mask);
4083
+
ret = xe_tlb_inval_range_tilemask_submit(xe, xe_vma_vm(vma)->usm.asid,
4084
+
xe_vma_start(vma), xe_vma_end(vma),
4085
+
tile_mask, batch);
4048
4086
4049
4087
/* WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() */
4050
4088
WRITE_ONCE(vma->tile_invalidated, vma->tile_mask);
4089
+
return ret;
4090
+
}
4051
4091
4092
+
/**
4093
+
* xe_vm_invalidate_vma - invalidate GPU mappings for VMA without a lock
4094
+
* @vma: VMA to invalidate
4095
+
*
4096
+
* Walks a list of page tables leaves which it memset the entries owned by this
4097
+
* VMA to zero, invalidates the TLBs, and block until TLBs invalidation is
4098
+
* complete.
4099
+
*
4100
+
* Returns 0 for success, negative error code otherwise.
4101
+
*/
4102
+
int xe_vm_invalidate_vma(struct xe_vma *vma)
4103
+
{
4104
+
struct xe_tlb_inval_batch batch;
4105
+
int ret;
4106
+
4107
+
ret = xe_vm_invalidate_vma_submit(vma, &batch);
4108
+
if (ret)
4109
+
return ret;
4110
+
4111
+
xe_tlb_inval_batch_wait(&batch);
4052
4112
return ret;
4053
4113
}
4054
4114
+2
-3
drivers/gpu/drm/xe/xe_vm.h
+2
-3
drivers/gpu/drm/xe/xe_vm.h
···
240
240
struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm,
241
241
struct xe_svm_range *range);
242
242
243
-
int xe_vm_range_tilemask_tlb_inval(struct xe_vm *vm, u64 start,
244
-
u64 end, u8 tile_mask);
245
-
246
243
int xe_vm_invalidate_vma(struct xe_vma *vma);
244
+
245
+
int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch);
247
246
248
247
int xe_vm_validate_protected(struct xe_vm *vm);
249
248
+9
-1
drivers/gpu/drm/xe/xe_vm_madvise.c
+9
-1
drivers/gpu/drm/xe/xe_vm_madvise.c
···
12
12
#include "xe_pat.h"
13
13
#include "xe_pt.h"
14
14
#include "xe_svm.h"
15
+
#include "xe_tlb_inval.h"
15
16
16
17
struct xe_vmas_in_madvise_range {
17
18
u64 addr;
···
236
235
static int xe_vm_invalidate_madvise_range(struct xe_vm *vm, u64 start, u64 end)
237
236
{
238
237
u8 tile_mask = xe_zap_ptes_in_madvise_range(vm, start, end);
238
+
struct xe_tlb_inval_batch batch;
239
+
int err;
239
240
240
241
if (!tile_mask)
241
242
return 0;
242
243
243
244
xe_device_wmb(vm->xe);
244
245
245
-
return xe_vm_range_tilemask_tlb_inval(vm, start, end, tile_mask);
246
+
err = xe_tlb_inval_range_tilemask_submit(vm->xe, vm->usm.asid, start, end,
247
+
tile_mask, &batch);
248
+
if (!err)
249
+
xe_tlb_inval_batch_wait(&batch);
250
+
251
+
return err;
246
252
}
247
253
248
254
static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_xe_madvise *args)
+3
drivers/gpu/drm/xe/xe_vm_types.h
+3
drivers/gpu/drm/xe/xe_vm_types.h
···
18
18
#include "xe_device_types.h"
19
19
#include "xe_pt_types.h"
20
20
#include "xe_range_fence.h"
21
+
#include "xe_tlb_inval_types.h"
21
22
#include "xe_userptr.h"
22
23
23
24
struct drm_pagemap;
···
377
376
bool immediate;
378
377
/** @read_only: Read only */
379
378
bool invalidate_on_bind;
379
+
/** @request_decompress: schedule decompression for GPU map */
380
+
bool request_decompress;
380
381
/** @pat_index: The pat index to use for this operation. */
381
382
u16 pat_index;
382
383
};
+5
-4
drivers/gpu/drm/xe/xe_wa.c
+5
-4
drivers/gpu/drm/xe/xe_wa.c
···
164
164
MEDIA_VERSION_RANGE(1301, 3500)),
165
165
XE_RTP_ACTIONS(SET(GUC_INTR_CHICKEN, DISABLE_SIGNALING_ENGINES))
166
166
},
167
+
{ XE_RTP_NAME("14026578760"),
168
+
XE_RTP_RULES(GRAPHICS_VERSION_RANGE(3510, 3511), OR,
169
+
MEDIA_VERSION(3503)),
170
+
XE_RTP_ACTIONS(SET(GAMSTLB_CTRL, DIS_PEND_GPA_LINK))
171
+
},
167
172
168
173
/* DG1 */
169
174
···
305
300
XE_RTP_RULES(GRAPHICS_VERSION(3510), GRAPHICS_STEP(A0, B0)),
306
301
XE_RTP_ACTIONS(SET(MMIOATSREQLIMIT_GAM_WALK_3D,
307
302
DIS_ATS_WRONLY_PG))
308
-
},
309
-
{ XE_RTP_NAME("16028780921"),
310
-
XE_RTP_RULES(GRAPHICS_VERSION(3510), GRAPHICS_STEP(A0, B0)),
311
-
XE_RTP_ACTIONS(SET(CCCHKNREG2, LOCALITYDIS))
312
303
},
313
304
{ XE_RTP_NAME("14026144927"),
314
305
XE_RTP_RULES(GRAPHICS_VERSION(3510), GRAPHICS_STEP(A0, B0)),
+75
include/drm/drm_ras.h
+75
include/drm/drm_ras.h
···
1
+
/* SPDX-License-Identifier: MIT */
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#ifndef __DRM_RAS_H__
7
+
#define __DRM_RAS_H__
8
+
9
+
#include <uapi/drm/drm_ras.h>
10
+
11
+
/**
12
+
* struct drm_ras_node - A DRM RAS Node
13
+
*/
14
+
struct drm_ras_node {
15
+
/** @id: Unique identifier for the node. Dynamically assigned. */
16
+
u32 id;
17
+
/**
18
+
* @device_name: Human-readable name of the device. Given by the driver.
19
+
*/
20
+
const char *device_name;
21
+
/** @node_name: Human-readable name of the node. Given by the driver. */
22
+
const char *node_name;
23
+
/** @type: Type of the node (enum drm_ras_node_type). */
24
+
enum drm_ras_node_type type;
25
+
26
+
/* Error-Counter Related Callback and Variables */
27
+
28
+
/** @error_counter_range: Range of valid Error IDs for this node. */
29
+
struct {
30
+
/** @first: First valid Error ID. */
31
+
u32 first;
32
+
/** @last: Last valid Error ID. Mandatory entry. */
33
+
u32 last;
34
+
} error_counter_range;
35
+
36
+
/**
37
+
* @query_error_counter:
38
+
*
39
+
* This callback is used by drm-ras to query a specific error counter.
40
+
* Used for input check and to iterate all error counters in a node.
41
+
*
42
+
* Driver should expect query_error_counter() to be called with
43
+
* error_id from `error_counter_range.first` to
44
+
* `error_counter_range.last`.
45
+
*
46
+
* The @query_error_counter is a mandatory callback for
47
+
* error_counter_node.
48
+
*
49
+
* Returns: 0 on success,
50
+
* -ENOENT when error_id is not supported as an indication that
51
+
* drm_ras should silently skip this entry. Used for
52
+
* supporting non-contiguous error ranges.
53
+
* Driver is responsible for maintaining the list of
54
+
* supported error IDs in the range of first to last.
55
+
* Other negative values on errors that should terminate the
56
+
* netlink query.
57
+
*/
58
+
int (*query_error_counter)(struct drm_ras_node *node, u32 error_id,
59
+
const char **name, u32 *val);
60
+
61
+
/** @priv: Driver private data */
62
+
void *priv;
63
+
};
64
+
65
+
struct drm_device;
66
+
67
+
#if IS_ENABLED(CONFIG_DRM_RAS)
68
+
int drm_ras_node_register(struct drm_ras_node *node);
69
+
void drm_ras_node_unregister(struct drm_ras_node *node);
70
+
#else
71
+
static inline int drm_ras_node_register(struct drm_ras_node *node) { return 0; }
72
+
static inline void drm_ras_node_unregister(struct drm_ras_node *node) { }
73
+
#endif
74
+
75
+
#endif
+17
include/drm/drm_ras_genl_family.h
+17
include/drm/drm_ras_genl_family.h
···
1
+
/* SPDX-License-Identifier: MIT */
2
+
/*
3
+
* Copyright © 2026 Intel Corporation
4
+
*/
5
+
6
+
#ifndef __DRM_RAS_GENL_FAMILY_H__
7
+
#define __DRM_RAS_GENL_FAMILY_H__
8
+
9
+
#if IS_ENABLED(CONFIG_DRM_RAS)
10
+
int drm_ras_genl_family_register(void);
11
+
void drm_ras_genl_family_unregister(void);
12
+
#else
13
+
static inline int drm_ras_genl_family_register(void) { return 0; }
14
+
static inline void drm_ras_genl_family_unregister(void) { }
15
+
#endif
16
+
17
+
#endif
+42
include/linux/mmu_notifier.h
+42
include/linux/mmu_notifier.h
···
234
234
};
235
235
236
236
/**
237
+
* struct mmu_interval_notifier_finish - mmu_interval_notifier two-pass abstraction
238
+
* @link: Lockless list link for the notifiers pending pass list
239
+
* @notifier: The mmu_interval_notifier for which the finish pass is called.
240
+
*
241
+
* Allocate, typically using GFP_NOWAIT in the interval notifier's start pass.
242
+
* Note that with a large number of notifiers implementing two passes,
243
+
* allocation with GFP_NOWAIT will become increasingly likely to fail, so consider
244
+
* implementing a small pool instead of using kmalloc() allocations.
245
+
*
246
+
* If the implementation needs to pass data between the start and the finish passes,
247
+
* the recommended way is to embed struct mmu_interval_notifier_finish into a larger
248
+
* structure that also contains the data needed to be shared. Keep in mind that
249
+
* a notifier callback can be invoked in parallel, and each invocation needs its
250
+
* own struct mmu_interval_notifier_finish.
251
+
*
252
+
* If allocation fails, then the &mmu_interval_notifier_ops->invalidate_start op
253
+
* needs to implements the full notifier functionality. Please refer to its
254
+
* documentation.
255
+
*/
256
+
struct mmu_interval_notifier_finish {
257
+
struct llist_node link;
258
+
struct mmu_interval_notifier *notifier;
259
+
};
260
+
261
+
/**
237
262
* struct mmu_interval_notifier_ops
238
263
* @invalidate: Upon return the caller must stop using any SPTEs within this
239
264
* range. This function can sleep. Return false only if sleeping
240
265
* was required but mmu_notifier_range_blockable(range) is false.
266
+
* @invalidate_start: Similar to @invalidate, but intended for two-pass notifier
267
+
* callbacks where the call to @invalidate_start is the first
268
+
* pass and any struct mmu_interval_notifier_finish pointer
269
+
* returned in the @finish parameter describes the finish pass.
270
+
* If *@finish is %NULL on return, then no final pass will be
271
+
* called, and @invalidate_start needs to implement the full
272
+
* notifier, behaving like @invalidate. The value of *@finish
273
+
* is guaranteed to be %NULL at function entry.
274
+
* @invalidate_finish: Called as the second pass for any notifier that returned
275
+
* a non-NULL *@finish from @invalidate_start. The @finish
276
+
* pointer passed here is the same one returned by
277
+
* @invalidate_start.
241
278
*/
242
279
struct mmu_interval_notifier_ops {
243
280
bool (*invalidate)(struct mmu_interval_notifier *interval_sub,
244
281
const struct mmu_notifier_range *range,
245
282
unsigned long cur_seq);
283
+
bool (*invalidate_start)(struct mmu_interval_notifier *interval_sub,
284
+
const struct mmu_notifier_range *range,
285
+
unsigned long cur_seq,
286
+
struct mmu_interval_notifier_finish **finish);
287
+
void (*invalidate_finish)(struct mmu_interval_notifier_finish *finish);
246
288
};
247
289
248
290
struct mmu_interval_notifier {
+49
include/uapi/drm/drm_ras.h
+49
include/uapi/drm/drm_ras.h
···
1
+
/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
2
+
/* Do not edit directly, auto-generated from: */
3
+
/* Documentation/netlink/specs/drm_ras.yaml */
4
+
/* YNL-GEN uapi header */
5
+
/* To regenerate run: tools/net/ynl/ynl-regen.sh */
6
+
7
+
#ifndef _UAPI_LINUX_DRM_RAS_H
8
+
#define _UAPI_LINUX_DRM_RAS_H
9
+
10
+
#define DRM_RAS_FAMILY_NAME "drm-ras"
11
+
#define DRM_RAS_FAMILY_VERSION 1
12
+
13
+
/*
14
+
* Type of the node. Currently, only error-counter nodes are supported, which
15
+
* expose reliability counters for a hardware/software component.
16
+
*/
17
+
enum drm_ras_node_type {
18
+
DRM_RAS_NODE_TYPE_ERROR_COUNTER = 1,
19
+
};
20
+
21
+
enum {
22
+
DRM_RAS_A_NODE_ATTRS_NODE_ID = 1,
23
+
DRM_RAS_A_NODE_ATTRS_DEVICE_NAME,
24
+
DRM_RAS_A_NODE_ATTRS_NODE_NAME,
25
+
DRM_RAS_A_NODE_ATTRS_NODE_TYPE,
26
+
27
+
__DRM_RAS_A_NODE_ATTRS_MAX,
28
+
DRM_RAS_A_NODE_ATTRS_MAX = (__DRM_RAS_A_NODE_ATTRS_MAX - 1)
29
+
};
30
+
31
+
enum {
32
+
DRM_RAS_A_ERROR_COUNTER_ATTRS_NODE_ID = 1,
33
+
DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_ID,
34
+
DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_NAME,
35
+
DRM_RAS_A_ERROR_COUNTER_ATTRS_ERROR_VALUE,
36
+
37
+
__DRM_RAS_A_ERROR_COUNTER_ATTRS_MAX,
38
+
DRM_RAS_A_ERROR_COUNTER_ATTRS_MAX = (__DRM_RAS_A_ERROR_COUNTER_ATTRS_MAX - 1)
39
+
};
40
+
41
+
enum {
42
+
DRM_RAS_CMD_LIST_NODES = 1,
43
+
DRM_RAS_CMD_GET_ERROR_COUNTER,
44
+
45
+
__DRM_RAS_CMD_MAX,
46
+
DRM_RAS_CMD_MAX = (__DRM_RAS_CMD_MAX - 1)
47
+
};
48
+
49
+
#endif /* _UAPI_LINUX_DRM_RAS_H */
+95
include/uapi/drm/xe_drm.h
+95
include/uapi/drm/xe_drm.h
···
406
406
* - %DRM_XE_QUERY_CONFIG_FLAG_HAS_NO_COMPRESSION_HINT - Flag is set if the
407
407
* device supports the userspace hint %DRM_XE_GEM_CREATE_FLAG_NO_COMPRESSION.
408
408
* This is exposed only on Xe2+.
409
+
* - %DRM_XE_QUERY_CONFIG_FLAG_HAS_DISABLE_STATE_CACHE_PERF_FIX - Flag is set
410
+
* if a queue can be creaed with
411
+
* %DRM_XE_EXEC_QUEUE_SET_DISABLE_STATE_CACHE_PERF_FIX
409
412
* - %DRM_XE_QUERY_CONFIG_MIN_ALIGNMENT - Minimal memory alignment
410
413
* required by this device, typically SZ_4K or SZ_64K
411
414
* - %DRM_XE_QUERY_CONFIG_VA_BITS - Maximum bits of a virtual address
···
428
425
#define DRM_XE_QUERY_CONFIG_FLAG_HAS_LOW_LATENCY (1 << 1)
429
426
#define DRM_XE_QUERY_CONFIG_FLAG_HAS_CPU_ADDR_MIRROR (1 << 2)
430
427
#define DRM_XE_QUERY_CONFIG_FLAG_HAS_NO_COMPRESSION_HINT (1 << 3)
428
+
#define DRM_XE_QUERY_CONFIG_FLAG_HAS_DISABLE_STATE_CACHE_PERF_FIX (1 << 4)
431
429
#define DRM_XE_QUERY_CONFIG_MIN_ALIGNMENT 2
432
430
#define DRM_XE_QUERY_CONFIG_VA_BITS 3
433
431
#define DRM_XE_QUERY_CONFIG_MAX_EXEC_QUEUE_PRIORITY 4
···
1057
1053
* not invoke autoreset. Neither will stack variables going out of scope.
1058
1054
* Therefore it's recommended to always explicitly reset the madvises when
1059
1055
* freeing the memory backing a region used in a &DRM_IOCTL_XE_MADVISE call.
1056
+
* - DRM_XE_VM_BIND_FLAG_DECOMPRESS - Request on-device decompression for a MAP.
1057
+
* When set on a MAP bind operation, request the driver schedule an on-device
1058
+
* in-place decompression (via the migrate/resolve path) for the GPU mapping
1059
+
* created by this bind. Only valid for DRM_XE_VM_BIND_OP_MAP; usage on
1060
+
* other ops is rejected. The bind's pat_index must select the device's
1061
+
* "no-compression" PAT. Only meaningful for VRAM-backed BOs on devices that
1062
+
* support Flat CCS and the required HW generation XE2+.
1060
1063
*
1061
1064
* The @prefetch_mem_region_instance for %DRM_XE_VM_BIND_OP_PREFETCH can also be:
1062
1065
* - %DRM_XE_CONSULT_MEM_ADVISE_PREF_LOC, which ensures prefetching occurs in
···
1171
1160
#define DRM_XE_VM_BIND_FLAG_CHECK_PXP (1 << 4)
1172
1161
#define DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR (1 << 5)
1173
1162
#define DRM_XE_VM_BIND_FLAG_MADVISE_AUTORESET (1 << 6)
1163
+
#define DRM_XE_VM_BIND_FLAG_DECOMPRESS (1 << 7)
1174
1164
/** @flags: Bind flags */
1175
1165
__u32 flags;
1176
1166
···
1297
1285
* - %DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY - Set the queue
1298
1286
* priority within the multi-queue group. Current valid priority values are 0–2
1299
1287
* (default is 1), with higher values indicating higher priority.
1288
+
* - %DRM_XE_EXEC_QUEUE_SET_DISABLE_STATE_CACHE_PERF_FIX - Set the queue to
1289
+
* enable render color cache keying on BTP+BTI instead of just BTI
1290
+
* (only valid for render queues).
1300
1291
*
1301
1292
* The example below shows how to use @drm_xe_exec_queue_create to create
1302
1293
* a simple exec_queue (no parallel submission) of class
···
1344
1329
#define DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_GROUP 4
1345
1330
#define DRM_XE_MULTI_GROUP_CREATE (1ull << 63)
1346
1331
#define DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY 5
1332
+
#define DRM_XE_EXEC_QUEUE_SET_DISABLE_STATE_CACHE_PERF_FIX 6
1347
1333
/** @extensions: Pointer to the first extension struct, if any */
1348
1334
__u64 extensions;
1349
1335
···
2372
2356
/** @reserved: Reserved */
2373
2357
__u64 reserved[2];
2374
2358
};
2359
+
2360
+
/**
2361
+
* DOC: Xe DRM RAS
2362
+
*
2363
+
* The enums and strings defined below map to the attributes of the DRM RAS Netlink Interface.
2364
+
* Refer to Documentation/netlink/specs/drm_ras.yaml for complete interface specification.
2365
+
*
2366
+
* Node Registration
2367
+
* =================
2368
+
*
2369
+
* The driver registers DRM RAS nodes for each error severity level.
2370
+
* enum drm_xe_ras_error_severity defines the node-id, while DRM_XE_RAS_ERROR_SEVERITY_NAMES maps
2371
+
* node-id to node-name.
2372
+
*
2373
+
* Error Classification
2374
+
* ====================
2375
+
*
2376
+
* Each node contains a list of error counters. Each error is identified by a error-id and
2377
+
* an error-name. enum drm_xe_ras_error_component defines the error-id, while
2378
+
* DRM_XE_RAS_ERROR_COMPONENT_NAMES maps error-id to error-name.
2379
+
*
2380
+
* User Interface
2381
+
* ==============
2382
+
*
2383
+
* To retrieve error values of a error counter, userspace applications should
2384
+
* follow the below steps:
2385
+
*
2386
+
* 1. Use command LIST_NODES to enumerate all available nodes
2387
+
* 2. Select node by node-id or node-name
2388
+
* 3. Use command GET_ERROR_COUNTERS to list errors of specific node
2389
+
* 4. Query specific error values using either error-id or error-name
2390
+
*
2391
+
* .. code-block:: C
2392
+
*
2393
+
* // Lookup tables for ID-to-name resolution
2394
+
* static const char *nodes[] = DRM_XE_RAS_ERROR_SEVERITY_NAMES;
2395
+
* static const char *errors[] = DRM_XE_RAS_ERROR_COMPONENT_NAMES;
2396
+
*
2397
+
*/
2398
+
2399
+
/**
2400
+
* enum drm_xe_ras_error_severity - DRM RAS error severity.
2401
+
*/
2402
+
enum drm_xe_ras_error_severity {
2403
+
/** @DRM_XE_RAS_ERR_SEV_CORRECTABLE: Correctable Error */
2404
+
DRM_XE_RAS_ERR_SEV_CORRECTABLE = 0,
2405
+
/** @DRM_XE_RAS_ERR_SEV_UNCORRECTABLE: Uncorrectable Error */
2406
+
DRM_XE_RAS_ERR_SEV_UNCORRECTABLE,
2407
+
/** @DRM_XE_RAS_ERR_SEV_MAX: Max severity */
2408
+
DRM_XE_RAS_ERR_SEV_MAX /* non-ABI */
2409
+
};
2410
+
2411
+
/**
2412
+
* enum drm_xe_ras_error_component - DRM RAS error component.
2413
+
*/
2414
+
enum drm_xe_ras_error_component {
2415
+
/** @DRM_XE_RAS_ERR_COMP_CORE_COMPUTE: Core Compute Error */
2416
+
DRM_XE_RAS_ERR_COMP_CORE_COMPUTE = 1,
2417
+
/** @DRM_XE_RAS_ERR_COMP_SOC_INTERNAL: SoC Internal Error */
2418
+
DRM_XE_RAS_ERR_COMP_SOC_INTERNAL,
2419
+
/** @DRM_XE_RAS_ERR_COMP_MAX: Max Error */
2420
+
DRM_XE_RAS_ERR_COMP_MAX /* non-ABI */
2421
+
};
2422
+
2423
+
/*
2424
+
* Error severity to name mapping.
2425
+
*/
2426
+
#define DRM_XE_RAS_ERROR_SEVERITY_NAMES { \
2427
+
[DRM_XE_RAS_ERR_SEV_CORRECTABLE] = "correctable-errors", \
2428
+
[DRM_XE_RAS_ERR_SEV_UNCORRECTABLE] = "uncorrectable-errors", \
2429
+
}
2430
+
2431
+
/*
2432
+
* Error component to name mapping.
2433
+
*/
2434
+
#define DRM_XE_RAS_ERROR_COMPONENT_NAMES { \
2435
+
[DRM_XE_RAS_ERR_COMP_CORE_COMPUTE] = "core-compute", \
2436
+
[DRM_XE_RAS_ERR_COMP_SOC_INTERNAL] = "soc-internal" \
2437
+
}
2375
2438
2376
2439
#if defined(__cplusplus)
2377
2440
}
+56
-9
mm/mmu_notifier.c
+56
-9
mm/mmu_notifier.c
···
260
260
}
261
261
EXPORT_SYMBOL_GPL(mmu_interval_read_begin);
262
262
263
+
static void mn_itree_finish_pass(struct llist_head *finish_passes)
264
+
{
265
+
struct llist_node *first = llist_reverse_order(__llist_del_all(finish_passes));
266
+
struct mmu_interval_notifier_finish *f, *next;
267
+
268
+
llist_for_each_entry_safe(f, next, first, link)
269
+
f->notifier->ops->invalidate_finish(f);
270
+
}
271
+
263
272
static void mn_itree_release(struct mmu_notifier_subscriptions *subscriptions,
264
273
struct mm_struct *mm)
265
274
{
···
280
271
.end = ULONG_MAX,
281
272
};
282
273
struct mmu_interval_notifier *interval_sub;
274
+
LLIST_HEAD(finish_passes);
283
275
unsigned long cur_seq;
284
276
bool ret;
285
277
···
288
278
mn_itree_inv_start_range(subscriptions, &range, &cur_seq);
289
279
interval_sub;
290
280
interval_sub = mn_itree_inv_next(interval_sub, &range)) {
291
-
ret = interval_sub->ops->invalidate(interval_sub, &range,
292
-
cur_seq);
281
+
if (interval_sub->ops->invalidate_start) {
282
+
struct mmu_interval_notifier_finish *finish = NULL;
283
+
284
+
ret = interval_sub->ops->invalidate_start(interval_sub,
285
+
&range,
286
+
cur_seq,
287
+
&finish);
288
+
if (ret && finish) {
289
+
finish->notifier = interval_sub;
290
+
__llist_add(&finish->link, &finish_passes);
291
+
}
292
+
293
+
} else {
294
+
ret = interval_sub->ops->invalidate(interval_sub,
295
+
&range,
296
+
cur_seq);
297
+
}
293
298
WARN_ON(!ret);
294
299
}
295
300
301
+
mn_itree_finish_pass(&finish_passes);
296
302
mn_itree_inv_end(subscriptions);
297
303
}
298
304
···
456
430
const struct mmu_notifier_range *range)
457
431
{
458
432
struct mmu_interval_notifier *interval_sub;
433
+
LLIST_HEAD(finish_passes);
459
434
unsigned long cur_seq;
435
+
int err = 0;
460
436
461
437
for (interval_sub =
462
438
mn_itree_inv_start_range(subscriptions, range, &cur_seq);
···
466
438
interval_sub = mn_itree_inv_next(interval_sub, range)) {
467
439
bool ret;
468
440
469
-
ret = interval_sub->ops->invalidate(interval_sub, range,
470
-
cur_seq);
441
+
if (interval_sub->ops->invalidate_start) {
442
+
struct mmu_interval_notifier_finish *finish = NULL;
443
+
444
+
ret = interval_sub->ops->invalidate_start(interval_sub,
445
+
range,
446
+
cur_seq,
447
+
&finish);
448
+
if (ret && finish) {
449
+
finish->notifier = interval_sub;
450
+
__llist_add(&finish->link, &finish_passes);
451
+
}
452
+
453
+
} else {
454
+
ret = interval_sub->ops->invalidate(interval_sub,
455
+
range,
456
+
cur_seq);
457
+
}
471
458
if (!ret) {
472
459
if (WARN_ON(mmu_notifier_range_blockable(range)))
473
460
continue;
474
-
goto out_would_block;
461
+
err = -EAGAIN;
462
+
break;
475
463
}
476
464
}
477
-
return 0;
478
465
479
-
out_would_block:
466
+
mn_itree_finish_pass(&finish_passes);
467
+
480
468
/*
481
469
* On -EAGAIN the non-blocking caller is not allowed to call
482
470
* invalidate_range_end()
483
471
*/
484
-
mn_itree_inv_end(subscriptions);
485
-
return -EAGAIN;
472
+
if (err)
473
+
mn_itree_inv_end(subscriptions);
474
+
475
+
return err;
486
476
}
487
477
488
478
static int mn_hlist_invalidate_range_start(
···
1022
976
struct mmu_notifier_subscriptions *subscriptions;
1023
977
int ret;
1024
978
979
+
WARN_ON_ONCE(ops->invalidate_start && !ops->invalidate_finish);
1025
980
might_lock(&mm->mmap_lock);
1026
981
1027
982
subscriptions = smp_load_acquire(&mm->notifier_subscriptions);