Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
genpt: Generic Page Table base API
The generic API is intended to be separated from the implementation of
page table algorithms. It contains only accessors for walking and
manipulating the table and helpers that are useful for building an
implementation. Memory management is not in the generic API, but part of
the implementation.
Using a multi-compilation approach the implementation module would include
headers in this order:
common.h
defs_FMT.h
pt_defs.h
FMT.h
pt_common.h
IMPLEMENTATION.h
Where each compilation unit would have a combination of FMT and
IMPLEMENTATION to produce a per-format per-implementation module.
The API is designed so that the format headers have minimal logic, and
default implementations are provided if the format doesn't include one.
Generally formats provide their code via an inline function using the
pattern:
static inline FMTpt_XX(..) {}
#define pt_XX FMTpt_XX
The common code then enforces a function signature so that there is no
drift in function arguments, or accidental polymorphic functions (as has
been slightly troublesome in mm). Use of function-like #defines are
avoided in the format even though many of the functions are small enough.
Provide kdocs for the API surface.
This is enough to implement the 8 initial format variations with all of
their features:
* Entries comprised of contiguous blocks of IO PTEs for larger page
sizes (AMDv1, ARMv8)
* Multi-level tables, up to 6 levels. Runtime selected top level
* The size of the top table level can be selected at runtime (ARM's
concatenated tables)
* The number of levels in the table can optionally increase dynamically
during map (AMDv1)
* Optional leaf entries at any level
* 32 bit/64 bit virtual and output addresses, using every bit
* Sign extended addressing (x86)
* Dirty tracking
A basic simple format takes about 200 lines to declare the require inline
functions.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Tested-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
authored by
Jason Gunthorpe
and committed by
Joerg Roedel
7c5b184d
fd714986
+1836
+1
.clang-format
+1
.clang-format
+2
drivers/iommu/Kconfig
+2
drivers/iommu/Kconfig
+20
drivers/iommu/generic_pt/Kconfig
+20
drivers/iommu/generic_pt/Kconfig
···
1
+
# SPDX-License-Identifier: GPL-2.0-only
2
+
3
+
menuconfig GENERIC_PT
4
+
bool "Generic Radix Page Table"
5
+
help
6
+
Generic library for building radix tree page tables.
7
+
8
+
Generic PT provides a set of HW page table formats and a common
9
+
set of APIs to work with them.
10
+
11
+
if GENERIC_PT
12
+
config DEBUG_GENERIC_PT
13
+
bool "Extra debugging checks for GENERIC_PT"
14
+
help
15
+
Enable extra run time debugging checks for GENERIC_PT code. This
16
+
incurs a runtime cost and should not be enabled for production
17
+
kernels.
18
+
19
+
The kunit tests require this to be enabled to get full coverage.
20
+
endif
+358
drivers/iommu/generic_pt/pt_common.h
+358
drivers/iommu/generic_pt/pt_common.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*
5
+
* This header is included after the format. It contains definitions
6
+
* that build on the format definitions to create the basic format API.
7
+
*
8
+
* The format API is listed here, with kdocs. The functions without bodies are
9
+
* implemented in the format using the pattern:
10
+
* static inline FMTpt_XXX(..) {..}
11
+
* #define pt_XXX FMTpt_XXX
12
+
*
13
+
* If the format doesn't implement a function then pt_fmt_defaults.h can provide
14
+
* a generic version.
15
+
*
16
+
* The routines marked "@pts: Entry to query" operate on the entire contiguous
17
+
* entry and can be called with a pts->index pointing to any sub item that makes
18
+
* up that entry.
19
+
*
20
+
* The header order is:
21
+
* pt_defs.h
22
+
* FMT.h
23
+
* pt_common.h
24
+
*/
25
+
#ifndef __GENERIC_PT_PT_COMMON_H
26
+
#define __GENERIC_PT_PT_COMMON_H
27
+
28
+
#include "pt_defs.h"
29
+
#include "pt_fmt_defaults.h"
30
+
31
+
/**
32
+
* pt_attr_from_entry() - Convert the permission bits back to attrs
33
+
* @pts: Entry to convert from
34
+
* @attrs: Resulting attrs
35
+
*
36
+
* Fill in the attrs with the permission bits encoded in the current leaf entry.
37
+
* The attrs should be usable with pt_install_leaf_entry() to reconstruct the
38
+
* same entry.
39
+
*/
40
+
static inline void pt_attr_from_entry(const struct pt_state *pts,
41
+
struct pt_write_attrs *attrs);
42
+
43
+
/**
44
+
* pt_can_have_leaf() - True if the current level can have an OA entry
45
+
* @pts: The current level
46
+
*
47
+
* True if the current level can support pt_install_leaf_entry(). A leaf
48
+
* entry produce an OA.
49
+
*/
50
+
static inline bool pt_can_have_leaf(const struct pt_state *pts);
51
+
52
+
/**
53
+
* pt_can_have_table() - True if the current level can have a lower table
54
+
* @pts: The current level
55
+
*
56
+
* Every level except 0 is allowed to have a lower table.
57
+
*/
58
+
static inline bool pt_can_have_table(const struct pt_state *pts)
59
+
{
60
+
/* No further tables at level 0 */
61
+
return pts->level > 0;
62
+
}
63
+
64
+
/**
65
+
* pt_clear_entries() - Make entries empty (non-present)
66
+
* @pts: Starting table index
67
+
* @num_contig_lg2: Number of contiguous items to clear
68
+
*
69
+
* Clear a run of entries. A cleared entry will load back as PT_ENTRY_EMPTY
70
+
* and does not have any effect on table walking. The starting index must be
71
+
* aligned to num_contig_lg2.
72
+
*/
73
+
static inline void pt_clear_entries(struct pt_state *pts,
74
+
unsigned int num_contig_lg2);
75
+
76
+
/**
77
+
* pt_entry_make_write_dirty() - Make an entry dirty
78
+
* @pts: Table entry to change
79
+
*
80
+
* Make pt_entry_is_write_dirty() return true for this entry. This can be called
81
+
* asynchronously with any other table manipulation under a RCU lock and must
82
+
* not corrupt the table.
83
+
*/
84
+
static inline bool pt_entry_make_write_dirty(struct pt_state *pts);
85
+
86
+
/**
87
+
* pt_entry_make_write_clean() - Make the entry write clean
88
+
* @pts: Table entry to change
89
+
*
90
+
* Modify the entry so that pt_entry_is_write_dirty() == false. The HW will
91
+
* eventually be notified of this change via a TLB flush, which is the point
92
+
* that the HW must become synchronized. Any "write dirty" prior to the TLB
93
+
* flush can be lost, but once the TLB flush completes all writes must make
94
+
* their entries write dirty.
95
+
*
96
+
* The format should alter the entry in a way that is compatible with any
97
+
* concurrent update from HW. The entire contiguous entry is changed.
98
+
*/
99
+
static inline void pt_entry_make_write_clean(struct pt_state *pts);
100
+
101
+
/**
102
+
* pt_entry_is_write_dirty() - True if the entry has been written to
103
+
* @pts: Entry to query
104
+
*
105
+
* "write dirty" means that the HW has written to the OA translated
106
+
* by this entry. If the entry is contiguous then the consolidated
107
+
* "write dirty" for all the items must be returned.
108
+
*/
109
+
static inline bool pt_entry_is_write_dirty(const struct pt_state *pts);
110
+
111
+
/**
112
+
* pt_dirty_supported() - True if the page table supports dirty tracking
113
+
* @common: Page table to query
114
+
*/
115
+
static inline bool pt_dirty_supported(struct pt_common *common);
116
+
117
+
/**
118
+
* pt_entry_num_contig_lg2() - Number of contiguous items for this leaf entry
119
+
* @pts: Entry to query
120
+
*
121
+
* Return the number of contiguous items this leaf entry spans. If the entry
122
+
* is single item it returns ilog2(1).
123
+
*/
124
+
static inline unsigned int pt_entry_num_contig_lg2(const struct pt_state *pts);
125
+
126
+
/**
127
+
* pt_entry_oa() - Output Address for this leaf entry
128
+
* @pts: Entry to query
129
+
*
130
+
* Return the output address for the start of the entry. If the entry
131
+
* is contiguous this returns the same value for each sub-item. I.e.::
132
+
*
133
+
* log2_mod(pt_entry_oa(), pt_entry_oa_lg2sz()) == 0
134
+
*
135
+
* See pt_item_oa(). The format should implement one of these two functions
136
+
* depending on how it stores the OAs in the table.
137
+
*/
138
+
static inline pt_oaddr_t pt_entry_oa(const struct pt_state *pts);
139
+
140
+
/**
141
+
* pt_entry_oa_lg2sz() - Return the size of an OA entry
142
+
* @pts: Entry to query
143
+
*
144
+
* If the entry is not contiguous this returns pt_table_item_lg2sz(), otherwise
145
+
* it returns the total VA/OA size of the entire contiguous entry.
146
+
*/
147
+
static inline unsigned int pt_entry_oa_lg2sz(const struct pt_state *pts)
148
+
{
149
+
return pt_entry_num_contig_lg2(pts) + pt_table_item_lg2sz(pts);
150
+
}
151
+
152
+
/**
153
+
* pt_entry_oa_exact() - Return the complete OA for an entry
154
+
* @pts: Entry to query
155
+
*
156
+
* During iteration the first entry could have a VA with an offset from the
157
+
* natural start of the entry. Return the exact OA including the pts's VA
158
+
* offset.
159
+
*/
160
+
static inline pt_oaddr_t pt_entry_oa_exact(const struct pt_state *pts)
161
+
{
162
+
return _pt_entry_oa_fast(pts) |
163
+
log2_mod(pts->range->va, pt_entry_oa_lg2sz(pts));
164
+
}
165
+
166
+
/**
167
+
* pt_full_va_prefix() - The top bits of the VA
168
+
* @common: Page table to query
169
+
*
170
+
* This is usually 0, but some formats have their VA space going downward from
171
+
* PT_VADDR_MAX, and will return that instead. This value must always be
172
+
* adjusted by struct pt_common max_vasz_lg2.
173
+
*/
174
+
static inline pt_vaddr_t pt_full_va_prefix(const struct pt_common *common);
175
+
176
+
/**
177
+
* pt_has_system_page_size() - True if level 0 can install a PAGE_SHIFT entry
178
+
* @common: Page table to query
179
+
*
180
+
* If true the caller can use, at level 0, pt_install_leaf_entry(PAGE_SHIFT).
181
+
* This is useful to create optimized paths for common cases of PAGE_SIZE
182
+
* mappings.
183
+
*/
184
+
static inline bool pt_has_system_page_size(const struct pt_common *common);
185
+
186
+
/**
187
+
* pt_install_leaf_entry() - Write a leaf entry to the table
188
+
* @pts: Table index to change
189
+
* @oa: Output Address for this leaf
190
+
* @oasz_lg2: Size in VA/OA for this leaf
191
+
* @attrs: Attributes to modify the entry
192
+
*
193
+
* A leaf OA entry will return PT_ENTRY_OA from pt_load_entry(). It translates
194
+
* the VA indicated by pts to the given OA.
195
+
*
196
+
* For a single item non-contiguous entry oasz_lg2 is pt_table_item_lg2sz().
197
+
* For contiguous it is pt_table_item_lg2sz() + num_contig_lg2.
198
+
*
199
+
* This must not be called if pt_can_have_leaf() == false. Contiguous sizes
200
+
* not indicated by pt_possible_sizes() must not be specified.
201
+
*/
202
+
static inline void pt_install_leaf_entry(struct pt_state *pts, pt_oaddr_t oa,
203
+
unsigned int oasz_lg2,
204
+
const struct pt_write_attrs *attrs);
205
+
206
+
/**
207
+
* pt_install_table() - Write a table entry to the table
208
+
* @pts: Table index to change
209
+
* @table_pa: CPU physical address of the lower table's memory
210
+
* @attrs: Attributes to modify the table index
211
+
*
212
+
* A table entry will return PT_ENTRY_TABLE from pt_load_entry(). The table_pa
213
+
* is the table at pts->level - 1. This is done by cmpxchg so pts must have the
214
+
* current entry loaded. The pts is updated with the installed entry.
215
+
*
216
+
* This must not be called if pt_can_have_table() == false.
217
+
*
218
+
* Returns: true if the table was installed successfully.
219
+
*/
220
+
static inline bool pt_install_table(struct pt_state *pts, pt_oaddr_t table_pa,
221
+
const struct pt_write_attrs *attrs);
222
+
223
+
/**
224
+
* pt_item_oa() - Output Address for this leaf item
225
+
* @pts: Item to query
226
+
*
227
+
* Return the output address for this item. If the item is part of a contiguous
228
+
* entry it returns the value of the OA for this individual sub item.
229
+
*
230
+
* See pt_entry_oa(). The format should implement one of these two functions
231
+
* depending on how it stores the OA's in the table.
232
+
*/
233
+
static inline pt_oaddr_t pt_item_oa(const struct pt_state *pts);
234
+
235
+
/**
236
+
* pt_load_entry_raw() - Read from the location pts points at into the pts
237
+
* @pts: Table index to load
238
+
*
239
+
* Return the type of entry that was loaded. pts->entry will be filled in with
240
+
* the entry's content. See pt_load_entry()
241
+
*/
242
+
static inline enum pt_entry_type pt_load_entry_raw(struct pt_state *pts);
243
+
244
+
/**
245
+
* pt_max_oa_lg2() - Return the maximum OA the table format can hold
246
+
* @common: Page table to query
247
+
*
248
+
* The value oalog2_to_max_int(pt_max_oa_lg2()) is the MAX for the
249
+
* OA. This is the absolute maximum address the table can hold. struct pt_common
250
+
* max_oasz_lg2 sets a lower dynamic maximum based on HW capability.
251
+
*/
252
+
static inline unsigned int
253
+
pt_max_oa_lg2(const struct pt_common *common);
254
+
255
+
/**
256
+
* pt_num_items_lg2() - Return the number of items in this table level
257
+
* @pts: The current level
258
+
*
259
+
* The number of items in a table level defines the number of bits this level
260
+
* decodes from the VA. This function is not called for the top level,
261
+
* so it does not need to compute a special value for the top case. The
262
+
* result for the top is based on pt_common max_vasz_lg2.
263
+
*
264
+
* The value is used as part of determining the table indexes via the
265
+
* equation::
266
+
*
267
+
* log2_mod(log2_div(VA, pt_table_item_lg2sz()), pt_num_items_lg2())
268
+
*/
269
+
static inline unsigned int pt_num_items_lg2(const struct pt_state *pts);
270
+
271
+
/**
272
+
* pt_pgsz_lg2_to_level - Return the level that maps the page size
273
+
* @common: Page table to query
274
+
* @pgsize_lg2: Log2 page size
275
+
*
276
+
* Returns the table level that will map the given page size. The page
277
+
* size must be part of the pt_possible_sizes() for some level.
278
+
*/
279
+
static inline unsigned int pt_pgsz_lg2_to_level(struct pt_common *common,
280
+
unsigned int pgsize_lg2);
281
+
282
+
/**
283
+
* pt_possible_sizes() - Return a bitmap of possible output sizes at this level
284
+
* @pts: The current level
285
+
*
286
+
* Each level has a list of possible output sizes that can be installed as
287
+
* leaf entries. If pt_can_have_leaf() is false returns zero.
288
+
*
289
+
* Otherwise the bit in position pt_table_item_lg2sz() should be set indicating
290
+
* that a non-contiguous single item leaf entry is supported. The following
291
+
* pt_num_items_lg2() number of bits can be set indicating contiguous entries
292
+
* are supported. Bit pt_table_item_lg2sz() + pt_num_items_lg2() must not be
293
+
* set, contiguous entries cannot span the entire table.
294
+
*
295
+
* The OR of pt_possible_sizes() of all levels is the typical bitmask of all
296
+
* supported sizes in the entire table.
297
+
*/
298
+
static inline pt_vaddr_t pt_possible_sizes(const struct pt_state *pts);
299
+
300
+
/**
301
+
* pt_table_item_lg2sz() - Size of a single item entry in this table level
302
+
* @pts: The current level
303
+
*
304
+
* The size of the item specifies how much VA and OA a single item occupies.
305
+
*
306
+
* See pt_entry_oa_lg2sz() for the same value including the effect of contiguous
307
+
* entries.
308
+
*/
309
+
static inline unsigned int pt_table_item_lg2sz(const struct pt_state *pts);
310
+
311
+
/**
312
+
* pt_table_oa_lg2sz() - Return the VA/OA size of the entire table
313
+
* @pts: The current level
314
+
*
315
+
* Return the size of VA decoded by the entire table level.
316
+
*/
317
+
static inline unsigned int pt_table_oa_lg2sz(const struct pt_state *pts)
318
+
{
319
+
if (pts->range->top_level == pts->level)
320
+
return pts->range->max_vasz_lg2;
321
+
return min_t(unsigned int, pts->range->common->max_vasz_lg2,
322
+
pt_num_items_lg2(pts) + pt_table_item_lg2sz(pts));
323
+
}
324
+
325
+
/**
326
+
* pt_table_pa() - Return the CPU physical address of the table entry
327
+
* @pts: Entry to query
328
+
*
329
+
* This is only ever called on PT_ENTRY_TABLE entries. Must return the same
330
+
* value passed to pt_install_table().
331
+
*/
332
+
static inline pt_oaddr_t pt_table_pa(const struct pt_state *pts);
333
+
334
+
/**
335
+
* pt_table_ptr() - Return a CPU pointer for a table item
336
+
* @pts: Entry to query
337
+
*
338
+
* Same as pt_table_pa() but returns a CPU pointer.
339
+
*/
340
+
static inline struct pt_table_p *pt_table_ptr(const struct pt_state *pts)
341
+
{
342
+
return __va(pt_table_pa(pts));
343
+
}
344
+
345
+
/**
346
+
* pt_load_entry() - Read from the location pts points at into the pts
347
+
* @pts: Table index to load
348
+
*
349
+
* Set the type of entry that was loaded. pts->entry and pts->table_lower
350
+
* will be filled in with the entry's content.
351
+
*/
352
+
static inline void pt_load_entry(struct pt_state *pts)
353
+
{
354
+
pts->type = pt_load_entry_raw(pts);
355
+
if (pts->type == PT_ENTRY_TABLE)
356
+
pts->table_lower = pt_table_ptr(pts);
357
+
}
358
+
#endif
+329
drivers/iommu/generic_pt/pt_defs.h
+329
drivers/iommu/generic_pt/pt_defs.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*
5
+
* This header is included before the format. It contains definitions
6
+
* that are required to compile the format. The header order is:
7
+
* pt_defs.h
8
+
* fmt_XX.h
9
+
* pt_common.h
10
+
*/
11
+
#ifndef __GENERIC_PT_DEFS_H
12
+
#define __GENERIC_PT_DEFS_H
13
+
14
+
#include <linux/generic_pt/common.h>
15
+
16
+
#include <linux/types.h>
17
+
#include <linux/atomic.h>
18
+
#include <linux/bits.h>
19
+
#include <linux/limits.h>
20
+
#include <linux/bug.h>
21
+
#include <linux/kconfig.h>
22
+
#include "pt_log2.h"
23
+
24
+
/* Header self-compile default defines */
25
+
#ifndef pt_write_attrs
26
+
typedef u64 pt_vaddr_t;
27
+
typedef u64 pt_oaddr_t;
28
+
#endif
29
+
30
+
struct pt_table_p;
31
+
32
+
enum {
33
+
PT_VADDR_MAX = sizeof(pt_vaddr_t) == 8 ? U64_MAX : U32_MAX,
34
+
PT_VADDR_MAX_LG2 = sizeof(pt_vaddr_t) == 8 ? 64 : 32,
35
+
PT_OADDR_MAX = sizeof(pt_oaddr_t) == 8 ? U64_MAX : U32_MAX,
36
+
PT_OADDR_MAX_LG2 = sizeof(pt_oaddr_t) == 8 ? 64 : 32,
37
+
};
38
+
39
+
/*
40
+
* The format instantiation can have features wired off or on to optimize the
41
+
* code gen. Supported features are just a reflection of what the current set of
42
+
* kernel users want to use.
43
+
*/
44
+
#ifndef PT_SUPPORTED_FEATURES
45
+
#define PT_SUPPORTED_FEATURES 0
46
+
#endif
47
+
48
+
/*
49
+
* When in debug mode we compile all formats with all features. This allows the
50
+
* kunit to test the full matrix. SIGN_EXTEND can't co-exist with DYNAMIC_TOP or
51
+
* FULL_VA.
52
+
*/
53
+
#if IS_ENABLED(CONFIG_DEBUG_GENERIC_PT)
54
+
enum {
55
+
PT_ORIG_SUPPORTED_FEATURES = PT_SUPPORTED_FEATURES,
56
+
PT_DEBUG_SUPPORTED_FEATURES =
57
+
UINT_MAX &
58
+
~((PT_ORIG_SUPPORTED_FEATURES & BIT(PT_FEAT_SIGN_EXTEND)) ?
59
+
BIT(PT_FEAT_DYNAMIC_TOP) | BIT(PT_FEAT_FULL_VA) :
60
+
BIT(PT_FEAT_SIGN_EXTEND)),
61
+
};
62
+
#undef PT_SUPPORTED_FEATURES
63
+
#define PT_SUPPORTED_FEATURES PT_DEBUG_SUPPORTED_FEATURES
64
+
#endif
65
+
66
+
#ifndef PT_FORCE_ENABLED_FEATURES
67
+
#define PT_FORCE_ENABLED_FEATURES 0
68
+
#endif
69
+
70
+
/**
71
+
* DOC: Generic Page Table Language
72
+
*
73
+
* Language used in Generic Page Table
74
+
* VA
75
+
* The input address to the page table, often the virtual address.
76
+
* OA
77
+
* The output address from the page table, often the physical address.
78
+
* leaf
79
+
* An entry that results in an output address.
80
+
* start/end
81
+
* An half-open range, e.g. [0,0) refers to no VA.
82
+
* start/last
83
+
* An inclusive closed range, e.g. [0,0] refers to the VA 0
84
+
* common
85
+
* The generic page table container struct pt_common
86
+
* level
87
+
* Level 0 is always a table of only leaves with no futher table pointers.
88
+
* Increasing levels increase the size of the table items. The least
89
+
* significant VA bits used to index page tables are used to index the Level
90
+
* 0 table. The various labels for table levels used by HW descriptions are
91
+
* not used.
92
+
* top_level
93
+
* The inclusive highest level of the table. A two-level table
94
+
* has a top level of 1.
95
+
* table
96
+
* A linear array of translation items for that level.
97
+
* index
98
+
* The position in a table of an element: item = table[index]
99
+
* item
100
+
* A single index in a table
101
+
* entry
102
+
* A single logical element in a table. If contiguous pages are not
103
+
* supported then item and entry are the same thing, otherwise entry refers
104
+
* to all the items that comprise a single contiguous translation.
105
+
* item/entry_size
106
+
* The number of bytes of VA the table index translates for.
107
+
* If the item is a table entry then the next table covers
108
+
* this size. If the entry translates to an output address then the
109
+
* full OA is: OA | (VA % entry_size)
110
+
* contig_count
111
+
* The number of consecutive items fused into a single entry.
112
+
* item_size * contig_count is the size of that entry's translation.
113
+
* lg2
114
+
* Indicates the value is encoded as log2, i.e. 1<<x is the actual value.
115
+
* Normally the compiler is fine to optimize divide and mod with log2 values
116
+
* automatically when inlining, however if the values are not constant
117
+
* expressions it can't. So we do it by hand; we want to avoid 64-bit
118
+
* divmod.
119
+
*/
120
+
121
+
/* Returned by pt_load_entry() and for_each_pt_level_entry() */
122
+
enum pt_entry_type {
123
+
PT_ENTRY_EMPTY,
124
+
/* Entry is valid and points to a lower table level */
125
+
PT_ENTRY_TABLE,
126
+
/* Entry is valid and returns an output address */
127
+
PT_ENTRY_OA,
128
+
};
129
+
130
+
struct pt_range {
131
+
struct pt_common *common;
132
+
struct pt_table_p *top_table;
133
+
pt_vaddr_t va;
134
+
pt_vaddr_t last_va;
135
+
u8 top_level;
136
+
u8 max_vasz_lg2;
137
+
};
138
+
139
+
/*
140
+
* Similar to xa_state, this records information about an in-progress parse at a
141
+
* single level.
142
+
*/
143
+
struct pt_state {
144
+
struct pt_range *range;
145
+
struct pt_table_p *table;
146
+
struct pt_table_p *table_lower;
147
+
u64 entry;
148
+
enum pt_entry_type type;
149
+
unsigned short index;
150
+
unsigned short end_index;
151
+
u8 level;
152
+
};
153
+
154
+
#define pt_cur_table(pts, type) ((type *)((pts)->table))
155
+
156
+
/*
157
+
* Try to install a new table pointer. The locking methodology requires this to
158
+
* be atomic (multiple threads can race to install a pointer). The losing
159
+
* threads will fail the atomic and return false. They should free any memory
160
+
* and reparse the table level again.
161
+
*/
162
+
#if !IS_ENABLED(CONFIG_GENERIC_ATOMIC64)
163
+
static inline bool pt_table_install64(struct pt_state *pts, u64 table_entry)
164
+
{
165
+
u64 *entryp = pt_cur_table(pts, u64) + pts->index;
166
+
u64 old_entry = pts->entry;
167
+
bool ret;
168
+
169
+
/*
170
+
* Ensure the zero'd table content itself is visible before its PTE can
171
+
* be. release is a NOP on !SMP, but the HW is still doing an acquire.
172
+
*/
173
+
if (!IS_ENABLED(CONFIG_SMP))
174
+
dma_wmb();
175
+
ret = try_cmpxchg64_release(entryp, &old_entry, table_entry);
176
+
if (ret)
177
+
pts->entry = table_entry;
178
+
return ret;
179
+
}
180
+
#endif
181
+
182
+
static inline bool pt_table_install32(struct pt_state *pts, u32 table_entry)
183
+
{
184
+
u32 *entryp = pt_cur_table(pts, u32) + pts->index;
185
+
u32 old_entry = pts->entry;
186
+
bool ret;
187
+
188
+
/*
189
+
* Ensure the zero'd table content itself is visible before its PTE can
190
+
* be. release is a NOP on !SMP, but the HW is still doing an acquire.
191
+
*/
192
+
if (!IS_ENABLED(CONFIG_SMP))
193
+
dma_wmb();
194
+
ret = try_cmpxchg_release(entryp, &old_entry, table_entry);
195
+
if (ret)
196
+
pts->entry = table_entry;
197
+
return ret;
198
+
}
199
+
200
+
#define PT_SUPPORTED_FEATURE(feature_nr) (PT_SUPPORTED_FEATURES & BIT(feature_nr))
201
+
202
+
static inline bool pt_feature(const struct pt_common *common,
203
+
unsigned int feature_nr)
204
+
{
205
+
if (PT_FORCE_ENABLED_FEATURES & BIT(feature_nr))
206
+
return true;
207
+
if (!PT_SUPPORTED_FEATURE(feature_nr))
208
+
return false;
209
+
return common->features & BIT(feature_nr);
210
+
}
211
+
212
+
static inline bool pts_feature(const struct pt_state *pts,
213
+
unsigned int feature_nr)
214
+
{
215
+
return pt_feature(pts->range->common, feature_nr);
216
+
}
217
+
218
+
/*
219
+
* PT_WARN_ON is used for invariants that the kunit should be checking can't
220
+
* happen.
221
+
*/
222
+
#if IS_ENABLED(CONFIG_DEBUG_GENERIC_PT)
223
+
#define PT_WARN_ON WARN_ON
224
+
#else
225
+
static inline bool PT_WARN_ON(bool condition)
226
+
{
227
+
return false;
228
+
}
229
+
#endif
230
+
231
+
/* These all work on the VA type */
232
+
#define log2_to_int(a_lg2) log2_to_int_t(pt_vaddr_t, a_lg2)
233
+
#define log2_to_max_int(a_lg2) log2_to_max_int_t(pt_vaddr_t, a_lg2)
234
+
#define log2_div(a, b_lg2) log2_div_t(pt_vaddr_t, a, b_lg2)
235
+
#define log2_div_eq(a, b, c_lg2) log2_div_eq_t(pt_vaddr_t, a, b, c_lg2)
236
+
#define log2_mod(a, b_lg2) log2_mod_t(pt_vaddr_t, a, b_lg2)
237
+
#define log2_mod_eq_max(a, b_lg2) log2_mod_eq_max_t(pt_vaddr_t, a, b_lg2)
238
+
#define log2_set_mod(a, val, b_lg2) log2_set_mod_t(pt_vaddr_t, a, val, b_lg2)
239
+
#define log2_set_mod_max(a, b_lg2) log2_set_mod_max_t(pt_vaddr_t, a, b_lg2)
240
+
#define log2_mul(a, b_lg2) log2_mul_t(pt_vaddr_t, a, b_lg2)
241
+
#define vaffs(a) ffs_t(pt_vaddr_t, a)
242
+
#define vafls(a) fls_t(pt_vaddr_t, a)
243
+
#define vaffz(a) ffz_t(pt_vaddr_t, a)
244
+
245
+
/*
246
+
* The full VA (fva) versions permit the lg2 value to be == PT_VADDR_MAX_LG2 and
247
+
* generate a useful defined result. The non-fva versions will malfunction at
248
+
* this extreme.
249
+
*/
250
+
static inline pt_vaddr_t fvalog2_div(pt_vaddr_t a, unsigned int b_lg2)
251
+
{
252
+
if (PT_SUPPORTED_FEATURE(PT_FEAT_FULL_VA) && b_lg2 == PT_VADDR_MAX_LG2)
253
+
return 0;
254
+
return log2_div_t(pt_vaddr_t, a, b_lg2);
255
+
}
256
+
257
+
static inline pt_vaddr_t fvalog2_mod(pt_vaddr_t a, unsigned int b_lg2)
258
+
{
259
+
if (PT_SUPPORTED_FEATURE(PT_FEAT_FULL_VA) && b_lg2 == PT_VADDR_MAX_LG2)
260
+
return a;
261
+
return log2_mod_t(pt_vaddr_t, a, b_lg2);
262
+
}
263
+
264
+
static inline bool fvalog2_div_eq(pt_vaddr_t a, pt_vaddr_t b,
265
+
unsigned int c_lg2)
266
+
{
267
+
if (PT_SUPPORTED_FEATURE(PT_FEAT_FULL_VA) && c_lg2 == PT_VADDR_MAX_LG2)
268
+
return true;
269
+
return log2_div_eq_t(pt_vaddr_t, a, b, c_lg2);
270
+
}
271
+
272
+
static inline pt_vaddr_t fvalog2_set_mod(pt_vaddr_t a, pt_vaddr_t val,
273
+
unsigned int b_lg2)
274
+
{
275
+
if (PT_SUPPORTED_FEATURE(PT_FEAT_FULL_VA) && b_lg2 == PT_VADDR_MAX_LG2)
276
+
return val;
277
+
return log2_set_mod_t(pt_vaddr_t, a, val, b_lg2);
278
+
}
279
+
280
+
static inline pt_vaddr_t fvalog2_set_mod_max(pt_vaddr_t a, unsigned int b_lg2)
281
+
{
282
+
if (PT_SUPPORTED_FEATURE(PT_FEAT_FULL_VA) && b_lg2 == PT_VADDR_MAX_LG2)
283
+
return PT_VADDR_MAX;
284
+
return log2_set_mod_max_t(pt_vaddr_t, a, b_lg2);
285
+
}
286
+
287
+
/* These all work on the OA type */
288
+
#define oalog2_to_int(a_lg2) log2_to_int_t(pt_oaddr_t, a_lg2)
289
+
#define oalog2_to_max_int(a_lg2) log2_to_max_int_t(pt_oaddr_t, a_lg2)
290
+
#define oalog2_div(a, b_lg2) log2_div_t(pt_oaddr_t, a, b_lg2)
291
+
#define oalog2_div_eq(a, b, c_lg2) log2_div_eq_t(pt_oaddr_t, a, b, c_lg2)
292
+
#define oalog2_mod(a, b_lg2) log2_mod_t(pt_oaddr_t, a, b_lg2)
293
+
#define oalog2_mod_eq_max(a, b_lg2) log2_mod_eq_max_t(pt_oaddr_t, a, b_lg2)
294
+
#define oalog2_set_mod(a, val, b_lg2) log2_set_mod_t(pt_oaddr_t, a, val, b_lg2)
295
+
#define oalog2_set_mod_max(a, b_lg2) log2_set_mod_max_t(pt_oaddr_t, a, b_lg2)
296
+
#define oalog2_mul(a, b_lg2) log2_mul_t(pt_oaddr_t, a, b_lg2)
297
+
#define oaffs(a) ffs_t(pt_oaddr_t, a)
298
+
#define oafls(a) fls_t(pt_oaddr_t, a)
299
+
#define oaffz(a) ffz_t(pt_oaddr_t, a)
300
+
301
+
static inline uintptr_t _pt_top_set(struct pt_table_p *table_mem,
302
+
unsigned int top_level)
303
+
{
304
+
return top_level | (uintptr_t)table_mem;
305
+
}
306
+
307
+
static inline void pt_top_set(struct pt_common *common,
308
+
struct pt_table_p *table_mem,
309
+
unsigned int top_level)
310
+
{
311
+
WRITE_ONCE(common->top_of_table, _pt_top_set(table_mem, top_level));
312
+
}
313
+
314
+
static inline void pt_top_set_level(struct pt_common *common,
315
+
unsigned int top_level)
316
+
{
317
+
pt_top_set(common, NULL, top_level);
318
+
}
319
+
320
+
static inline unsigned int pt_top_get_level(const struct pt_common *common)
321
+
{
322
+
return READ_ONCE(common->top_of_table) % (1 << PT_TOP_LEVEL_BITS);
323
+
}
324
+
325
+
static inline bool pt_check_install_leaf_args(struct pt_state *pts,
326
+
pt_oaddr_t oa,
327
+
unsigned int oasz_lg2);
328
+
329
+
#endif
+233
drivers/iommu/generic_pt/pt_fmt_defaults.h
+233
drivers/iommu/generic_pt/pt_fmt_defaults.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*
5
+
* Default definitions for formats that don't define these functions.
6
+
*/
7
+
#ifndef __GENERIC_PT_PT_FMT_DEFAULTS_H
8
+
#define __GENERIC_PT_PT_FMT_DEFAULTS_H
9
+
10
+
#include "pt_defs.h"
11
+
#include <linux/log2.h>
12
+
13
+
/* Header self-compile default defines */
14
+
#ifndef pt_load_entry_raw
15
+
#include "fmt/amdv1.h"
16
+
#endif
17
+
18
+
/*
19
+
* The format must provide PT_GRANULE_LG2SZ, PT_TABLEMEM_LG2SZ, and
20
+
* PT_ITEM_WORD_SIZE. They must be the same at every level excluding the top.
21
+
*/
22
+
#ifndef pt_table_item_lg2sz
23
+
static inline unsigned int pt_table_item_lg2sz(const struct pt_state *pts)
24
+
{
25
+
return PT_GRANULE_LG2SZ +
26
+
(PT_TABLEMEM_LG2SZ - ilog2(PT_ITEM_WORD_SIZE)) * pts->level;
27
+
}
28
+
#endif
29
+
30
+
#ifndef pt_pgsz_lg2_to_level
31
+
static inline unsigned int pt_pgsz_lg2_to_level(struct pt_common *common,
32
+
unsigned int pgsize_lg2)
33
+
{
34
+
return ((unsigned int)(pgsize_lg2 - PT_GRANULE_LG2SZ)) /
35
+
(PT_TABLEMEM_LG2SZ - ilog2(PT_ITEM_WORD_SIZE));
36
+
}
37
+
#endif
38
+
39
+
/*
40
+
* If not supplied by the format then contiguous pages are not supported.
41
+
*
42
+
* If contiguous pages are supported then the format must also provide
43
+
* pt_contig_count_lg2() if it supports a single contiguous size per level,
44
+
* or pt_possible_sizes() if it supports multiple sizes per level.
45
+
*/
46
+
#ifndef pt_entry_num_contig_lg2
47
+
static inline unsigned int pt_entry_num_contig_lg2(const struct pt_state *pts)
48
+
{
49
+
return ilog2(1);
50
+
}
51
+
52
+
/*
53
+
* Return the number of contiguous OA items forming an entry at this table level
54
+
*/
55
+
static inline unsigned short pt_contig_count_lg2(const struct pt_state *pts)
56
+
{
57
+
return ilog2(1);
58
+
}
59
+
#endif
60
+
61
+
/* If not supplied by the format then dirty tracking is not supported */
62
+
#ifndef pt_entry_is_write_dirty
63
+
static inline bool pt_entry_is_write_dirty(const struct pt_state *pts)
64
+
{
65
+
return false;
66
+
}
67
+
68
+
static inline void pt_entry_make_write_clean(struct pt_state *pts)
69
+
{
70
+
}
71
+
72
+
static inline bool pt_dirty_supported(struct pt_common *common)
73
+
{
74
+
return false;
75
+
}
76
+
#else
77
+
/* If not supplied then dirty tracking is always enabled */
78
+
#ifndef pt_dirty_supported
79
+
static inline bool pt_dirty_supported(struct pt_common *common)
80
+
{
81
+
return true;
82
+
}
83
+
#endif
84
+
#endif
85
+
86
+
#ifndef pt_entry_make_write_dirty
87
+
static inline bool pt_entry_make_write_dirty(struct pt_state *pts)
88
+
{
89
+
return false;
90
+
}
91
+
#endif
92
+
93
+
/*
94
+
* Format supplies either:
95
+
* pt_entry_oa - OA is at the start of a contiguous entry
96
+
* or
97
+
* pt_item_oa - OA is adjusted for every item in a contiguous entry
98
+
*
99
+
* Build the missing one
100
+
*
101
+
* The internal helper _pt_entry_oa_fast() allows generating
102
+
* an efficient pt_entry_oa_exact(), it doesn't care which
103
+
* option is selected.
104
+
*/
105
+
#ifdef pt_entry_oa
106
+
static inline pt_oaddr_t pt_item_oa(const struct pt_state *pts)
107
+
{
108
+
return pt_entry_oa(pts) |
109
+
log2_mul(pts->index, pt_table_item_lg2sz(pts));
110
+
}
111
+
#define _pt_entry_oa_fast pt_entry_oa
112
+
#endif
113
+
114
+
#ifdef pt_item_oa
115
+
static inline pt_oaddr_t pt_entry_oa(const struct pt_state *pts)
116
+
{
117
+
return log2_set_mod(pt_item_oa(pts), 0,
118
+
pt_entry_num_contig_lg2(pts) +
119
+
pt_table_item_lg2sz(pts));
120
+
}
121
+
#define _pt_entry_oa_fast pt_item_oa
122
+
#endif
123
+
124
+
/*
125
+
* If not supplied by the format then use the constant
126
+
* PT_MAX_OUTPUT_ADDRESS_LG2.
127
+
*/
128
+
#ifndef pt_max_oa_lg2
129
+
static inline unsigned int
130
+
pt_max_oa_lg2(const struct pt_common *common)
131
+
{
132
+
return PT_MAX_OUTPUT_ADDRESS_LG2;
133
+
}
134
+
#endif
135
+
136
+
#ifndef pt_has_system_page_size
137
+
static inline bool pt_has_system_page_size(const struct pt_common *common)
138
+
{
139
+
return PT_GRANULE_LG2SZ == PAGE_SHIFT;
140
+
}
141
+
#endif
142
+
143
+
/*
144
+
* If not supplied by the format then assume only one contiguous size determined
145
+
* by pt_contig_count_lg2()
146
+
*/
147
+
#ifndef pt_possible_sizes
148
+
static inline unsigned short pt_contig_count_lg2(const struct pt_state *pts);
149
+
150
+
/* Return a bitmap of possible leaf page sizes at this level */
151
+
static inline pt_vaddr_t pt_possible_sizes(const struct pt_state *pts)
152
+
{
153
+
unsigned int isz_lg2 = pt_table_item_lg2sz(pts);
154
+
155
+
if (!pt_can_have_leaf(pts))
156
+
return 0;
157
+
return log2_to_int(isz_lg2) |
158
+
log2_to_int(pt_contig_count_lg2(pts) + isz_lg2);
159
+
}
160
+
#endif
161
+
162
+
/* If not supplied by the format then use 0. */
163
+
#ifndef pt_full_va_prefix
164
+
static inline pt_vaddr_t pt_full_va_prefix(const struct pt_common *common)
165
+
{
166
+
return 0;
167
+
}
168
+
#endif
169
+
170
+
/* If not supplied by the format then zero fill using PT_ITEM_WORD_SIZE */
171
+
#ifndef pt_clear_entries
172
+
static inline void pt_clear_entries64(struct pt_state *pts,
173
+
unsigned int num_contig_lg2)
174
+
{
175
+
u64 *tablep = pt_cur_table(pts, u64) + pts->index;
176
+
u64 *end = tablep + log2_to_int(num_contig_lg2);
177
+
178
+
PT_WARN_ON(log2_mod(pts->index, num_contig_lg2));
179
+
for (; tablep != end; tablep++)
180
+
WRITE_ONCE(*tablep, 0);
181
+
}
182
+
183
+
static inline void pt_clear_entries32(struct pt_state *pts,
184
+
unsigned int num_contig_lg2)
185
+
{
186
+
u32 *tablep = pt_cur_table(pts, u32) + pts->index;
187
+
u32 *end = tablep + log2_to_int(num_contig_lg2);
188
+
189
+
PT_WARN_ON(log2_mod(pts->index, num_contig_lg2));
190
+
for (; tablep != end; tablep++)
191
+
WRITE_ONCE(*tablep, 0);
192
+
}
193
+
194
+
static inline void pt_clear_entries(struct pt_state *pts,
195
+
unsigned int num_contig_lg2)
196
+
{
197
+
if (PT_ITEM_WORD_SIZE == sizeof(u32))
198
+
pt_clear_entries32(pts, num_contig_lg2);
199
+
else
200
+
pt_clear_entries64(pts, num_contig_lg2);
201
+
}
202
+
#define pt_clear_entries pt_clear_entries
203
+
#endif
204
+
205
+
/*
206
+
* Format can call in the pt_install_leaf_entry() to check the arguments are all
207
+
* aligned correctly.
208
+
*/
209
+
static inline bool pt_check_install_leaf_args(struct pt_state *pts,
210
+
pt_oaddr_t oa,
211
+
unsigned int oasz_lg2)
212
+
{
213
+
unsigned int isz_lg2 = pt_table_item_lg2sz(pts);
214
+
215
+
if (PT_WARN_ON(oalog2_mod(oa, oasz_lg2)))
216
+
return false;
217
+
218
+
#ifdef pt_possible_sizes
219
+
if (PT_WARN_ON(isz_lg2 > oasz_lg2 ||
220
+
oasz_lg2 > isz_lg2 + pt_num_items_lg2(pts)))
221
+
return false;
222
+
#else
223
+
if (PT_WARN_ON(oasz_lg2 != isz_lg2 &&
224
+
oasz_lg2 != isz_lg2 + pt_contig_count_lg2(pts)))
225
+
return false;
226
+
#endif
227
+
228
+
if (PT_WARN_ON(oalog2_mod(pts->index, oasz_lg2 - isz_lg2)))
229
+
return false;
230
+
return true;
231
+
}
232
+
233
+
#endif
+636
drivers/iommu/generic_pt/pt_iter.h
+636
drivers/iommu/generic_pt/pt_iter.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*
5
+
* Iterators for Generic Page Table
6
+
*/
7
+
#ifndef __GENERIC_PT_PT_ITER_H
8
+
#define __GENERIC_PT_PT_ITER_H
9
+
10
+
#include "pt_common.h"
11
+
12
+
#include <linux/errno.h>
13
+
14
+
/*
15
+
* Use to mangle symbols so that backtraces and the symbol table are
16
+
* understandable. Any non-inlined function should get mangled like this.
17
+
*/
18
+
#define NS(fn) CONCATENATE(PTPFX, fn)
19
+
20
+
/**
21
+
* pt_check_range() - Validate the range can be iterated
22
+
* @range: Range to validate
23
+
*
24
+
* Check that VA and last_va fall within the permitted range of VAs. If the
25
+
* format is using PT_FEAT_SIGN_EXTEND then this also checks the sign extension
26
+
* is correct.
27
+
*/
28
+
static inline int pt_check_range(struct pt_range *range)
29
+
{
30
+
pt_vaddr_t prefix;
31
+
32
+
PT_WARN_ON(!range->max_vasz_lg2);
33
+
34
+
if (pt_feature(range->common, PT_FEAT_SIGN_EXTEND)) {
35
+
PT_WARN_ON(range->common->max_vasz_lg2 != range->max_vasz_lg2);
36
+
prefix = fvalog2_div(range->va, range->max_vasz_lg2 - 1) ?
37
+
PT_VADDR_MAX :
38
+
0;
39
+
} else {
40
+
prefix = pt_full_va_prefix(range->common);
41
+
}
42
+
43
+
if (!fvalog2_div_eq(range->va, prefix, range->max_vasz_lg2) ||
44
+
!fvalog2_div_eq(range->last_va, prefix, range->max_vasz_lg2))
45
+
return -ERANGE;
46
+
return 0;
47
+
}
48
+
49
+
/**
50
+
* pt_index_to_va() - Update range->va to the current pts->index
51
+
* @pts: Iteration State
52
+
*
53
+
* Adjust range->va to match the current index. This is done in a lazy manner
54
+
* since computing the VA takes several instructions and is rarely required.
55
+
*/
56
+
static inline void pt_index_to_va(struct pt_state *pts)
57
+
{
58
+
pt_vaddr_t lower_va;
59
+
60
+
lower_va = log2_mul(pts->index, pt_table_item_lg2sz(pts));
61
+
pts->range->va = fvalog2_set_mod(pts->range->va, lower_va,
62
+
pt_table_oa_lg2sz(pts));
63
+
}
64
+
65
+
/*
66
+
* Add index_count_lg2 number of entries to pts's VA and index. The VA will be
67
+
* adjusted to the end of the contiguous block if it is currently in the middle.
68
+
*/
69
+
static inline void _pt_advance(struct pt_state *pts,
70
+
unsigned int index_count_lg2)
71
+
{
72
+
pts->index = log2_set_mod(pts->index + log2_to_int(index_count_lg2), 0,
73
+
index_count_lg2);
74
+
}
75
+
76
+
/**
77
+
* pt_entry_fully_covered() - Check if the item or entry is entirely contained
78
+
* within pts->range
79
+
* @pts: Iteration State
80
+
* @oasz_lg2: The size of the item to check, pt_table_item_lg2sz() or
81
+
* pt_entry_oa_lg2sz()
82
+
*
83
+
* Returns: true if the item is fully enclosed by the pts->range.
84
+
*/
85
+
static inline bool pt_entry_fully_covered(const struct pt_state *pts,
86
+
unsigned int oasz_lg2)
87
+
{
88
+
struct pt_range *range = pts->range;
89
+
90
+
/* Range begins at the start of the entry */
91
+
if (log2_mod(pts->range->va, oasz_lg2))
92
+
return false;
93
+
94
+
/* Range ends past the end of the entry */
95
+
if (!log2_div_eq(range->va, range->last_va, oasz_lg2))
96
+
return true;
97
+
98
+
/* Range ends at the end of the entry */
99
+
return log2_mod_eq_max(range->last_va, oasz_lg2);
100
+
}
101
+
102
+
/**
103
+
* pt_range_to_index() - Starting index for an iteration
104
+
* @pts: Iteration State
105
+
*
106
+
* Return: the starting index for the iteration in pts.
107
+
*/
108
+
static inline unsigned int pt_range_to_index(const struct pt_state *pts)
109
+
{
110
+
unsigned int isz_lg2 = pt_table_item_lg2sz(pts);
111
+
112
+
PT_WARN_ON(pts->level > pts->range->top_level);
113
+
if (pts->range->top_level == pts->level)
114
+
return log2_div(fvalog2_mod(pts->range->va,
115
+
pts->range->max_vasz_lg2),
116
+
isz_lg2);
117
+
return log2_mod(log2_div(pts->range->va, isz_lg2),
118
+
pt_num_items_lg2(pts));
119
+
}
120
+
121
+
/**
122
+
* pt_range_to_end_index() - Ending index iteration
123
+
* @pts: Iteration State
124
+
*
125
+
* Return: the last index for the iteration in pts.
126
+
*/
127
+
static inline unsigned int pt_range_to_end_index(const struct pt_state *pts)
128
+
{
129
+
unsigned int isz_lg2 = pt_table_item_lg2sz(pts);
130
+
struct pt_range *range = pts->range;
131
+
unsigned int num_entries_lg2;
132
+
133
+
if (range->va == range->last_va)
134
+
return pts->index + 1;
135
+
136
+
if (pts->range->top_level == pts->level)
137
+
return log2_div(fvalog2_mod(pts->range->last_va,
138
+
pts->range->max_vasz_lg2),
139
+
isz_lg2) +
140
+
1;
141
+
142
+
num_entries_lg2 = pt_num_items_lg2(pts);
143
+
144
+
/* last_va falls within this table */
145
+
if (log2_div_eq(range->va, range->last_va, num_entries_lg2 + isz_lg2))
146
+
return log2_mod(log2_div(pts->range->last_va, isz_lg2),
147
+
num_entries_lg2) +
148
+
1;
149
+
150
+
return log2_to_int(num_entries_lg2);
151
+
}
152
+
153
+
static inline void _pt_iter_first(struct pt_state *pts)
154
+
{
155
+
pts->index = pt_range_to_index(pts);
156
+
pts->end_index = pt_range_to_end_index(pts);
157
+
PT_WARN_ON(pts->index > pts->end_index);
158
+
}
159
+
160
+
static inline bool _pt_iter_load(struct pt_state *pts)
161
+
{
162
+
if (pts->index >= pts->end_index)
163
+
return false;
164
+
pt_load_entry(pts);
165
+
return true;
166
+
}
167
+
168
+
/**
169
+
* pt_next_entry() - Advance pts to the next entry
170
+
* @pts: Iteration State
171
+
*
172
+
* Update pts to go to the next index at this level. If pts is pointing at a
173
+
* contiguous entry then the index may advance my more than one.
174
+
*/
175
+
static inline void pt_next_entry(struct pt_state *pts)
176
+
{
177
+
if (pts->type == PT_ENTRY_OA &&
178
+
!__builtin_constant_p(pt_entry_num_contig_lg2(pts) == 0))
179
+
_pt_advance(pts, pt_entry_num_contig_lg2(pts));
180
+
else
181
+
pts->index++;
182
+
pt_index_to_va(pts);
183
+
}
184
+
185
+
/**
186
+
* for_each_pt_level_entry() - For loop wrapper over entries in the range
187
+
* @pts: Iteration State
188
+
*
189
+
* This is the basic iteration primitive. It iterates over all the entries in
190
+
* pts->range that fall within the pts's current table level. Each step does
191
+
* pt_load_entry(pts).
192
+
*/
193
+
#define for_each_pt_level_entry(pts) \
194
+
for (_pt_iter_first(pts); _pt_iter_load(pts); pt_next_entry(pts))
195
+
196
+
/**
197
+
* pt_load_single_entry() - Version of pt_load_entry() usable within a walker
198
+
* @pts: Iteration State
199
+
*
200
+
* Alternative to for_each_pt_level_entry() if the walker function uses only a
201
+
* single entry.
202
+
*/
203
+
static inline enum pt_entry_type pt_load_single_entry(struct pt_state *pts)
204
+
{
205
+
pts->index = pt_range_to_index(pts);
206
+
pt_load_entry(pts);
207
+
return pts->type;
208
+
}
209
+
210
+
static __always_inline struct pt_range _pt_top_range(struct pt_common *common,
211
+
uintptr_t top_of_table)
212
+
{
213
+
struct pt_range range = {
214
+
.common = common,
215
+
.top_table =
216
+
(struct pt_table_p *)(top_of_table &
217
+
~(uintptr_t)PT_TOP_LEVEL_MASK),
218
+
.top_level = top_of_table % (1 << PT_TOP_LEVEL_BITS),
219
+
};
220
+
struct pt_state pts = { .range = &range, .level = range.top_level };
221
+
unsigned int max_vasz_lg2;
222
+
223
+
max_vasz_lg2 = common->max_vasz_lg2;
224
+
if (pt_feature(common, PT_FEAT_DYNAMIC_TOP) &&
225
+
pts.level != PT_MAX_TOP_LEVEL)
226
+
max_vasz_lg2 = min_t(unsigned int, common->max_vasz_lg2,
227
+
pt_num_items_lg2(&pts) +
228
+
pt_table_item_lg2sz(&pts));
229
+
230
+
/*
231
+
* The top range will default to the lower region only with sign extend.
232
+
*/
233
+
range.max_vasz_lg2 = max_vasz_lg2;
234
+
if (pt_feature(common, PT_FEAT_SIGN_EXTEND))
235
+
max_vasz_lg2--;
236
+
237
+
range.va = fvalog2_set_mod(pt_full_va_prefix(common), 0, max_vasz_lg2);
238
+
range.last_va =
239
+
fvalog2_set_mod_max(pt_full_va_prefix(common), max_vasz_lg2);
240
+
return range;
241
+
}
242
+
243
+
/**
244
+
* pt_top_range() - Return a range that spans part of the top level
245
+
* @common: Table
246
+
*
247
+
* For PT_FEAT_SIGN_EXTEND this will return the lower range, and cover half the
248
+
* total page table. Otherwise it returns the entire page table.
249
+
*/
250
+
static __always_inline struct pt_range pt_top_range(struct pt_common *common)
251
+
{
252
+
/*
253
+
* The top pointer can change without locking. We capture the value and
254
+
* it's level here and are safe to walk it so long as both values are
255
+
* captured without tearing.
256
+
*/
257
+
return _pt_top_range(common, READ_ONCE(common->top_of_table));
258
+
}
259
+
260
+
/**
261
+
* pt_all_range() - Return a range that spans the entire page table
262
+
* @common: Table
263
+
*
264
+
* The returned range spans the whole page table. Due to how PT_FEAT_SIGN_EXTEND
265
+
* is supported range->va and range->last_va will be incorrect during the
266
+
* iteration and must not be accessed.
267
+
*/
268
+
static inline struct pt_range pt_all_range(struct pt_common *common)
269
+
{
270
+
struct pt_range range = pt_top_range(common);
271
+
272
+
if (!pt_feature(common, PT_FEAT_SIGN_EXTEND))
273
+
return range;
274
+
275
+
/*
276
+
* Pretend the table is linear from 0 without a sign extension. This
277
+
* generates the correct indexes for iteration.
278
+
*/
279
+
range.last_va = fvalog2_set_mod_max(0, range.max_vasz_lg2);
280
+
return range;
281
+
}
282
+
283
+
/**
284
+
* pt_upper_range() - Return a range that spans part of the top level
285
+
* @common: Table
286
+
*
287
+
* For PT_FEAT_SIGN_EXTEND this will return the upper range, and cover half the
288
+
* total page table. Otherwise it returns the entire page table.
289
+
*/
290
+
static inline struct pt_range pt_upper_range(struct pt_common *common)
291
+
{
292
+
struct pt_range range = pt_top_range(common);
293
+
294
+
if (!pt_feature(common, PT_FEAT_SIGN_EXTEND))
295
+
return range;
296
+
297
+
range.va = fvalog2_set_mod(PT_VADDR_MAX, 0, range.max_vasz_lg2 - 1);
298
+
range.last_va = PT_VADDR_MAX;
299
+
return range;
300
+
}
301
+
302
+
/**
303
+
* pt_make_range() - Return a range that spans part of the table
304
+
* @common: Table
305
+
* @va: Start address
306
+
* @last_va: Last address
307
+
*
308
+
* The caller must validate the range with pt_check_range() before using it.
309
+
*/
310
+
static __always_inline struct pt_range
311
+
pt_make_range(struct pt_common *common, pt_vaddr_t va, pt_vaddr_t last_va)
312
+
{
313
+
struct pt_range range =
314
+
_pt_top_range(common, READ_ONCE(common->top_of_table));
315
+
316
+
range.va = va;
317
+
range.last_va = last_va;
318
+
319
+
return range;
320
+
}
321
+
322
+
/*
323
+
* Span a slice of the table starting at a lower table level from an active
324
+
* walk.
325
+
*/
326
+
static __always_inline struct pt_range
327
+
pt_make_child_range(const struct pt_range *parent, pt_vaddr_t va,
328
+
pt_vaddr_t last_va)
329
+
{
330
+
struct pt_range range = *parent;
331
+
332
+
range.va = va;
333
+
range.last_va = last_va;
334
+
335
+
PT_WARN_ON(last_va < va);
336
+
PT_WARN_ON(pt_check_range(&range));
337
+
338
+
return range;
339
+
}
340
+
341
+
/**
342
+
* pt_init() - Initialize a pt_state on the stack
343
+
* @range: Range pointer to embed in the state
344
+
* @level: Table level for the state
345
+
* @table: Pointer to the table memory at level
346
+
*
347
+
* Helper to initialize the on-stack pt_state from walker arguments.
348
+
*/
349
+
static __always_inline struct pt_state
350
+
pt_init(struct pt_range *range, unsigned int level, struct pt_table_p *table)
351
+
{
352
+
struct pt_state pts = {
353
+
.range = range,
354
+
.table = table,
355
+
.level = level,
356
+
};
357
+
return pts;
358
+
}
359
+
360
+
/**
361
+
* pt_init_top() - Initialize a pt_state on the stack
362
+
* @range: Range pointer to embed in the state
363
+
*
364
+
* The pt_state points to the top most level.
365
+
*/
366
+
static __always_inline struct pt_state pt_init_top(struct pt_range *range)
367
+
{
368
+
return pt_init(range, range->top_level, range->top_table);
369
+
}
370
+
371
+
typedef int (*pt_level_fn_t)(struct pt_range *range, void *arg,
372
+
unsigned int level, struct pt_table_p *table);
373
+
374
+
/**
375
+
* pt_descend() - Recursively invoke the walker for the lower level
376
+
* @pts: Iteration State
377
+
* @arg: Value to pass to the function
378
+
* @fn: Walker function to call
379
+
*
380
+
* pts must point to a table item. Invoke fn as a walker on the table
381
+
* pts points to.
382
+
*/
383
+
static __always_inline int pt_descend(struct pt_state *pts, void *arg,
384
+
pt_level_fn_t fn)
385
+
{
386
+
int ret;
387
+
388
+
if (PT_WARN_ON(!pts->table_lower))
389
+
return -EINVAL;
390
+
391
+
ret = (*fn)(pts->range, arg, pts->level - 1, pts->table_lower);
392
+
return ret;
393
+
}
394
+
395
+
/**
396
+
* pt_walk_range() - Walk over a VA range
397
+
* @range: Range pointer
398
+
* @fn: Walker function to call
399
+
* @arg: Value to pass to the function
400
+
*
401
+
* Walk over a VA range. The caller should have done a validity check, at
402
+
* least calling pt_check_range(), when building range. The walk will
403
+
* start at the top most table.
404
+
*/
405
+
static __always_inline int pt_walk_range(struct pt_range *range,
406
+
pt_level_fn_t fn, void *arg)
407
+
{
408
+
return fn(range, arg, range->top_level, range->top_table);
409
+
}
410
+
411
+
/*
412
+
* pt_walk_descend() - Recursively invoke the walker for a slice of a lower
413
+
* level
414
+
* @pts: Iteration State
415
+
* @va: Start address
416
+
* @last_va: Last address
417
+
* @fn: Walker function to call
418
+
* @arg: Value to pass to the function
419
+
*
420
+
* With pts pointing at a table item this will descend and over a slice of the
421
+
* lower table. The caller must ensure that va/last_va are within the table
422
+
* item. This creates a new walk and does not alter pts or pts->range.
423
+
*/
424
+
static __always_inline int pt_walk_descend(const struct pt_state *pts,
425
+
pt_vaddr_t va, pt_vaddr_t last_va,
426
+
pt_level_fn_t fn, void *arg)
427
+
{
428
+
struct pt_range range = pt_make_child_range(pts->range, va, last_va);
429
+
430
+
if (PT_WARN_ON(!pt_can_have_table(pts)) ||
431
+
PT_WARN_ON(!pts->table_lower))
432
+
return -EINVAL;
433
+
434
+
return fn(&range, arg, pts->level - 1, pts->table_lower);
435
+
}
436
+
437
+
/*
438
+
* pt_walk_descend_all() - Recursively invoke the walker for a table item
439
+
* @parent_pts: Iteration State
440
+
* @fn: Walker function to call
441
+
* @arg: Value to pass to the function
442
+
*
443
+
* With pts pointing at a table item this will descend and over the entire lower
444
+
* table. This creates a new walk and does not alter pts or pts->range.
445
+
*/
446
+
static __always_inline int
447
+
pt_walk_descend_all(const struct pt_state *parent_pts, pt_level_fn_t fn,
448
+
void *arg)
449
+
{
450
+
unsigned int isz_lg2 = pt_table_item_lg2sz(parent_pts);
451
+
452
+
return pt_walk_descend(parent_pts,
453
+
log2_set_mod(parent_pts->range->va, 0, isz_lg2),
454
+
log2_set_mod_max(parent_pts->range->va, isz_lg2),
455
+
fn, arg);
456
+
}
457
+
458
+
/**
459
+
* pt_range_slice() - Return a range that spans indexes
460
+
* @pts: Iteration State
461
+
* @start_index: Starting index within pts
462
+
* @end_index: Ending index within pts
463
+
*
464
+
* Create a range than spans an index range of the current table level
465
+
* pt_state points at.
466
+
*/
467
+
static inline struct pt_range pt_range_slice(const struct pt_state *pts,
468
+
unsigned int start_index,
469
+
unsigned int end_index)
470
+
{
471
+
unsigned int table_lg2sz = pt_table_oa_lg2sz(pts);
472
+
pt_vaddr_t last_va;
473
+
pt_vaddr_t va;
474
+
475
+
va = fvalog2_set_mod(pts->range->va,
476
+
log2_mul(start_index, pt_table_item_lg2sz(pts)),
477
+
table_lg2sz);
478
+
last_va = fvalog2_set_mod(
479
+
pts->range->va,
480
+
log2_mul(end_index, pt_table_item_lg2sz(pts)) - 1, table_lg2sz);
481
+
return pt_make_child_range(pts->range, va, last_va);
482
+
}
483
+
484
+
/**
485
+
* pt_top_memsize_lg2()
486
+
* @common: Table
487
+
* @top_of_table: Top of table value from _pt_top_set()
488
+
*
489
+
* Compute the allocation size of the top table. For PT_FEAT_DYNAMIC_TOP this
490
+
* will compute the top size assuming the table will grow.
491
+
*/
492
+
static inline unsigned int pt_top_memsize_lg2(struct pt_common *common,
493
+
uintptr_t top_of_table)
494
+
{
495
+
struct pt_range range = _pt_top_range(common, top_of_table);
496
+
struct pt_state pts = pt_init_top(&range);
497
+
unsigned int num_items_lg2;
498
+
499
+
num_items_lg2 = common->max_vasz_lg2 - pt_table_item_lg2sz(&pts);
500
+
if (range.top_level != PT_MAX_TOP_LEVEL &&
501
+
pt_feature(common, PT_FEAT_DYNAMIC_TOP))
502
+
num_items_lg2 = min(num_items_lg2, pt_num_items_lg2(&pts));
503
+
504
+
/* Round up the allocation size to the minimum alignment */
505
+
return max(ffs_t(u64, PT_TOP_PHYS_MASK),
506
+
num_items_lg2 + ilog2(PT_ITEM_WORD_SIZE));
507
+
}
508
+
509
+
/**
510
+
* pt_compute_best_pgsize() - Determine the best page size for leaf entries
511
+
* @pgsz_bitmap: Permitted page sizes
512
+
* @va: Starting virtual address for the leaf entry
513
+
* @last_va: Last virtual address for the leaf entry, sets the max page size
514
+
* @oa: Starting output address for the leaf entry
515
+
*
516
+
* Compute the largest page size for va, last_va, and oa together and return it
517
+
* in lg2. The largest page size depends on the format's supported page sizes at
518
+
* this level, and the relative alignment of the VA and OA addresses. 0 means
519
+
* the OA cannot be stored with the provided pgsz_bitmap.
520
+
*/
521
+
static inline unsigned int pt_compute_best_pgsize(pt_vaddr_t pgsz_bitmap,
522
+
pt_vaddr_t va,
523
+
pt_vaddr_t last_va,
524
+
pt_oaddr_t oa)
525
+
{
526
+
unsigned int best_pgsz_lg2;
527
+
unsigned int pgsz_lg2;
528
+
pt_vaddr_t len = last_va - va + 1;
529
+
pt_vaddr_t mask;
530
+
531
+
if (PT_WARN_ON(va >= last_va))
532
+
return 0;
533
+
534
+
/*
535
+
* Given a VA/OA pair the best page size is the largest page size
536
+
* where:
537
+
*
538
+
* 1) VA and OA start at the page. Bitwise this is the count of least
539
+
* significant 0 bits.
540
+
* This also implies that last_va/oa has the same prefix as va/oa.
541
+
*/
542
+
mask = va | oa;
543
+
544
+
/*
545
+
* 2) The page size is not larger than the last_va (length). Since page
546
+
* sizes are always power of two this can't be larger than the
547
+
* largest power of two factor of the length.
548
+
*/
549
+
mask |= log2_to_int(vafls(len) - 1);
550
+
551
+
best_pgsz_lg2 = vaffs(mask);
552
+
553
+
/* Choose the highest bit <= best_pgsz_lg2 */
554
+
if (best_pgsz_lg2 < PT_VADDR_MAX_LG2 - 1)
555
+
pgsz_bitmap = log2_mod(pgsz_bitmap, best_pgsz_lg2 + 1);
556
+
557
+
pgsz_lg2 = vafls(pgsz_bitmap);
558
+
if (!pgsz_lg2)
559
+
return 0;
560
+
561
+
pgsz_lg2--;
562
+
563
+
PT_WARN_ON(log2_mod(va, pgsz_lg2) != 0);
564
+
PT_WARN_ON(oalog2_mod(oa, pgsz_lg2) != 0);
565
+
PT_WARN_ON(va + log2_to_int(pgsz_lg2) - 1 > last_va);
566
+
PT_WARN_ON(!log2_div_eq(va, va + log2_to_int(pgsz_lg2) - 1, pgsz_lg2));
567
+
PT_WARN_ON(
568
+
!oalog2_div_eq(oa, oa + log2_to_int(pgsz_lg2) - 1, pgsz_lg2));
569
+
return pgsz_lg2;
570
+
}
571
+
572
+
#define _PT_MAKE_CALL_LEVEL(fn) \
573
+
static __always_inline int fn(struct pt_range *range, void *arg, \
574
+
unsigned int level, \
575
+
struct pt_table_p *table) \
576
+
{ \
577
+
static_assert(PT_MAX_TOP_LEVEL <= 5); \
578
+
if (level == 0) \
579
+
return CONCATENATE(fn, 0)(range, arg, 0, table); \
580
+
if (level == 1 || PT_MAX_TOP_LEVEL == 1) \
581
+
return CONCATENATE(fn, 1)(range, arg, 1, table); \
582
+
if (level == 2 || PT_MAX_TOP_LEVEL == 2) \
583
+
return CONCATENATE(fn, 2)(range, arg, 2, table); \
584
+
if (level == 3 || PT_MAX_TOP_LEVEL == 3) \
585
+
return CONCATENATE(fn, 3)(range, arg, 3, table); \
586
+
if (level == 4 || PT_MAX_TOP_LEVEL == 4) \
587
+
return CONCATENATE(fn, 4)(range, arg, 4, table); \
588
+
return CONCATENATE(fn, 5)(range, arg, 5, table); \
589
+
}
590
+
591
+
static inline int __pt_make_level_fn_err(struct pt_range *range, void *arg,
592
+
unsigned int unused_level,
593
+
struct pt_table_p *table)
594
+
{
595
+
static_assert(PT_MAX_TOP_LEVEL <= 5);
596
+
return -EPROTOTYPE;
597
+
}
598
+
599
+
#define __PT_MAKE_LEVEL_FN(fn, level, descend_fn, do_fn) \
600
+
static inline int fn(struct pt_range *range, void *arg, \
601
+
unsigned int unused_level, \
602
+
struct pt_table_p *table) \
603
+
{ \
604
+
return do_fn(range, arg, level, table, descend_fn); \
605
+
}
606
+
607
+
/**
608
+
* PT_MAKE_LEVELS() - Build an unwound walker
609
+
* @fn: Name of the walker function
610
+
* @do_fn: Function to call at each level
611
+
*
612
+
* This builds a function call tree that can be fully inlined.
613
+
* The caller must provide a function body in an __always_inline function::
614
+
*
615
+
* static __always_inline int do(struct pt_range *range, void *arg,
616
+
* unsigned int level, struct pt_table_p *table,
617
+
* pt_level_fn_t descend_fn)
618
+
*
619
+
* An inline function will be created for each table level that calls do_fn with
620
+
* a compile time constant for level and a pointer to the next lower function.
621
+
* This generates an optimally inlined walk where each of the functions sees a
622
+
* constant level and can codegen the exact constants/etc for that level.
623
+
*
624
+
* Note this can produce a lot of code!
625
+
*/
626
+
#define PT_MAKE_LEVELS(fn, do_fn) \
627
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 0), 0, __pt_make_level_fn_err, \
628
+
do_fn); \
629
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 1), 1, CONCATENATE(fn, 0), do_fn); \
630
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 2), 2, CONCATENATE(fn, 1), do_fn); \
631
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 3), 3, CONCATENATE(fn, 2), do_fn); \
632
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 4), 4, CONCATENATE(fn, 3), do_fn); \
633
+
__PT_MAKE_LEVEL_FN(CONCATENATE(fn, 5), 5, CONCATENATE(fn, 4), do_fn); \
634
+
_PT_MAKE_CALL_LEVEL(fn)
635
+
636
+
#endif
+122
drivers/iommu/generic_pt/pt_log2.h
+122
drivers/iommu/generic_pt/pt_log2.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*
5
+
* Helper macros for working with log2 values
6
+
*
7
+
*/
8
+
#ifndef __GENERIC_PT_LOG2_H
9
+
#define __GENERIC_PT_LOG2_H
10
+
#include <linux/bitops.h>
11
+
#include <linux/limits.h>
12
+
13
+
/* Compute a */
14
+
#define log2_to_int_t(type, a_lg2) ((type)(((type)1) << (a_lg2)))
15
+
static_assert(log2_to_int_t(unsigned int, 0) == 1);
16
+
17
+
/* Compute a - 1 (aka all low bits set) */
18
+
#define log2_to_max_int_t(type, a_lg2) ((type)(log2_to_int_t(type, a_lg2) - 1))
19
+
20
+
/* Compute a / b */
21
+
#define log2_div_t(type, a, b_lg2) ((type)(((type)a) >> (b_lg2)))
22
+
static_assert(log2_div_t(unsigned int, 4, 2) == 1);
23
+
24
+
/*
25
+
* Compute:
26
+
* a / c == b / c
27
+
* aka the high bits are equal
28
+
*/
29
+
#define log2_div_eq_t(type, a, b, c_lg2) \
30
+
(log2_div_t(type, (a) ^ (b), c_lg2) == 0)
31
+
static_assert(log2_div_eq_t(unsigned int, 1, 1, 2));
32
+
33
+
/* Compute a % b */
34
+
#define log2_mod_t(type, a, b_lg2) \
35
+
((type)(((type)a) & log2_to_max_int_t(type, b_lg2)))
36
+
static_assert(log2_mod_t(unsigned int, 1, 2) == 1);
37
+
38
+
/*
39
+
* Compute:
40
+
* a % b == b - 1
41
+
* aka the low bits are all 1s
42
+
*/
43
+
#define log2_mod_eq_max_t(type, a, b_lg2) \
44
+
(log2_mod_t(type, a, b_lg2) == log2_to_max_int_t(type, b_lg2))
45
+
static_assert(log2_mod_eq_max_t(unsigned int, 3, 2));
46
+
47
+
/*
48
+
* Return a value such that:
49
+
* a / b == ret / b
50
+
* ret % b == val
51
+
* aka set the low bits to val. val must be < b
52
+
*/
53
+
#define log2_set_mod_t(type, a, val, b_lg2) \
54
+
((((type)(a)) & (~log2_to_max_int_t(type, b_lg2))) | ((type)(val)))
55
+
static_assert(log2_set_mod_t(unsigned int, 3, 1, 2) == 1);
56
+
57
+
/* Return a value such that:
58
+
* a / b == ret / b
59
+
* ret % b == b - 1
60
+
* aka set the low bits to all 1s
61
+
*/
62
+
#define log2_set_mod_max_t(type, a, b_lg2) \
63
+
(((type)(a)) | log2_to_max_int_t(type, b_lg2))
64
+
static_assert(log2_set_mod_max_t(unsigned int, 2, 2) == 3);
65
+
66
+
/* Compute a * b */
67
+
#define log2_mul_t(type, a, b_lg2) ((type)(((type)a) << (b_lg2)))
68
+
static_assert(log2_mul_t(unsigned int, 2, 2) == 8);
69
+
70
+
#define _dispatch_sz(type, fn, a) \
71
+
(sizeof(type) == 4 ? fn##32((u32)a) : fn##64(a))
72
+
73
+
/*
74
+
* Return the highest value such that:
75
+
* fls_t(u32, 0) == 0
76
+
* fls_t(u3, 1) == 1
77
+
* a >= log2_to_int(ret - 1)
78
+
* aka find last set bit
79
+
*/
80
+
static inline unsigned int fls32(u32 a)
81
+
{
82
+
return fls(a);
83
+
}
84
+
#define fls_t(type, a) _dispatch_sz(type, fls, a)
85
+
86
+
/*
87
+
* Return the highest value such that:
88
+
* ffs_t(u32, 0) == UNDEFINED
89
+
* ffs_t(u32, 1) == 0
90
+
* log_mod(a, ret) == 0
91
+
* aka find first set bit
92
+
*/
93
+
static inline unsigned int __ffs32(u32 a)
94
+
{
95
+
return __ffs(a);
96
+
}
97
+
#define ffs_t(type, a) _dispatch_sz(type, __ffs, a)
98
+
99
+
/*
100
+
* Return the highest value such that:
101
+
* ffz_t(u32, U32_MAX) == UNDEFINED
102
+
* ffz_t(u32, 0) == 0
103
+
* ffz_t(u32, 1) == 1
104
+
* log_mod(a, ret) == log_to_max_int(ret)
105
+
* aka find first zero bit
106
+
*/
107
+
static inline unsigned int ffz32(u32 a)
108
+
{
109
+
return ffz(a);
110
+
}
111
+
static inline unsigned int ffz64(u64 a)
112
+
{
113
+
if (sizeof(u64) == sizeof(unsigned long))
114
+
return ffz(a);
115
+
116
+
if ((u32)a == U32_MAX)
117
+
return ffz32(a >> 32) + 32;
118
+
return ffz32(a);
119
+
}
120
+
#define ffz_t(type, a) _dispatch_sz(type, ffz, a)
121
+
122
+
#endif
+135
include/linux/generic_pt/common.h
+135
include/linux/generic_pt/common.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
/*
3
+
* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
4
+
*/
5
+
#ifndef __GENERIC_PT_COMMON_H
6
+
#define __GENERIC_PT_COMMON_H
7
+
8
+
#include <linux/types.h>
9
+
#include <linux/build_bug.h>
10
+
#include <linux/bits.h>
11
+
12
+
/**
13
+
* DOC: Generic Radix Page Table
14
+
*
15
+
* Generic Radix Page Table is a set of functions and helpers to efficiently
16
+
* parse radix style page tables typically seen in HW implementations. The
17
+
* interface is built to deliver similar code generation as the mm's pte/pmd/etc
18
+
* system by fully inlining the exact code required to handle each table level.
19
+
*
20
+
* Like the mm subsystem each format contributes its parsing implementation
21
+
* under common names and the common code implements the required algorithms.
22
+
*
23
+
* The system is divided into three logical levels:
24
+
*
25
+
* - The page table format and its manipulation functions
26
+
* - Generic helpers to give a consistent API regardless of underlying format
27
+
* - An algorithm implementation (e.g. IOMMU/DRM/KVM/MM)
28
+
*
29
+
* Multiple implementations are supported. The intention is to have the generic
30
+
* format code be re-usable for whatever specialized implementation is required.
31
+
* The generic code is solely about the format of the radix tree; it does not
32
+
* include memory allocation or higher level decisions that are left for the
33
+
* implementation.
34
+
*
35
+
* The generic framework supports a superset of functions across many HW
36
+
* implementations:
37
+
*
38
+
* - Entries comprised of contiguous blocks of IO PTEs for larger page sizes
39
+
* - Multi-level tables, up to 6 levels. Runtime selected top level
40
+
* - Runtime variable table level size (ARM's concatenated tables)
41
+
* - Expandable top level allowing dynamic sizing of table levels
42
+
* - Optional leaf entries at any level
43
+
* - 32-bit/64-bit virtual and output addresses, using every address bit
44
+
* - Dirty tracking
45
+
* - Sign extended addressing
46
+
*/
47
+
48
+
/**
49
+
* struct pt_common - struct for all page table implementations
50
+
*/
51
+
struct pt_common {
52
+
/**
53
+
* @top_of_table: Encodes the table top pointer and the top level in a
54
+
* single value. Must use READ_ONCE/WRITE_ONCE to access it. The lower
55
+
* bits of the aligned table pointer are used for the level.
56
+
*/
57
+
uintptr_t top_of_table;
58
+
/**
59
+
* @max_oasz_lg2: Maximum number of bits the OA can contain. Upper bits
60
+
* must be zero. This may be less than what the page table format
61
+
* supports, but must not be more.
62
+
*/
63
+
u8 max_oasz_lg2;
64
+
/**
65
+
* @max_vasz_lg2: Maximum number of bits the VA can contain. Upper bits
66
+
* are 0 or 1 depending on pt_full_va_prefix(). This may be less than
67
+
* what the page table format supports, but must not be more. When
68
+
* PT_FEAT_DYNAMIC_TOP is set this reflects the maximum VA capability.
69
+
*/
70
+
u8 max_vasz_lg2;
71
+
/**
72
+
* @features: Bitmap of `enum pt_features`
73
+
*/
74
+
unsigned int features;
75
+
};
76
+
77
+
/* Encoding parameters for top_of_table */
78
+
enum {
79
+
PT_TOP_LEVEL_BITS = 3,
80
+
PT_TOP_LEVEL_MASK = GENMASK(PT_TOP_LEVEL_BITS - 1, 0),
81
+
};
82
+
83
+
/**
84
+
* enum pt_features - Features turned on in the table. Each symbol is a bit
85
+
* position.
86
+
*/
87
+
enum pt_features {
88
+
/**
89
+
* @PT_FEAT_FULL_VA: The table can span the full VA range from 0 to
90
+
* PT_VADDR_MAX.
91
+
*/
92
+
PT_FEAT_FULL_VA,
93
+
/**
94
+
* @PT_FEAT_DYNAMIC_TOP: The table's top level can be increased
95
+
* dynamically during map. This requires HW support for atomically
96
+
* setting both the table top pointer and the starting table level.
97
+
*/
98
+
PT_FEAT_DYNAMIC_TOP,
99
+
/**
100
+
* @PT_FEAT_SIGN_EXTEND: The top most bit of the valid VA range sign
101
+
* extends up to the full pt_vaddr_t. This divides the page table into
102
+
* three VA ranges::
103
+
*
104
+
* 0 -> 2^N - 1 Lower
105
+
* 2^N -> (MAX - 2^N - 1) Non-Canonical
106
+
* MAX - 2^N -> MAX Upper
107
+
*
108
+
* In this mode pt_common::max_vasz_lg2 includes the sign bit and the
109
+
* upper bits that don't fall within the translation are just validated.
110
+
*
111
+
* If not set there is no sign extension and valid VA goes from 0 to 2^N
112
+
* - 1.
113
+
*/
114
+
PT_FEAT_SIGN_EXTEND,
115
+
/**
116
+
* @PT_FEAT_FLUSH_RANGE: IOTLB maintenance is done by flushing IOVA
117
+
* ranges which will clean out any walk cache or any IOPTE fully
118
+
* contained by the range. The optimization objective is to minimize the
119
+
* number of flushes even if ranges include IOVA gaps that do not need
120
+
* to be flushed.
121
+
*/
122
+
PT_FEAT_FLUSH_RANGE,
123
+
/**
124
+
* @PT_FEAT_FLUSH_RANGE_NO_GAPS: Like PT_FEAT_FLUSH_RANGE except that
125
+
* the optimization objective is to only flush IOVA that has been
126
+
* changed. This mode is suitable for cases like hypervisor shadowing
127
+
* where flushing unchanged ranges may cause the hypervisor to reparse
128
+
* significant amount of page table.
129
+
*/
130
+
PT_FEAT_FLUSH_RANGE_NO_GAPS,
131
+
/* private: */
132
+
PT_FEAT_FMT_START,
133
+
};
134
+
135
+
#endif