Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
mm: introduce leaf entry type and use to simplify leaf entry logic
The kernel maintains leaf page table entries which contain either:
The kernel maintains leaf page table entries which contain either:
- Nothing ('none' entries)
- Present entries*
- Everything else that will cause a fault which the kernel handles
* Present entries are either entries the hardware can navigate without page
fault or special cases like NUMA hint protnone or PMD with cleared
present bit which contain hardware-valid entries modulo the present bit.
In the 'everything else' group we include swap entries, but we also
include a number of other things such as migration entries, device private
entries and marker entries.
Unfortunately this 'everything else' group expresses everything through a
swp_entry_t type, and these entries are referred to swap entries even
though they may well not contain a... swap entry.
This is compounded by the rather mind-boggling concept of a non-swap swap
entry (checked via non_swap_entry()) and the means by which we twist and
turn to satisfy this.
This patch lays the foundation for reducing this confusion.
We refer to 'everything else' as a 'software-define leaf entry' or
'softleaf'. for short And in fact we scoop up the 'none' entries into
this concept also so we are left with:
- Present entries.
- Softleaf entries (which may be empty).
This allows for radical simplification across the board - one can simply
convert any leaf page table entry to a leaf entry via softleaf_from_pte().
If the entry is present, we return an empty leaf entry, so it is assumed
the caller is aware that they must differentiate between the two
categories of page table entries, checking for the former via
pte_present().
As a result, we can eliminate a number of places where we would otherwise
need to use predicates to see if we can proceed with leaf page table entry
conversion and instead just go ahead and do it unconditionally.
We do so where we can, adjusting surrounding logic as necessary to
integrate the new softleaf_t logic as far as seems reasonable at this
stage.
We typedef swp_entry_t to softleaf_t for the time being until the
conversion can be complete, meaning everything remains compatible
regardless of which type is used. We will eventually remove swp_entry_t
when the conversion is complete.
We introduce a new header file to keep things clear - leafops.h - this
imports swapops.h so can direct replace swapops imports without issue, and
we do so in all the files that require it.
Additionally, add new leafops.h file to core mm maintainers entry.
Link: https://lkml.kernel.org/r/c879383aac77d96a03e4d38f7daba893cd35fc76.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Mathew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Wei Xu <weixugc@google.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
authored by
Lorenzo Stoakes
and committed by
Andrew Morton
68aa2fdb
c093cf45
+502
-164
+1
MAINTAINERS
+1
MAINTAINERS
+13
-13
fs/proc/task_mmu.c
+13
-13
fs/proc/task_mmu.c
···
14
14
#include <linux/rmap.h>
15
15
#include <linux/swap.h>
16
16
#include <linux/sched/mm.h>
17
-
#include <linux/swapops.h>
17
+
#include <linux/leafops.h>
18
18
#include <linux/mmu_notifier.h>
19
19
#include <linux/page_idle.h>
20
20
#include <linux/shmem_fs.h>
···
1231
1231
if (pte_present(ptent)) {
1232
1232
folio = page_folio(pte_page(ptent));
1233
1233
present = true;
1234
-
} else if (is_swap_pte(ptent)) {
1235
-
swp_entry_t swpent = pte_to_swp_entry(ptent);
1234
+
} else {
1235
+
const softleaf_t entry = softleaf_from_pte(ptent);
1236
1236
1237
-
if (is_pfn_swap_entry(swpent))
1238
-
folio = pfn_swap_entry_folio(swpent);
1237
+
if (softleaf_has_pfn(entry))
1238
+
folio = softleaf_to_folio(entry);
1239
1239
}
1240
1240
1241
1241
if (folio) {
···
1956
1956
flags |= PM_SWAP;
1957
1957
if (is_pfn_swap_entry(entry))
1958
1958
page = pfn_swap_entry_to_page(entry);
1959
-
if (pte_marker_entry_uffd_wp(entry))
1959
+
if (softleaf_is_uffd_wp_marker(entry))
1960
1960
flags |= PM_UFFD_WP;
1961
-
if (is_guard_swp_entry(entry))
1961
+
if (softleaf_is_guard_marker(entry))
1962
1962
flags |= PM_GUARD_REGION;
1963
1963
}
1964
1964
···
2331
2331
if (pte_soft_dirty(pte))
2332
2332
categories |= PAGE_IS_SOFT_DIRTY;
2333
2333
} else if (is_swap_pte(pte)) {
2334
-
swp_entry_t swp;
2334
+
softleaf_t entry;
2335
2335
2336
2336
categories |= PAGE_IS_SWAPPED;
2337
2337
if (!pte_swp_uffd_wp_any(pte))
2338
2338
categories |= PAGE_IS_WRITTEN;
2339
2339
2340
-
swp = pte_to_swp_entry(pte);
2341
-
if (is_guard_swp_entry(swp))
2340
+
entry = softleaf_from_pte(pte);
2341
+
if (softleaf_is_guard_marker(entry))
2342
2342
categories |= PAGE_IS_GUARD;
2343
2343
else if ((p->masks_of_interest & PAGE_IS_FILE) &&
2344
-
is_pfn_swap_entry(swp) &&
2345
-
!folio_test_anon(pfn_swap_entry_folio(swp)))
2344
+
softleaf_has_pfn(entry) &&
2345
+
!folio_test_anon(softleaf_to_folio(entry)))
2346
2346
categories |= PAGE_IS_FILE;
2347
2347
2348
2348
if (pte_swp_soft_dirty(pte))
···
2467
2467
{
2468
2468
unsigned long psize;
2469
2469
2470
-
if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
2470
+
if (is_hugetlb_entry_hwpoisoned(ptent) || pte_is_marker(ptent))
2471
2471
return;
2472
2472
2473
2473
psize = huge_page_size(hstate_vma(vma));
+3
-3
fs/userfaultfd.c
+3
-3
fs/userfaultfd.c
···
29
29
#include <linux/ioctl.h>
30
30
#include <linux/security.h>
31
31
#include <linux/hugetlb.h>
32
-
#include <linux/swapops.h>
32
+
#include <linux/leafops.h>
33
33
#include <linux/miscdevice.h>
34
34
#include <linux/uio.h>
35
35
···
251
251
if (huge_pte_none(pte))
252
252
return true;
253
253
/* UFFD PTE markers require userspace to resolve the fault. */
254
-
if (is_uffd_pte_marker(pte))
254
+
if (pte_is_uffd_marker(pte))
255
255
return true;
256
256
/*
257
257
* If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
···
337
337
if (pte_none(ptent))
338
338
goto out;
339
339
/* UFFD PTE markers require userspace to resolve the fault. */
340
-
if (is_uffd_pte_marker(ptent))
340
+
if (pte_is_uffd_marker(ptent))
341
341
goto out;
342
342
/*
343
343
* If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
+387
include/linux/leafops.h
+387
include/linux/leafops.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0 */
2
+
/*
3
+
* Describes operations that can be performed on software-defined page table
4
+
* leaf entries. These are abstracted from the hardware page table entries
5
+
* themselves by the softleaf_t type, see mm_types.h.
6
+
*/
7
+
#ifndef _LINUX_LEAFOPS_H
8
+
#define _LINUX_LEAFOPS_H
9
+
10
+
#include <linux/mm_types.h>
11
+
#include <linux/swapops.h>
12
+
#include <linux/swap.h>
13
+
14
+
#ifdef CONFIG_MMU
15
+
16
+
/* Temporary until swp_entry_t eliminated. */
17
+
#define LEAF_TYPE_SHIFT SWP_TYPE_SHIFT
18
+
19
+
enum softleaf_type {
20
+
/* Fundamental types. */
21
+
SOFTLEAF_NONE,
22
+
SOFTLEAF_SWAP,
23
+
/* Migration types. */
24
+
SOFTLEAF_MIGRATION_READ,
25
+
SOFTLEAF_MIGRATION_READ_EXCLUSIVE,
26
+
SOFTLEAF_MIGRATION_WRITE,
27
+
/* Device types. */
28
+
SOFTLEAF_DEVICE_PRIVATE_READ,
29
+
SOFTLEAF_DEVICE_PRIVATE_WRITE,
30
+
SOFTLEAF_DEVICE_EXCLUSIVE,
31
+
/* H/W posion types. */
32
+
SOFTLEAF_HWPOISON,
33
+
/* Marker types. */
34
+
SOFTLEAF_MARKER,
35
+
};
36
+
37
+
/**
38
+
* softleaf_mk_none() - Create an empty ('none') leaf entry.
39
+
* Returns: empty leaf entry.
40
+
*/
41
+
static inline softleaf_t softleaf_mk_none(void)
42
+
{
43
+
return ((softleaf_t) { 0 });
44
+
}
45
+
46
+
/**
47
+
* softleaf_from_pte() - Obtain a leaf entry from a PTE entry.
48
+
* @pte: PTE entry.
49
+
*
50
+
* If @pte is present (therefore not a leaf entry) the function returns an empty
51
+
* leaf entry. Otherwise, it returns a leaf entry.
52
+
*
53
+
* Returns: Leaf entry.
54
+
*/
55
+
static inline softleaf_t softleaf_from_pte(pte_t pte)
56
+
{
57
+
if (pte_present(pte) || pte_none(pte))
58
+
return softleaf_mk_none();
59
+
60
+
/* Temporary until swp_entry_t eliminated. */
61
+
return pte_to_swp_entry(pte);
62
+
}
63
+
64
+
/**
65
+
* softleaf_is_none() - Is the leaf entry empty?
66
+
* @entry: Leaf entry.
67
+
*
68
+
* Empty entries are typically the result of a 'none' page table leaf entry
69
+
* being converted to a leaf entry.
70
+
*
71
+
* Returns: true if the entry is empty, false otherwise.
72
+
*/
73
+
static inline bool softleaf_is_none(softleaf_t entry)
74
+
{
75
+
return entry.val == 0;
76
+
}
77
+
78
+
/**
79
+
* softleaf_type() - Identify the type of leaf entry.
80
+
* @enntry: Leaf entry.
81
+
*
82
+
* Returns: the leaf entry type associated with @entry.
83
+
*/
84
+
static inline enum softleaf_type softleaf_type(softleaf_t entry)
85
+
{
86
+
unsigned int type_num;
87
+
88
+
if (softleaf_is_none(entry))
89
+
return SOFTLEAF_NONE;
90
+
91
+
type_num = entry.val >> LEAF_TYPE_SHIFT;
92
+
93
+
if (type_num < MAX_SWAPFILES)
94
+
return SOFTLEAF_SWAP;
95
+
96
+
switch (type_num) {
97
+
#ifdef CONFIG_MIGRATION
98
+
case SWP_MIGRATION_READ:
99
+
return SOFTLEAF_MIGRATION_READ;
100
+
case SWP_MIGRATION_READ_EXCLUSIVE:
101
+
return SOFTLEAF_MIGRATION_READ_EXCLUSIVE;
102
+
case SWP_MIGRATION_WRITE:
103
+
return SOFTLEAF_MIGRATION_WRITE;
104
+
#endif
105
+
#ifdef CONFIG_DEVICE_PRIVATE
106
+
case SWP_DEVICE_WRITE:
107
+
return SOFTLEAF_DEVICE_PRIVATE_WRITE;
108
+
case SWP_DEVICE_READ:
109
+
return SOFTLEAF_DEVICE_PRIVATE_READ;
110
+
case SWP_DEVICE_EXCLUSIVE:
111
+
return SOFTLEAF_DEVICE_EXCLUSIVE;
112
+
#endif
113
+
#ifdef CONFIG_MEMORY_FAILURE
114
+
case SWP_HWPOISON:
115
+
return SOFTLEAF_HWPOISON;
116
+
#endif
117
+
case SWP_PTE_MARKER:
118
+
return SOFTLEAF_MARKER;
119
+
}
120
+
121
+
/* Unknown entry type. */
122
+
VM_WARN_ON_ONCE(1);
123
+
return SOFTLEAF_NONE;
124
+
}
125
+
126
+
/**
127
+
* softleaf_is_swap() - Is this leaf entry a swap entry?
128
+
* @entry: Leaf entry.
129
+
*
130
+
* Returns: true if the leaf entry is a swap entry, otherwise false.
131
+
*/
132
+
static inline bool softleaf_is_swap(softleaf_t entry)
133
+
{
134
+
return softleaf_type(entry) == SOFTLEAF_SWAP;
135
+
}
136
+
137
+
/**
138
+
* softleaf_is_migration() - Is this leaf entry a migration entry?
139
+
* @entry: Leaf entry.
140
+
*
141
+
* Returns: true if the leaf entry is a migration entry, otherwise false.
142
+
*/
143
+
static inline bool softleaf_is_migration(softleaf_t entry)
144
+
{
145
+
switch (softleaf_type(entry)) {
146
+
case SOFTLEAF_MIGRATION_READ:
147
+
case SOFTLEAF_MIGRATION_READ_EXCLUSIVE:
148
+
case SOFTLEAF_MIGRATION_WRITE:
149
+
return true;
150
+
default:
151
+
return false;
152
+
}
153
+
}
154
+
155
+
/**
156
+
* softleaf_is_device_private() - Is this leaf entry a device private entry?
157
+
* @entry: Leaf entry.
158
+
*
159
+
* Returns: true if the leaf entry is a device private entry, otherwise false.
160
+
*/
161
+
static inline bool softleaf_is_device_private(softleaf_t entry)
162
+
{
163
+
switch (softleaf_type(entry)) {
164
+
case SOFTLEAF_DEVICE_PRIVATE_WRITE:
165
+
case SOFTLEAF_DEVICE_PRIVATE_READ:
166
+
return true;
167
+
default:
168
+
return false;
169
+
}
170
+
}
171
+
172
+
/**
173
+
* softleaf_is_device_exclusive() - Is this leaf entry a device exclusive entry?
174
+
* @entry: Leaf entry.
175
+
*
176
+
* Returns: true if the leaf entry is a device exclusive entry, otherwise false.
177
+
*/
178
+
static inline bool softleaf_is_device_exclusive(softleaf_t entry)
179
+
{
180
+
return softleaf_type(entry) == SOFTLEAF_DEVICE_EXCLUSIVE;
181
+
}
182
+
183
+
/**
184
+
* softleaf_is_hwpoison() - Is this leaf entry a hardware poison entry?
185
+
* @entry: Leaf entry.
186
+
*
187
+
* Returns: true if the leaf entry is a hardware poison entry, otherwise false.
188
+
*/
189
+
static inline bool softleaf_is_hwpoison(softleaf_t entry)
190
+
{
191
+
return softleaf_type(entry) == SOFTLEAF_HWPOISON;
192
+
}
193
+
194
+
/**
195
+
* softleaf_is_marker() - Is this leaf entry a marker?
196
+
* @entry: Leaf entry.
197
+
*
198
+
* Returns: true if the leaf entry is a marker entry, otherwise false.
199
+
*/
200
+
static inline bool softleaf_is_marker(softleaf_t entry)
201
+
{
202
+
return softleaf_type(entry) == SOFTLEAF_MARKER;
203
+
}
204
+
205
+
/**
206
+
* softleaf_to_marker() - Obtain marker associated with leaf entry.
207
+
* @entry: Leaf entry, softleaf_is_marker(@entry) must return true.
208
+
*
209
+
* Returns: Marker associated with the leaf entry.
210
+
*/
211
+
static inline pte_marker softleaf_to_marker(softleaf_t entry)
212
+
{
213
+
VM_WARN_ON_ONCE(!softleaf_is_marker(entry));
214
+
215
+
return swp_offset(entry) & PTE_MARKER_MASK;
216
+
}
217
+
218
+
/**
219
+
* softleaf_has_pfn() - Does this leaf entry encode a valid PFN number?
220
+
* @entry: Leaf entry.
221
+
*
222
+
* A pfn swap entry is a special type of swap entry that always has a pfn stored
223
+
* in the swap offset. They can either be used to represent unaddressable device
224
+
* memory, to restrict access to a page undergoing migration or to represent a
225
+
* pfn which has been hwpoisoned and unmapped.
226
+
*
227
+
* Returns: true if the leaf entry encodes a PFN, otherwise false.
228
+
*/
229
+
static inline bool softleaf_has_pfn(softleaf_t entry)
230
+
{
231
+
/* Make sure the swp offset can always store the needed fields. */
232
+
BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS);
233
+
234
+
if (softleaf_is_migration(entry))
235
+
return true;
236
+
if (softleaf_is_device_private(entry))
237
+
return true;
238
+
if (softleaf_is_device_exclusive(entry))
239
+
return true;
240
+
if (softleaf_is_hwpoison(entry))
241
+
return true;
242
+
243
+
return false;
244
+
}
245
+
246
+
/**
247
+
* softleaf_to_pfn() - Obtain PFN encoded within leaf entry.
248
+
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
249
+
*
250
+
* Returns: The PFN associated with the leaf entry.
251
+
*/
252
+
static inline unsigned long softleaf_to_pfn(softleaf_t entry)
253
+
{
254
+
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
255
+
256
+
/* Temporary until swp_entry_t eliminated. */
257
+
return swp_offset_pfn(entry);
258
+
}
259
+
260
+
/**
261
+
* softleaf_to_page() - Obtains struct page for PFN encoded within leaf entry.
262
+
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
263
+
*
264
+
* Returns: Pointer to the struct page associated with the leaf entry's PFN.
265
+
*/
266
+
static inline struct page *softleaf_to_page(softleaf_t entry)
267
+
{
268
+
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
269
+
270
+
/* Temporary until swp_entry_t eliminated. */
271
+
return pfn_swap_entry_to_page(entry);
272
+
}
273
+
274
+
/**
275
+
* softleaf_to_folio() - Obtains struct folio for PFN encoded within leaf entry.
276
+
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
277
+
*
278
+
* Returns: Pointer to the struct folio associated with the leaf entry's PFN.
279
+
*/
280
+
static inline struct folio *softleaf_to_folio(softleaf_t entry)
281
+
{
282
+
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
283
+
284
+
/* Temporary until swp_entry_t eliminated. */
285
+
return pfn_swap_entry_folio(entry);
286
+
}
287
+
288
+
/**
289
+
* softleaf_is_poison_marker() - Is this leaf entry a poison marker?
290
+
* @entry: Leaf entry.
291
+
*
292
+
* The poison marker is set via UFFDIO_POISON. Userfaultfd-specific.
293
+
*
294
+
* Returns: true if the leaf entry is a poison marker, otherwise false.
295
+
*/
296
+
static inline bool softleaf_is_poison_marker(softleaf_t entry)
297
+
{
298
+
if (!softleaf_is_marker(entry))
299
+
return false;
300
+
301
+
return softleaf_to_marker(entry) & PTE_MARKER_POISONED;
302
+
}
303
+
304
+
/**
305
+
* softleaf_is_guard_marker() - Is this leaf entry a guard region marker?
306
+
* @entry: Leaf entry.
307
+
*
308
+
* Returns: true if the leaf entry is a guard marker, otherwise false.
309
+
*/
310
+
static inline bool softleaf_is_guard_marker(softleaf_t entry)
311
+
{
312
+
if (!softleaf_is_marker(entry))
313
+
return false;
314
+
315
+
return softleaf_to_marker(entry) & PTE_MARKER_GUARD;
316
+
}
317
+
318
+
/**
319
+
* softleaf_is_uffd_wp_marker() - Is this leaf entry a userfautlfd write protect
320
+
* marker?
321
+
* @entry: Leaf entry.
322
+
*
323
+
* Userfaultfd-specific.
324
+
*
325
+
* Returns: true if the leaf entry is a UFFD WP marker, otherwise false.
326
+
*/
327
+
static inline bool softleaf_is_uffd_wp_marker(softleaf_t entry)
328
+
{
329
+
if (!softleaf_is_marker(entry))
330
+
return false;
331
+
332
+
return softleaf_to_marker(entry) & PTE_MARKER_UFFD_WP;
333
+
}
334
+
335
+
/**
336
+
* pte_is_marker() - Does the PTE entry encode a marker leaf entry?
337
+
* @pte: PTE entry.
338
+
*
339
+
* Returns: true if this PTE is a marker leaf entry, otherwise false.
340
+
*/
341
+
static inline bool pte_is_marker(pte_t pte)
342
+
{
343
+
return softleaf_is_marker(softleaf_from_pte(pte));
344
+
}
345
+
346
+
/**
347
+
* pte_is_uffd_wp_marker() - Does this PTE entry encode a userfaultfd write
348
+
* protect marker leaf entry?
349
+
* @pte: PTE entry.
350
+
*
351
+
* Returns: true if this PTE is a UFFD WP marker leaf entry, otherwise false.
352
+
*/
353
+
static inline bool pte_is_uffd_wp_marker(pte_t pte)
354
+
{
355
+
const softleaf_t entry = softleaf_from_pte(pte);
356
+
357
+
return softleaf_is_uffd_wp_marker(entry);
358
+
}
359
+
360
+
/**
361
+
* pte_is_uffd_marker() - Does this PTE entry encode a userfault-specific marker
362
+
* leaf entry?
363
+
* @entry: Leaf entry.
364
+
*
365
+
* It's useful to be able to determine which leaf entries encode UFFD-specific
366
+
* markers so we can handle these correctly.
367
+
*
368
+
* Returns: true if this PTE entry is a UFFD-specific marker, otherwise false.
369
+
*/
370
+
static inline bool pte_is_uffd_marker(pte_t pte)
371
+
{
372
+
const softleaf_t entry = softleaf_from_pte(pte);
373
+
374
+
if (!softleaf_is_marker(entry))
375
+
return false;
376
+
377
+
/* UFFD WP, poisoned swap entries are UFFD-handled. */
378
+
if (softleaf_is_uffd_wp_marker(entry))
379
+
return true;
380
+
if (softleaf_is_poison_marker(entry))
381
+
return true;
382
+
383
+
return false;
384
+
}
385
+
386
+
#endif /* CONFIG_MMU */
387
+
#endif /* _LINUX_LEAFOPS_H */
+3
-3
include/linux/mm_inline.h
+3
-3
include/linux/mm_inline.h
···
8
8
#include <linux/swap.h>
9
9
#include <linux/string.h>
10
10
#include <linux/userfaultfd_k.h>
11
-
#include <linux/swapops.h>
11
+
#include <linux/leafops.h>
12
12
13
13
/**
14
14
* folio_is_file_lru - Should the folio be on a file LRU or anon LRU?
···
541
541
* The caller should insert a new pte created with make_pte_marker().
542
542
*/
543
543
static inline pte_marker copy_pte_marker(
544
-
swp_entry_t entry, struct vm_area_struct *dst_vma)
544
+
softleaf_t entry, struct vm_area_struct *dst_vma)
545
545
{
546
-
pte_marker srcm = pte_marker_get(entry);
546
+
const pte_marker srcm = softleaf_to_marker(entry);
547
547
/* Always copy error entries. */
548
548
pte_marker dstm = srcm & (PTE_MARKER_POISONED | PTE_MARKER_GUARD);
549
549
+25
include/linux/mm_types.h
+25
include/linux/mm_types.h
···
285
285
unsigned long val;
286
286
} swp_entry_t;
287
287
288
+
/**
289
+
* typedef softleaf_t - Describes a page table software leaf entry, abstracted
290
+
* from its architecture-specific encoding.
291
+
*
292
+
* Page table leaf entries are those which do not reference any descendent page
293
+
* tables but rather either reference a data page, are an empty (or 'none'
294
+
* entry), or contain a non-present entry.
295
+
*
296
+
* If referencing another page table or a data page then the page table entry is
297
+
* pertinent to hardware - that is it tells the hardware how to decode the page
298
+
* table entry.
299
+
*
300
+
* Otherwise it is a software-defined leaf page table entry, which this type
301
+
* describes. See leafops.h and specifically @softleaf_type for a list of all
302
+
* possible kinds of software leaf entry.
303
+
*
304
+
* A softleaf_t entry is abstracted from the hardware page table entry, so is
305
+
* not architecture-specific.
306
+
*
307
+
* NOTE: While we transition from the confusing swp_entry_t type used for this
308
+
* purpose, we simply alias this type. This will be removed once the
309
+
* transition is complete.
310
+
*/
311
+
typedef swp_entry_t softleaf_t;
312
+
288
313
#if defined(CONFIG_MEMCG) || defined(CONFIG_SLAB_OBJ_EXT)
289
314
/* We have some extra room after the refcount in tail pages. */
290
315
#define NR_PAGES_IN_LARGE_FOLIO
-28
include/linux/swapops.h
-28
include/linux/swapops.h
···
426
426
return swp_entry(SWP_PTE_MARKER, marker);
427
427
}
428
428
429
-
static inline bool is_pte_marker_entry(swp_entry_t entry)
430
-
{
431
-
return swp_type(entry) == SWP_PTE_MARKER;
432
-
}
433
-
434
-
static inline pte_marker pte_marker_get(swp_entry_t entry)
435
-
{
436
-
return swp_offset(entry) & PTE_MARKER_MASK;
437
-
}
438
-
439
-
static inline bool is_pte_marker(pte_t pte)
440
-
{
441
-
return is_swap_pte(pte) && is_pte_marker_entry(pte_to_swp_entry(pte));
442
-
}
443
-
444
429
static inline pte_t make_pte_marker(pte_marker marker)
445
430
{
446
431
return swp_entry_to_pte(make_pte_marker_entry(marker));
···
436
451
return make_pte_marker_entry(PTE_MARKER_POISONED);
437
452
}
438
453
439
-
static inline int is_poisoned_swp_entry(swp_entry_t entry)
440
-
{
441
-
return is_pte_marker_entry(entry) &&
442
-
(pte_marker_get(entry) & PTE_MARKER_POISONED);
443
-
444
-
}
445
-
446
454
static inline swp_entry_t make_guard_swp_entry(void)
447
455
{
448
456
return make_pte_marker_entry(PTE_MARKER_GUARD);
449
-
}
450
-
451
-
static inline int is_guard_swp_entry(swp_entry_t entry)
452
-
{
453
-
return is_pte_marker_entry(entry) &&
454
-
(pte_marker_get(entry) & PTE_MARKER_GUARD);
455
457
}
456
458
457
459
static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
+2
-49
include/linux/userfaultfd_k.h
+2
-49
include/linux/userfaultfd_k.h
···
16
16
#include <linux/fcntl.h>
17
17
#include <linux/mm.h>
18
18
#include <linux/swap.h>
19
-
#include <linux/swapops.h>
19
+
#include <linux/leafops.h>
20
20
#include <asm-generic/pgtable_uffd.h>
21
21
#include <linux/hugetlb_inline.h>
22
22
···
434
434
return userfaultfd_wp_unpopulated(vma);
435
435
}
436
436
437
-
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
438
-
{
439
-
#ifdef CONFIG_PTE_MARKER_UFFD_WP
440
-
return is_pte_marker_entry(entry) &&
441
-
(pte_marker_get(entry) & PTE_MARKER_UFFD_WP);
442
-
#else
443
-
return false;
444
-
#endif
445
-
}
446
-
447
-
static inline bool pte_marker_uffd_wp(pte_t pte)
448
-
{
449
-
#ifdef CONFIG_PTE_MARKER_UFFD_WP
450
-
swp_entry_t entry;
451
-
452
-
if (!is_swap_pte(pte))
453
-
return false;
454
-
455
-
entry = pte_to_swp_entry(pte);
456
-
457
-
return pte_marker_entry_uffd_wp(entry);
458
-
#else
459
-
return false;
460
-
#endif
461
-
}
462
-
463
437
/*
464
438
* Returns true if this is a swap pte and was uffd-wp wr-protected in either
465
439
* forms (pte marker or a normal swap pte), false otherwise.
···
447
473
if (pte_swp_uffd_wp(pte))
448
474
return true;
449
475
450
-
if (pte_marker_uffd_wp(pte))
476
+
if (pte_is_uffd_wp_marker(pte))
451
477
return true;
452
478
#endif
453
-
return false;
454
-
}
455
-
456
-
457
-
static inline bool is_uffd_pte_marker(pte_t pte)
458
-
{
459
-
swp_entry_t entry;
460
-
461
-
if (pte_present(pte))
462
-
return false;
463
-
464
-
entry = pte_to_swp_entry(pte);
465
-
if (!is_pte_marker_entry(entry))
466
-
return false;
467
-
468
-
/* UFFD WP, poisoned swap entries are UFFD handled. */
469
-
if (pte_marker_entry_uffd_wp(entry))
470
-
return true;
471
-
if (is_poisoned_swp_entry(entry))
472
-
return true;
473
-
474
479
return false;
475
480
}
476
481
+1
-1
mm/hmm.c
+1
-1
mm/hmm.c
···
249
249
* that will be correctly handled, so we need only check for UFFD WP
250
250
* here.
251
251
*/
252
-
if (pte_none(pte) || pte_marker_uffd_wp(pte)) {
252
+
if (pte_none(pte) || pte_is_uffd_wp_marker(pte)) {
253
253
required_fault =
254
254
hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0);
255
255
if (required_fault)
+19
-18
mm/hugetlb.c
+19
-18
mm/hugetlb.c
···
26
26
#include <linux/string_choices.h>
27
27
#include <linux/string_helpers.h>
28
28
#include <linux/swap.h>
29
-
#include <linux/swapops.h>
29
+
#include <linux/leafops.h>
30
30
#include <linux/jhash.h>
31
31
#include <linux/numa.h>
32
32
#include <linux/llist.h>
···
4956
4956
entry = huge_pte_clear_uffd_wp(entry);
4957
4957
set_huge_pte_at(dst, addr, dst_pte, entry, sz);
4958
4958
} else if (unlikely(is_hugetlb_entry_migration(entry))) {
4959
-
swp_entry_t swp_entry = pte_to_swp_entry(entry);
4959
+
softleaf_t softleaf = softleaf_from_pte(entry);
4960
4960
bool uffd_wp = pte_swp_uffd_wp(entry);
4961
4961
4962
-
if (!is_readable_migration_entry(swp_entry) && cow) {
4962
+
if (!is_readable_migration_entry(softleaf) && cow) {
4963
4963
/*
4964
4964
* COW mappings require pages in both
4965
4965
* parent and child to be set to read.
4966
4966
*/
4967
-
swp_entry = make_readable_migration_entry(
4968
-
swp_offset(swp_entry));
4969
-
entry = swp_entry_to_pte(swp_entry);
4967
+
softleaf = make_readable_migration_entry(
4968
+
swp_offset(softleaf));
4969
+
entry = swp_entry_to_pte(softleaf);
4970
4970
if (userfaultfd_wp(src_vma) && uffd_wp)
4971
4971
entry = pte_swp_mkuffd_wp(entry);
4972
4972
set_huge_pte_at(src, addr, src_pte, entry, sz);
···
4974
4974
if (!userfaultfd_wp(dst_vma))
4975
4975
entry = huge_pte_clear_uffd_wp(entry);
4976
4976
set_huge_pte_at(dst, addr, dst_pte, entry, sz);
4977
-
} else if (unlikely(is_pte_marker(entry))) {
4978
-
pte_marker marker = copy_pte_marker(
4979
-
pte_to_swp_entry(entry), dst_vma);
4977
+
} else if (unlikely(pte_is_marker(entry))) {
4978
+
const softleaf_t softleaf = softleaf_from_pte(entry);
4979
+
const pte_marker marker = copy_pte_marker(softleaf, dst_vma);
4980
4980
4981
4981
if (marker)
4982
4982
set_huge_pte_at(dst, addr, dst_pte,
···
5092
5092
5093
5093
pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz);
5094
5094
5095
-
if (need_clear_uffd_wp && pte_marker_uffd_wp(pte))
5095
+
if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))
5096
5096
huge_pte_clear(mm, new_addr, dst_pte, sz);
5097
5097
else {
5098
5098
if (need_clear_uffd_wp) {
···
5911
5911
* If this pte was previously wr-protected, keep it wr-protected even
5912
5912
* if populated.
5913
5913
*/
5914
-
if (unlikely(pte_marker_uffd_wp(vmf->orig_pte)))
5914
+
if (unlikely(pte_is_uffd_wp_marker(vmf->orig_pte)))
5915
5915
new_pte = huge_pte_mkuffd_wp(new_pte);
5916
5916
set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h));
5917
5917
···
6044
6044
*/
6045
6045
return hugetlb_no_page(mapping, &vmf);
6046
6046
6047
-
if (is_pte_marker(vmf.orig_pte)) {
6047
+
if (pte_is_marker(vmf.orig_pte)) {
6048
6048
const pte_marker marker =
6049
-
pte_marker_get(pte_to_swp_entry(vmf.orig_pte));
6049
+
softleaf_to_marker(softleaf_from_pte(vmf.orig_pte));
6050
6050
6051
6051
if (marker & PTE_MARKER_POISONED) {
6052
6052
ret = VM_FAULT_HWPOISON_LARGE |
···
6374
6374
* See comment about UFFD marker overwriting in
6375
6375
* mfill_atomic_install_pte().
6376
6376
*/
6377
-
if (!huge_pte_none(dst_ptep) && !is_uffd_pte_marker(dst_ptep))
6377
+
if (!huge_pte_none(dst_ptep) && !pte_is_uffd_marker(dst_ptep))
6378
6378
goto out_release_unlock;
6379
6379
6380
6380
if (folio_in_pagecache)
···
6495
6495
if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) {
6496
6496
/* Nothing to do. */
6497
6497
} else if (unlikely(is_hugetlb_entry_migration(pte))) {
6498
-
swp_entry_t entry = pte_to_swp_entry(pte);
6499
-
struct folio *folio = pfn_swap_entry_folio(entry);
6498
+
softleaf_t entry = softleaf_from_pte(pte);
6499
+
6500
+
struct folio *folio = softleaf_to_folio(entry);
6500
6501
pte_t newpte = pte;
6501
6502
6502
6503
if (is_writable_migration_entry(entry)) {
···
6517
6516
newpte = pte_swp_clear_uffd_wp(newpte);
6518
6517
if (!pte_same(pte, newpte))
6519
6518
set_huge_pte_at(mm, address, ptep, newpte, psize);
6520
-
} else if (unlikely(is_pte_marker(pte))) {
6519
+
} else if (unlikely(pte_is_marker(pte))) {
6521
6520
/*
6522
6521
* Do nothing on a poison marker; page is
6523
6522
* corrupted, permissions do not apply. Here
6524
6523
* pte_marker_uffd_wp()==true implies !poison
6525
6524
* because they're mutual exclusive.
6526
6525
*/
6527
-
if (pte_marker_uffd_wp(pte) && uffd_wp_resolve)
6526
+
if (pte_is_uffd_wp_marker(pte) && uffd_wp_resolve)
6528
6527
/* Safe to modify directly (non-present->none). */
6529
6528
huge_pte_clear(mm, address, ptep, psize);
6530
6529
} else if (!huge_pte_none(pte)) {
+8
-8
mm/madvise.c
+8
-8
mm/madvise.c
···
29
29
#include <linux/backing-dev.h>
30
30
#include <linux/pagewalk.h>
31
31
#include <linux/swap.h>
32
-
#include <linux/swapops.h>
32
+
#include <linux/leafops.h>
33
33
#include <linux/shmem_fs.h>
34
34
#include <linux/mmu_notifier.h>
35
35
···
690
690
* (page allocation + zeroing).
691
691
*/
692
692
if (!pte_present(ptent)) {
693
-
swp_entry_t entry;
693
+
softleaf_t entry = softleaf_from_pte(ptent);
694
694
695
-
entry = pte_to_swp_entry(ptent);
696
-
if (!non_swap_entry(entry)) {
695
+
if (softleaf_is_swap(entry)) {
697
696
max_nr = (end - addr) / PAGE_SIZE;
698
697
nr = swap_pte_batch(pte, max_nr, ptent);
699
698
nr_swap -= nr;
700
699
free_swap_and_cache_nr(entry, nr);
701
700
clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm);
702
-
} else if (is_hwpoison_entry(entry) ||
703
-
is_poisoned_swp_entry(entry)) {
701
+
} else if (softleaf_is_hwpoison(entry) ||
702
+
softleaf_is_poison_marker(entry)) {
704
703
pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
705
704
}
706
705
continue;
···
1070
1071
1071
1072
static bool is_guard_pte_marker(pte_t ptent)
1072
1073
{
1073
-
return is_swap_pte(ptent) &&
1074
-
is_guard_swp_entry(pte_to_swp_entry(ptent));
1074
+
const softleaf_t entry = softleaf_from_pte(ptent);
1075
+
1076
+
return softleaf_is_guard_marker(entry);
1075
1077
}
1076
1078
1077
1079
static int guard_install_pud_entry(pud_t *pud, unsigned long addr,
+21
-20
mm/memory.c
+21
-20
mm/memory.c
···
60
60
#include <linux/writeback.h>
61
61
#include <linux/memcontrol.h>
62
62
#include <linux/mmu_notifier.h>
63
-
#include <linux/swapops.h>
63
+
#include <linux/leafops.h>
64
64
#include <linux/elf.h>
65
65
#include <linux/gfp.h>
66
66
#include <linux/migrate.h>
···
109
109
if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID))
110
110
return false;
111
111
112
-
return pte_marker_uffd_wp(vmf->orig_pte);
112
+
return pte_is_uffd_wp_marker(vmf->orig_pte);
113
113
}
114
114
115
115
/*
···
927
927
{
928
928
vm_flags_t vm_flags = dst_vma->vm_flags;
929
929
pte_t orig_pte = ptep_get(src_pte);
930
+
softleaf_t entry = softleaf_from_pte(orig_pte);
930
931
pte_t pte = orig_pte;
931
932
struct folio *folio;
932
933
struct page *page;
933
-
swp_entry_t entry = pte_to_swp_entry(orig_pte);
934
934
935
935
if (likely(!non_swap_entry(entry))) {
936
936
if (swap_duplicate(entry) < 0)
···
1016
1016
if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte))
1017
1017
return -EBUSY;
1018
1018
return -ENOENT;
1019
-
} else if (is_pte_marker_entry(entry)) {
1019
+
} else if (softleaf_is_marker(entry)) {
1020
1020
pte_marker marker = copy_pte_marker(entry, dst_vma);
1021
1021
1022
1022
if (marker)
···
1711
1711
unsigned int max_nr, unsigned long addr,
1712
1712
struct zap_details *details, int *rss, bool *any_skipped)
1713
1713
{
1714
-
swp_entry_t entry;
1714
+
softleaf_t entry;
1715
1715
int nr = 1;
1716
1716
1717
1717
*any_skipped = true;
1718
-
entry = pte_to_swp_entry(ptent);
1719
-
if (is_device_private_entry(entry) ||
1720
-
is_device_exclusive_entry(entry)) {
1721
-
struct page *page = pfn_swap_entry_to_page(entry);
1718
+
entry = softleaf_from_pte(ptent);
1719
+
if (softleaf_is_device_private(entry) ||
1720
+
softleaf_is_device_exclusive(entry)) {
1721
+
struct page *page = softleaf_to_page(entry);
1722
1722
struct folio *folio = page_folio(page);
1723
1723
1724
1724
if (unlikely(!should_zap_folio(details, folio)))
···
1733
1733
rss[mm_counter(folio)]--;
1734
1734
folio_remove_rmap_pte(folio, page, vma);
1735
1735
folio_put(folio);
1736
-
} else if (!non_swap_entry(entry)) {
1736
+
} else if (softleaf_is_swap(entry)) {
1737
1737
/* Genuine swap entries, hence a private anon pages */
1738
1738
if (!should_zap_cows(details))
1739
1739
return 1;
···
1741
1741
nr = swap_pte_batch(pte, max_nr, ptent);
1742
1742
rss[MM_SWAPENTS] -= nr;
1743
1743
free_swap_and_cache_nr(entry, nr);
1744
-
} else if (is_migration_entry(entry)) {
1745
-
struct folio *folio = pfn_swap_entry_folio(entry);
1744
+
} else if (softleaf_is_migration(entry)) {
1745
+
struct folio *folio = softleaf_to_folio(entry);
1746
1746
1747
1747
if (!should_zap_folio(details, folio))
1748
1748
return 1;
1749
1749
rss[mm_counter(folio)]--;
1750
-
} else if (pte_marker_entry_uffd_wp(entry)) {
1750
+
} else if (softleaf_is_uffd_wp_marker(entry)) {
1751
1751
/*
1752
1752
* For anon: always drop the marker; for file: only
1753
1753
* drop the marker if explicitly requested.
1754
1754
*/
1755
1755
if (!vma_is_anonymous(vma) && !zap_drop_markers(details))
1756
1756
return 1;
1757
-
} else if (is_guard_swp_entry(entry)) {
1757
+
} else if (softleaf_is_guard_marker(entry)) {
1758
1758
/*
1759
1759
* Ordinary zapping should not remove guard PTE
1760
1760
* markers. Only do so if we should remove PTE markers
···
1762
1762
*/
1763
1763
if (!zap_drop_markers(details))
1764
1764
return 1;
1765
-
} else if (is_hwpoison_entry(entry) || is_poisoned_swp_entry(entry)) {
1765
+
} else if (softleaf_is_hwpoison(entry) ||
1766
+
softleaf_is_poison_marker(entry)) {
1766
1767
if (!should_zap_cows(details))
1767
1768
return 1;
1768
1769
} else {
···
4381
4380
*
4382
4381
* This should also cover the case where e.g. the pte changed
4383
4382
* quickly from a PTE_MARKER_UFFD_WP into PTE_MARKER_POISONED.
4384
-
* So is_pte_marker() check is not enough to safely drop the pte.
4383
+
* So pte_is_marker() check is not enough to safely drop the pte.
4385
4384
*/
4386
4385
if (pte_same(vmf->orig_pte, ptep_get(vmf->pte)))
4387
4386
pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte);
···
4415
4414
4416
4415
static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
4417
4416
{
4418
-
swp_entry_t entry = pte_to_swp_entry(vmf->orig_pte);
4419
-
unsigned long marker = pte_marker_get(entry);
4417
+
const softleaf_t entry = softleaf_from_pte(vmf->orig_pte);
4418
+
const pte_marker marker = softleaf_to_marker(entry);
4420
4419
4421
4420
/*
4422
4421
* PTE markers should never be empty. If anything weird happened,
···
4433
4432
if (marker & PTE_MARKER_GUARD)
4434
4433
return VM_FAULT_SIGSEGV;
4435
4434
4436
-
if (pte_marker_entry_uffd_wp(entry))
4435
+
if (softleaf_is_uffd_wp_marker(entry))
4437
4436
return pte_marker_handle_uffd_wp(vmf);
4438
4437
4439
4438
/* This is an unknown pte marker */
···
4681
4680
}
4682
4681
} else if (is_hwpoison_entry(entry)) {
4683
4682
ret = VM_FAULT_HWPOISON;
4684
-
} else if (is_pte_marker_entry(entry)) {
4683
+
} else if (softleaf_is_marker(entry)) {
4685
4684
ret = handle_pte_marker(vmf);
4686
4685
} else {
4687
4686
print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
+3
-3
mm/mincore.c
+3
-3
mm/mincore.c
···
14
14
#include <linux/mman.h>
15
15
#include <linux/syscalls.h>
16
16
#include <linux/swap.h>
17
-
#include <linux/swapops.h>
17
+
#include <linux/leafops.h>
18
18
#include <linux/shmem_fs.h>
19
19
#include <linux/hugetlb.h>
20
20
#include <linux/pgtable.h>
···
42
42
} else {
43
43
const pte_t ptep = huge_ptep_get(walk->mm, addr, pte);
44
44
45
-
if (huge_pte_none(ptep) || is_pte_marker(ptep))
45
+
if (huge_pte_none(ptep) || pte_is_marker(ptep))
46
46
present = 0;
47
47
else
48
48
present = 1;
···
187
187
188
188
step = 1;
189
189
/* We need to do cache lookup too for markers */
190
-
if (pte_none(pte) || is_pte_marker(pte))
190
+
if (pte_none(pte) || pte_is_marker(pte))
191
191
__mincore_unmapped_range(addr, addr + PAGE_SIZE,
192
192
vma, vec);
193
193
else if (pte_present(pte)) {
+3
-3
mm/mprotect.c
+3
-3
mm/mprotect.c
···
326
326
newpte = swp_entry_to_pte(entry);
327
327
if (pte_swp_uffd_wp(oldpte))
328
328
newpte = pte_swp_mkuffd_wp(newpte);
329
-
} else if (is_pte_marker_entry(entry)) {
329
+
} else if (softleaf_is_marker(entry)) {
330
330
/*
331
331
* Ignore error swap entries unconditionally,
332
332
* because any access should sigbus/sigsegv
333
333
* anyway.
334
334
*/
335
-
if (is_poisoned_swp_entry(entry) ||
336
-
is_guard_swp_entry(entry))
335
+
if (softleaf_is_poison_marker(entry) ||
336
+
softleaf_is_guard_marker(entry))
337
337
continue;
338
338
/*
339
339
* If this is uffd-wp pte marker and we'd like
+2
-2
mm/mremap.c
+2
-2
mm/mremap.c
···
17
17
#include <linux/swap.h>
18
18
#include <linux/capability.h>
19
19
#include <linux/fs.h>
20
-
#include <linux/swapops.h>
20
+
#include <linux/leafops.h>
21
21
#include <linux/highmem.h>
22
22
#include <linux/security.h>
23
23
#include <linux/syscalls.h>
···
288
288
pte = move_pte(pte, old_addr, new_addr);
289
289
pte = move_soft_dirty_pte(pte);
290
290
291
-
if (need_clear_uffd_wp && pte_marker_uffd_wp(pte))
291
+
if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))
292
292
pte_clear(mm, new_addr, new_ptep);
293
293
else {
294
294
if (need_clear_uffd_wp) {
+4
-7
mm/page_vma_mapped.c
+4
-7
mm/page_vma_mapped.c
···
3
3
#include <linux/rmap.h>
4
4
#include <linux/hugetlb.h>
5
5
#include <linux/swap.h>
6
-
#include <linux/swapops.h>
6
+
#include <linux/leafops.h>
7
7
8
8
#include "internal.h"
9
9
···
107
107
pte_t ptent = ptep_get(pvmw->pte);
108
108
109
109
if (pvmw->flags & PVMW_MIGRATION) {
110
-
swp_entry_t entry;
111
-
if (!is_swap_pte(ptent))
112
-
return false;
113
-
entry = pte_to_swp_entry(ptent);
110
+
const softleaf_t entry = softleaf_from_pte(ptent);
114
111
115
-
if (!is_migration_entry(entry))
112
+
if (!softleaf_is_migration(entry))
116
113
return false;
117
114
118
-
pfn = swp_offset_pfn(entry);
115
+
pfn = softleaf_to_pfn(entry);
119
116
} else if (is_swap_pte(ptent)) {
120
117
swp_entry_t entry;
121
118
+4
-3
mm/shmem.c
+4
-3
mm/shmem.c
···
66
66
#include <linux/falloc.h>
67
67
#include <linux/splice.h>
68
68
#include <linux/security.h>
69
-
#include <linux/swapops.h>
69
+
#include <linux/leafops.h>
70
70
#include <linux/mempolicy.h>
71
71
#include <linux/namei.h>
72
72
#include <linux/ctype.h>
···
2286
2286
struct address_space *mapping = inode->i_mapping;
2287
2287
struct mm_struct *fault_mm = vma ? vma->vm_mm : NULL;
2288
2288
struct shmem_inode_info *info = SHMEM_I(inode);
2289
-
swp_entry_t swap, index_entry;
2289
+
swp_entry_t swap;
2290
+
softleaf_t index_entry;
2290
2291
struct swap_info_struct *si;
2291
2292
struct folio *folio = NULL;
2292
2293
bool skip_swapcache = false;
···
2299
2298
swap = index_entry;
2300
2299
*foliop = NULL;
2301
2300
2302
-
if (is_poisoned_swp_entry(index_entry))
2301
+
if (softleaf_is_poison_marker(index_entry))
2303
2302
return -EIO;
2304
2303
2305
2304
si = get_swap_device(index_entry);
+3
-3
mm/userfaultfd.c
+3
-3
mm/userfaultfd.c
···
10
10
#include <linux/pagemap.h>
11
11
#include <linux/rmap.h>
12
12
#include <linux/swap.h>
13
-
#include <linux/swapops.h>
13
+
#include <linux/leafops.h>
14
14
#include <linux/userfaultfd_k.h>
15
15
#include <linux/mmu_notifier.h>
16
16
#include <linux/hugetlb.h>
···
208
208
* MISSING|WP registered, we firstly wr-protect a none pte which has no
209
209
* page cache page backing it, then access the page.
210
210
*/
211
-
if (!pte_none(dst_ptep) && !is_uffd_pte_marker(dst_ptep))
211
+
if (!pte_none(dst_ptep) && !pte_is_uffd_marker(dst_ptep))
212
212
goto out_unlock;
213
213
214
214
if (page_in_cache) {
···
590
590
if (!uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) {
591
591
const pte_t ptep = huge_ptep_get(dst_mm, dst_addr, dst_pte);
592
592
593
-
if (!huge_pte_none(ptep) && !is_uffd_pte_marker(ptep)) {
593
+
if (!huge_pte_none(ptep) && !pte_is_uffd_marker(ptep)) {
594
594
err = -EEXIST;
595
595
hugetlb_vma_unlock_read(dst_vma);
596
596
mutex_unlock(&hugetlb_fault_mutex_table[hash]);