tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm/mprotect: move softleaf code out of the main function

Patch series "mm/mprotect: micro-optimization work", v3.

Micro-optimize the change_protection functionality and the
change_pte_range() routine. This set of functions works in an incredibly
tight loop, and even small inefficiencies are incredibly evident when spun
hundreds, thousands or hundreds of thousands of times.

There was an attempt to keep the batching functionality as much as
possible, which introduced some part of the slowness, but not all of it.
Removing it for !arm64 architectures would speed mprotect() up even
further, but could easily pessimize cases where large folios are mapped
(which is not as rare as it seems, particularly when it comes to the page
cache these days).

The micro-benchmark used for the tests was [0] (usable using
google/benchmark and g++ -O2 -lbenchmark repro.cpp)

This resulted in the following (first entry is baseline):

---------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------
mprotect_bench 85967 ns 85967 ns 6935
mprotect_bench 70684 ns 70684 ns 9887


After the patchset we can observe an ~18% speedup in mprotect. Wonderful
for the elusive mprotect-based workloads!

Testing & more ideas welcome. I suspect there is plenty of improvement
possible but it would require more time than what I have on my hands right
now. The entire inlined function (which inlines into change_protection())
is gigantic - I'm not surprised this is so finnicky.

Note: per my profiling, the next _big_ bottleneck here is
modify_prot_start_ptes, exactly on the xchg() done by x86.
ptep_get_and_clear() is _expensive_. I don't think there's a properly
safe way to go about it since we do depend on the D bit quite a lot. This
might not be such an issue on other architectures.

Luke Yang reported [1]:

: On average, we see improvements ranging from a minimum of 5% to a
: maximum of 55%, with most improvements showing around a 25% speed up in
: the libmicro/mprot_tw4m micro benchmark.


This patch (of 2):

Move softleaf change_pte_range code into a separate function. This makes
the change_pte_range() function a good bit smaller, and lessens cognitive
load when reading through the function.

Link: https://lore.kernel.org/20260402141628.3367596-1-pfalcato@suse.de
Link: https://lore.kernel.org/20260402141628.3367596-2-pfalcato@suse.de
Link: https://lore.kernel.org/all/aY8-XuFZ7zCvXulB@luyang-thinkpadp1gen7.toromso.csb/
Link: https://gist.github.com/heatd/1450d273005aba91fa5744f44dfcd933 [0]
Link: https://lore.kernel.org/CAL2CeBxT4jtJ+LxYb6=BNxNMGinpgD_HYH5gGxOP-45Q2OncqQ@mail.gmail.com [1]
Signed-off-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Luke Yang <luyang@redhat.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jiri Hladky <jhladky@redhat.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Pedro Falcato and committed by
Andrew Morton 3bc181c1 19999e47

+67 -60
+67 -60
mm/mprotect.c
··· 211 211 commit_anon_folio_batch(vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb); 212 212 } 213 213 214 + static long change_softleaf_pte(struct vm_area_struct *vma, 215 + unsigned long addr, pte_t *pte, pte_t oldpte, unsigned long cp_flags) 216 + { 217 + const bool uffd_wp = cp_flags & MM_CP_UFFD_WP; 218 + const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; 219 + softleaf_t entry = softleaf_from_pte(oldpte); 220 + pte_t newpte; 221 + 222 + if (softleaf_is_migration_write(entry)) { 223 + const struct folio *folio = softleaf_to_folio(entry); 224 + 225 + /* 226 + * A protection check is difficult so 227 + * just be safe and disable write 228 + */ 229 + if (folio_test_anon(folio)) 230 + entry = make_readable_exclusive_migration_entry(swp_offset(entry)); 231 + else 232 + entry = make_readable_migration_entry(swp_offset(entry)); 233 + newpte = swp_entry_to_pte(entry); 234 + if (pte_swp_soft_dirty(oldpte)) 235 + newpte = pte_swp_mksoft_dirty(newpte); 236 + } else if (softleaf_is_device_private_write(entry)) { 237 + /* 238 + * We do not preserve soft-dirtiness. See 239 + * copy_nonpresent_pte() for explanation. 240 + */ 241 + entry = make_readable_device_private_entry(swp_offset(entry)); 242 + newpte = swp_entry_to_pte(entry); 243 + if (pte_swp_uffd_wp(oldpte)) 244 + newpte = pte_swp_mkuffd_wp(newpte); 245 + } else if (softleaf_is_marker(entry)) { 246 + /* 247 + * Ignore error swap entries unconditionally, 248 + * because any access should sigbus/sigsegv 249 + * anyway. 250 + */ 251 + if (softleaf_is_poison_marker(entry) || 252 + softleaf_is_guard_marker(entry)) 253 + return 0; 254 + /* 255 + * If this is uffd-wp pte marker and we'd like 256 + * to unprotect it, drop it; the next page 257 + * fault will trigger without uffd trapping. 258 + */ 259 + if (uffd_wp_resolve) { 260 + pte_clear(vma->vm_mm, addr, pte); 261 + return 1; 262 + } 263 + return 0; 264 + } else { 265 + newpte = oldpte; 266 + } 267 + 268 + if (uffd_wp) 269 + newpte = pte_swp_mkuffd_wp(newpte); 270 + else if (uffd_wp_resolve) 271 + newpte = pte_swp_clear_uffd_wp(newpte); 272 + 273 + if (!pte_same(oldpte, newpte)) { 274 + set_pte_at(vma->vm_mm, addr, pte, newpte); 275 + return 1; 276 + } 277 + return 0; 278 + } 279 + 214 280 static long change_pte_range(struct mmu_gather *tlb, 215 281 struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, 216 282 unsigned long end, pgprot_t newprot, unsigned long cp_flags) ··· 383 317 pages++; 384 318 } 385 319 } else { 386 - softleaf_t entry = softleaf_from_pte(oldpte); 387 - pte_t newpte; 388 - 389 - if (softleaf_is_migration_write(entry)) { 390 - const struct folio *folio = softleaf_to_folio(entry); 391 - 392 - /* 393 - * A protection check is difficult so 394 - * just be safe and disable write 395 - */ 396 - if (folio_test_anon(folio)) 397 - entry = make_readable_exclusive_migration_entry( 398 - swp_offset(entry)); 399 - else 400 - entry = make_readable_migration_entry(swp_offset(entry)); 401 - newpte = swp_entry_to_pte(entry); 402 - if (pte_swp_soft_dirty(oldpte)) 403 - newpte = pte_swp_mksoft_dirty(newpte); 404 - } else if (softleaf_is_device_private_write(entry)) { 405 - /* 406 - * We do not preserve soft-dirtiness. See 407 - * copy_nonpresent_pte() for explanation. 408 - */ 409 - entry = make_readable_device_private_entry( 410 - swp_offset(entry)); 411 - newpte = swp_entry_to_pte(entry); 412 - if (pte_swp_uffd_wp(oldpte)) 413 - newpte = pte_swp_mkuffd_wp(newpte); 414 - } else if (softleaf_is_marker(entry)) { 415 - /* 416 - * Ignore error swap entries unconditionally, 417 - * because any access should sigbus/sigsegv 418 - * anyway. 419 - */ 420 - if (softleaf_is_poison_marker(entry) || 421 - softleaf_is_guard_marker(entry)) 422 - continue; 423 - /* 424 - * If this is uffd-wp pte marker and we'd like 425 - * to unprotect it, drop it; the next page 426 - * fault will trigger without uffd trapping. 427 - */ 428 - if (uffd_wp_resolve) { 429 - pte_clear(vma->vm_mm, addr, pte); 430 - pages++; 431 - } 432 - continue; 433 - } else { 434 - newpte = oldpte; 435 - } 436 - 437 - if (uffd_wp) 438 - newpte = pte_swp_mkuffd_wp(newpte); 439 - else if (uffd_wp_resolve) 440 - newpte = pte_swp_clear_uffd_wp(newpte); 441 - 442 - if (!pte_same(oldpte, newpte)) { 443 - set_pte_at(vma->vm_mm, addr, pte, newpte); 444 - pages++; 445 - } 320 + pages += change_softleaf_pte(vma, addr, pte, oldpte, cp_flags); 446 321 } 447 322 } while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end); 448 323 lazy_mmu_mode_disable();