mm: implement sticky VMA flags

It is useful to be able to designate that certain flags are 'sticky', that
is, if two VMAs are merged one with a flag of this nature and one without,
the merged VMA sets this flag.

As a result we ignore these flags for the purposes of determining VMA flag
differences between VMAs being considered for merge.

This patch therefore updates the VMA merge logic to perform this action,
with flags possessing this property being described in the VM_STICKY
bitmap.

Those flags which ought to be ignored for the purposes of VMA merge are
described in the VM_IGNORE_MERGE bitmap, which the VMA merge logic is also
updated to use.

As part of this change we place VM_SOFTDIRTY in VM_IGNORE_MERGE as it
already had this behaviour, alongside VM_STICKY as sticky flags by
implication must not disallow merge.

Ultimately it seems that we should make VM_SOFTDIRTY a sticky flag in its
own right, but this change is out of scope for this series.

The only sticky flag designated as such is VM_MAYBE_GUARD, so as a result
of this change, once the VMA flag is set upon guard region installation,
VMAs with guard ranges will now not have their merge behaviour impacted as
a result and can be freely merged with other VMAs without VM_MAYBE_GUARD
set.

Also update the comments for vma_modify_flags() to directly reference
sticky flags now we have established the concept.

We also update the VMA userland tests to account for the changes.

Link: https://lkml.kernel.org/r/22ad5269f7669d62afb42ce0c79bad70b994c58d.1763460113.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Andrei Vagin <avagin@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes and committed by

Andrew Morton 6 months ago 64212ba0 9119d6c2

+75 -19

4 changed files

expand all

include

linux

mm.h

vma.c

vma.h

tools

testing

vma

vma_internal.h

+28

include/linux/mm.h

··· 528 528 #define VM_FLAGS_CLEAR (ARCH_VM_PKEY_FLAGS | VM_ARCH_CLEAR) 529 529 530 530 /* 531 + * Flags which should be 'sticky' on merge - that is, flags which, when one VMA 532 + * possesses it but the other does not, the merged VMA should nonetheless have 533 + * applied to it: 534 + * 535 + * VM_MAYBE_GUARD - If a VMA may have guard regions in place it implies that 536 + * mapped page tables may contain metadata not described by the 537 + * VMA and thus any merged VMA may also contain this metadata, 538 + * and thus we must make this flag sticky. 539 + */ 540 + #define VM_STICKY VM_MAYBE_GUARD 541 + 542 + /* 543 + * VMA flags we ignore for the purposes of merge, i.e. one VMA possessing one 544 + * of these flags and the other not does not preclude a merge. 545 + * 546 + * VM_SOFTDIRTY - Should not prevent from VMA merging, if we match the flags but 547 + * dirty bit -- the caller should mark merged VMA as dirty. If 548 + * dirty bit won't be excluded from comparison, we increase 549 + * pressure on the memory system forcing the kernel to generate 550 + * new VMAs when old one could be extended instead. 551 + * 552 + * VM_STICKY - When merging VMAs, VMA flags must match, unless they are 553 + * 'sticky'. If any sticky flags exist in either VMA, we simply 554 + * set all of them on the merged VMA. 555 + */ 556 + #define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY) 557 + 558 + /* 531 559 * mapping from the currently active vm_flags protection bits (the 532 560 * low four bits) to a page protection mask.. 533 561 */

+15 -13

mm/vma.c

··· 89 89 90 90 if (!mpol_equal(vmg->policy, vma_policy(vma))) 91 91 return false; 92 - /* 93 - * VM_SOFTDIRTY should not prevent from VMA merging, if we 94 - * match the flags but dirty bit -- the caller should mark 95 - * merged VMA as dirty. If dirty bit won't be excluded from 96 - * comparison, we increase pressure on the memory system forcing 97 - * the kernel to generate new VMAs when old one could be 98 - * extended instead. 99 - */ 100 - if ((vma->vm_flags ^ vmg->vm_flags) & ~VM_SOFTDIRTY) 92 + if ((vma->vm_flags ^ vmg->vm_flags) & ~VM_IGNORE_MERGE) 101 93 return false; 102 94 if (vma->vm_file != vmg->file) 103 95 return false; ··· 800 808 static __must_check struct vm_area_struct *vma_merge_existing_range( 801 809 struct vma_merge_struct *vmg) 802 810 { 811 + vm_flags_t sticky_flags = vmg->vm_flags & VM_STICKY; 803 812 struct vm_area_struct *middle = vmg->middle; 804 813 struct vm_area_struct *prev = vmg->prev; 805 814 struct vm_area_struct *next; ··· 893 900 if (merge_right) { 894 901 vma_start_write(next); 895 902 vmg->target = next; 903 + sticky_flags |= (next->vm_flags & VM_STICKY); 896 904 } 897 905 898 906 if (merge_left) { 899 907 vma_start_write(prev); 900 908 vmg->target = prev; 909 + sticky_flags |= (prev->vm_flags & VM_STICKY); 901 910 } 902 911 903 912 if (merge_both) { ··· 969 974 if (err || commit_merge(vmg)) 970 975 goto abort; 971 976 977 + vm_flags_set(vmg->target, sticky_flags); 972 978 khugepaged_enter_vma(vmg->target, vmg->vm_flags); 973 979 vmg->state = VMA_MERGE_SUCCESS; 974 980 return vmg->target; ··· 1120 1124 bool remove_next = false; 1121 1125 struct vm_area_struct *target = vmg->target; 1122 1126 struct vm_area_struct *next = vmg->next; 1127 + vm_flags_t sticky_flags; 1128 + 1129 + sticky_flags = vmg->vm_flags & VM_STICKY; 1130 + sticky_flags |= target->vm_flags & VM_STICKY; 1123 1131 1124 1132 VM_WARN_ON_VMG(!target, vmg); 1125 1133 ··· 1133 1133 if (next && (target != next) && (vmg->end == next->vm_end)) { 1134 1134 int ret; 1135 1135 1136 + sticky_flags |= next->vm_flags & VM_STICKY; 1136 1137 remove_next = true; 1137 1138 /* This should already have been checked by this point. */ 1138 1139 VM_WARN_ON_VMG(!can_merge_remove_vma(next), vmg); ··· 1160 1159 if (commit_merge(vmg)) 1161 1160 goto nomem; 1162 1161 1162 + vm_flags_set(target, sticky_flags); 1163 1163 return 0; 1164 1164 1165 1165 nomem: ··· 1656 1654 return ret; 1657 1655 1658 1656 /* 1659 - * For a merge to succeed, the flags must match those requested. For 1660 - * flags which do not obey typical merge rules (i.e. do not need to 1661 - * match), we must let the caller know about them. 1657 + * For a merge to succeed, the flags must match those 1658 + * requested. However, sticky flags may have been retained, so propagate 1659 + * them to the caller. 1662 1660 */ 1663 1661 if (vmg.state == VMA_MERGE_SUCCESS) 1664 1662 *vm_flags_ptr = ret->vm_flags; ··· 1908 1906 return a->vm_end == b->vm_start && 1909 1907 mpol_equal(vma_policy(a), vma_policy(b)) && 1910 1908 a->vm_file == b->vm_file && 1911 - !((a->vm_flags ^ b->vm_flags) & ~(VM_ACCESS_FLAGS | VM_SOFTDIRTY)) && 1909 + !((a->vm_flags ^ b->vm_flags) & ~(VM_ACCESS_FLAGS | VM_IGNORE_MERGE)) && 1912 1910 b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT); 1913 1911 } 1914 1912

+4 -6

mm/vma.h

··· 273 273 * @start: The start of the range to update. May be offset within @vma. 274 274 * @end: The exclusive end of the range to update, may be offset within @vma. 275 275 * @vm_flags_ptr: A pointer to the VMA flags that the @start to @end range is 276 - * about to be set to. On merge, this will be updated to include any additional 277 - * flags which remain in place. 276 + * about to be set to. On merge, this will be updated to include sticky flags. 278 277 * 279 278 * IMPORTANT: The actual modification being requested here is NOT applied, 280 279 * rather the VMA is perhaps split, perhaps merged to accommodate the change, 281 280 * and the caller is expected to perform the actual modification. 282 281 * 283 - * In order to account for VMA flags which may persist (e.g. soft-dirty), the 284 - * @vm_flags_ptr parameter points to the requested flags which are then updated 285 - * so the caller, should they overwrite any existing flags, correctly retains 286 - * these. 282 + * In order to account for sticky VMA flags, the @vm_flags_ptr parameter points 283 + * to the requested flags which are then updated so the caller, should they 284 + * overwrite any existing flags, correctly retains these. 287 285 * 288 286 * Returns: A VMA which contains the range @start to @end ready to have its 289 287 * flags altered to *@vm_flags.

+28

tools/testing/vma/vma_internal.h

··· 117 117 #define VM_SEALED VM_NONE 118 118 #endif 119 119 120 + /* 121 + * Flags which should be 'sticky' on merge - that is, flags which, when one VMA 122 + * possesses it but the other does not, the merged VMA should nonetheless have 123 + * applied to it: 124 + * 125 + * VM_MAYBE_GUARD - If a VMA may have guard regions in place it implies that 126 + * mapped page tables may contain metadata not described by the 127 + * VMA and thus any merged VMA may also contain this metadata, 128 + * and thus we must make this flag sticky. 129 + */ 130 + #define VM_STICKY VM_MAYBE_GUARD 131 + 132 + /* 133 + * VMA flags we ignore for the purposes of merge, i.e. one VMA possessing one 134 + * of these flags and the other not does not preclude a merge. 135 + * 136 + * VM_SOFTDIRTY - Should not prevent from VMA merging, if we match the flags but 137 + * dirty bit -- the caller should mark merged VMA as dirty. If 138 + * dirty bit won't be excluded from comparison, we increase 139 + * pressure on the memory system forcing the kernel to generate 140 + * new VMAs when old one could be extended instead. 141 + * 142 + * VM_STICKY - When merging VMAs, VMA flags must match, unless they are 143 + * 'sticky'. If any sticky flags exist in either VMA, we simply 144 + * set all of them on the merged VMA. 145 + */ 146 + #define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY) 147 + 120 148 #define FIRST_USER_ADDRESS 0UL 121 149 #define USER_PGTABLES_CEILING 0UL 122 150

Configure Feed

Configure Feed