mm/vma: clean up __vma_enter/exit_locked()

These functions are very confusing indeed. 'Entering' a lock could be
interpreted as acquiring it, but this is not what these functions are
interacting with.

Equally they don't indicate at all what kind of lock we are 'entering' or
'exiting'. Finally they are misleading as we invoke these functions when
we already hold a write lock to detach a VMA.

These functions are explicitly simply 'entering' and 'exiting' a state in
which we hold the EXCLUSIVE lock in order that we can either mark the VMA
as being write-locked, or mark the VMA detached.

Rename the functions accordingly, and also update
__vma_end_exclude_readers() to return detached state with a __must_check
directive, as it is simply clumsy to pass an output pointer here to
detached state and inconsistent vs. __vma_start_exclude_readers().

Finally, remove the unnecessary 'inline' directives.

No functional change intended.

Link: https://lkml.kernel.org/r/33273be9389712347d69987c408ca7436f0c1b22.1769198904.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Waiman Long <longman@redhat.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes and committed by

Andrew Morton 3 months ago 28f590f3 e5aeb75d

+42 -20

2 changed files

expand all

include

linux

mmap_lock.h

mmap_lock.c

+2 -2

include/linux/mmap_lock.h

··· 211 211 newcnt = __vma_refcount_put_return(vma); 212 212 213 213 /* 214 - * __vma_enter_locked() may be sleeping waiting for readers to drop 215 - * their reference count, so wake it up if we were the last reader 214 + * __vma_start_exclude_readers() may be sleeping waiting for readers to 215 + * drop their reference count, so wake it up if we were the last reader 216 216 * blocking it from being acquired. 217 217 * 218 218 * We may be raced by other readers temporarily incrementing the

+40 -18

mm/mmap_lock.c

··· 46 46 #ifdef CONFIG_MMU 47 47 #ifdef CONFIG_PER_VMA_LOCK 48 48 49 - static inline void __vma_exit_locked(struct vm_area_struct *vma, bool *detached) 49 + /* 50 + * Now that all readers have been evicted, mark the VMA as being out of the 51 + * 'exclude readers' state. 52 + * 53 + * Returns true if the VMA is now detached, otherwise false. 54 + */ 55 + static bool __must_check __vma_end_exclude_readers(struct vm_area_struct *vma) 50 56 { 51 - *detached = refcount_sub_and_test(VM_REFCNT_EXCLUDE_READERS_FLAG, 52 - &vma->vm_refcnt); 57 + bool detached; 58 + 59 + detached = refcount_sub_and_test(VM_REFCNT_EXCLUDE_READERS_FLAG, 60 + &vma->vm_refcnt); 53 61 __vma_lockdep_release_exclusive(vma); 62 + return detached; 54 63 } 55 64 56 65 /* 57 - * __vma_enter_locked() returns 0 immediately if the vma is not 58 - * attached, otherwise it waits for any current readers to finish and 59 - * returns 1. Returns -EINTR if a signal is received while waiting. 66 + * Mark the VMA as being in a state of excluding readers, check to see if any 67 + * VMA read locks are indeed held, and if so wait for them to be released. 68 + * 69 + * Note that this function pairs with vma_refcount_put() which will wake up this 70 + * thread when it detects that the last reader has released its lock. 71 + * 72 + * The state parameter ought to be set to TASK_UNINTERRUPTIBLE in cases where we 73 + * wish the thread to sleep uninterruptibly or TASK_KILLABLE if a fatal signal 74 + * is permitted to kill it. 75 + * 76 + * The function will return 0 immediately if the VMA is detached, or wait for 77 + * readers and return 1 once they have all exited, leaving the VMA exclusively 78 + * locked. 79 + * 80 + * If the function returns 1, the caller is required to invoke 81 + * __vma_end_exclude_readers() once the exclusive state is no longer required. 82 + * 83 + * If state is set to something other than TASK_UNINTERRUPTIBLE, the function 84 + * may also return -EINTR to indicate a fatal signal was received while waiting. 60 85 */ 61 - static inline int __vma_enter_locked(struct vm_area_struct *vma, 86 + static int __vma_start_exclude_readers(struct vm_area_struct *vma, 62 87 bool detaching, int state) 63 88 { 64 89 int err; ··· 110 85 refcount_read(&vma->vm_refcnt) == tgt_refcnt, 111 86 state); 112 87 if (err) { 113 - bool detached; 114 - 115 - __vma_exit_locked(vma, &detached); 116 - if (detached) { 88 + if (__vma_end_exclude_readers(vma)) { 117 89 /* 118 90 * The wait failed, but the last reader went away 119 - * as well. Tell the caller the VMA is detached. 91 + * as well. Tell the caller the VMA is detached. 120 92 */ 121 93 WARN_ON_ONCE(!detaching); 122 94 err = 0; ··· 130 108 { 131 109 int locked; 132 110 133 - locked = __vma_enter_locked(vma, false, state); 111 + locked = __vma_start_exclude_readers(vma, false, state); 134 112 if (locked < 0) 135 113 return locked; 136 114 ··· 143 121 WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); 144 122 145 123 if (locked) { 146 - bool detached; 124 + bool detached = __vma_end_exclude_readers(vma); 147 125 148 - __vma_exit_locked(vma, &detached); 149 - WARN_ON_ONCE(detached); /* vma should remain attached */ 126 + /* The VMA should remain attached. */ 127 + WARN_ON_ONCE(detached); 150 128 } 151 129 152 130 return 0; ··· 170 148 */ 171 149 if (unlikely(__vma_refcount_put_return(vma))) { 172 150 /* Wait until vma is detached with no readers. */ 173 - if (__vma_enter_locked(vma, true, TASK_UNINTERRUPTIBLE)) { 151 + if (__vma_start_exclude_readers(vma, true, TASK_UNINTERRUPTIBLE)) { 174 152 bool detached; 175 153 176 154 /* 177 155 * Once this is complete, no readers can increment the 178 156 * reference count, and the VMA is marked detached. 179 157 */ 180 - __vma_exit_locked(vma, &detached); 158 + detached = __vma_end_exclude_readers(vma); 181 159 WARN_ON_ONCE(!detached); 182 160 } 183 161 }

Configure Feed

Configure Feed