unwind deferred: Use SRCU unwind_deferred_task_work()

Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

kernel os linux

Instead of using the callback_mutex to protect the link list of callbacks
in unwind_deferred_task_work(), use SRCU instead. This gets called every
time a task exits that has to record a stack trace that was requested.
This can happen for many tasks on several CPUs at the same time. A mutex
is a bottleneck and can cause a bit of contention and slow down performance.

As the callbacks themselves are allowed to sleep, regular RCU cannot be
used to protect the list. Instead use SRCU, as that still allows the
callbacks to sleep and the list can be read without needing to hold the
callback_mutex.

Link: https://lore.kernel.org/all/ca9bd83a-6c80-4ee0-a83c-224b9d60b755@efficios.com/

Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Indu Bhagat <indu.bhagat@oracle.com>
Cc: "Jose E. Marchesi" <jemarch@gnu.org>
Cc: Beau Belgrave <beaub@linux.microsoft.com>
Cc: Jens Remus <jremus@linux.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Florian Weimer <fweimer@redhat.com>
Cc: Sam James <sam@gentoo.org>
Link: https://lore.kernel.org/20250729182406.331548065@kernel.org
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>

Steven Rostedt 10 months ago 357eda2d 858fa8a3

+21 -6

1 changed file

expand all

kernel

unwind

deferred.c

+21 -6

kernel/unwind/deferred.c

··· 41 41 #define UNWIND_MAX_ENTRIES \ 42 42 ((SZ_4K - sizeof(struct unwind_cache)) / sizeof(long)) 43 43 44 - /* Guards adding to and reading the list of callbacks */ 44 + /* Guards adding to or removing from the list of callbacks */ 45 45 static DEFINE_MUTEX(callback_mutex); 46 46 static LIST_HEAD(callbacks); 47 47 ··· 49 49 50 50 /* Zero'd bits are available for assigning callback users */ 51 51 static unsigned long unwind_mask = RESERVED_BITS; 52 + DEFINE_STATIC_SRCU(unwind_srcu); 52 53 53 54 static inline bool unwind_pending(struct unwind_task_info *info) 54 55 { ··· 175 174 176 175 cookie = info->id.id; 177 176 178 - guard(mutex)(&callback_mutex); 179 - list_for_each_entry(work, &callbacks, list) { 177 + guard(srcu)(&unwind_srcu); 178 + list_for_each_entry_srcu(work, &callbacks, list, 179 + srcu_read_lock_held(&unwind_srcu)) { 180 180 if (test_bit(work->bit, &bits)) { 181 181 work->func(work, &trace, cookie); 182 182 if (info->cache) ··· 215 213 { 216 214 struct unwind_task_info *info = &current->unwind_info; 217 215 unsigned long old, bits; 218 - unsigned long bit = BIT(work->bit); 216 + unsigned long bit; 219 217 int ret; 220 218 221 219 *cookie = 0; ··· 231 229 */ 232 230 if (WARN_ON_ONCE(!CAN_USE_IN_NMI && in_nmi())) 233 231 return -EINVAL; 232 + 233 + /* Do not allow cancelled works to request again */ 234 + bit = READ_ONCE(work->bit); 235 + if (WARN_ON_ONCE(bit < 0)) 236 + return -EINVAL; 237 + 238 + /* Only need the mask now */ 239 + bit = BIT(bit); 234 240 235 241 guard(irqsave)(); 236 242 ··· 291 281 return; 292 282 293 283 guard(mutex)(&callback_mutex); 294 - list_del(&work->list); 284 + list_del_rcu(&work->list); 285 + 286 + /* Do not allow any more requests and prevent callbacks */ 287 + work->bit = -1; 295 288 296 289 __clear_bit(bit, &unwind_mask); 290 + 291 + synchronize_srcu(&unwind_srcu); 297 292 298 293 guard(rcu)(); 299 294 /* Clear this bit from all threads */ ··· 322 307 work->bit = ffz(unwind_mask); 323 308 __set_bit(work->bit, &unwind_mask); 324 309 325 - list_add(&work->list, &callbacks); 310 + list_add_rcu(&work->list, &callbacks); 326 311 work->func = func; 327 312 return 0; 328 313 }

Configure Feed

Configure Feed