tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

unwind_user/deferred: Add deferred unwinding interface

Add an interface for scheduling task work to unwind the user space stack
before returning to user space. This solves several problems for its
callers:

- Ensure the unwind happens in task context even if the caller may be
running in interrupt context.

- Avoid duplicate unwinds, whether called multiple times by the same
caller or by different callers.

- Create a "context cookie" which allows trace post-processing to
correlate kernel unwinds/traces with the user unwind.

A concept of a "cookie" is created to detect when the stacktrace is the
same. A cookie is generated the first time a user space stacktrace is
requested after the task enters the kernel. As the stacktrace is saved on
the task_struct while the task is in the kernel, if another request comes
in, if the cookie is still the same, it will use the saved stacktrace,
and not have to regenerate one.

The cookie is passed to the caller on request, and when the stacktrace is
generated upon returning to user space, it calls the requester's callback
with the cookie as well as the stacktrace. The cookie is cleared
when it goes back to user space. Note, this currently adds another
conditional to the unwind_reset_info() path that is always called
returning to user space, but future changes will put this back to a single
conditional.

A global list is created and protected by a global mutex that holds
tracers that register with the unwind infrastructure. The number of
registered tracers will be limited in future changes. Each perf program or
ftrace instance will register its own descriptor to use for deferred
unwind stack traces.

Note, in the function unwind_deferred_task_work() that gets called when
returning to user space, it uses a global mutex for synchronization which
will cause a big bottleneck. This will be replaced by SRCU, but that
change adds some complex synchronization that deservers its own commit.

Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Indu Bhagat <indu.bhagat@oracle.com>
Cc: "Jose E. Marchesi" <jemarch@gnu.org>
Cc: Beau Belgrave <beaub@linux.microsoft.com>
Cc: Jens Remus <jremus@linux.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Florian Weimer <fweimer@redhat.com>
Cc: Sam James <sam@gentoo.org>
Link: https://lore.kernel.org/20250729182405.488066537@kernel.org
Co-developed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>

authored by

Josh Poimboeuf and committed by
Steven Rostedt (Google) 2dffa355 b9c73524

+203 -1
+24
include/linux/unwind_deferred.h
··· 2 2 #ifndef _LINUX_UNWIND_USER_DEFERRED_H 3 3 #define _LINUX_UNWIND_USER_DEFERRED_H 4 4 5 + #include <linux/task_work.h> 5 6 #include <linux/unwind_user.h> 6 7 #include <linux/unwind_deferred_types.h> 8 + 9 + struct unwind_work; 10 + 11 + typedef void (*unwind_callback_t)(struct unwind_work *work, struct unwind_stacktrace *trace, u64 cookie); 12 + 13 + struct unwind_work { 14 + struct list_head list; 15 + unwind_callback_t func; 16 + }; 7 17 8 18 #ifdef CONFIG_UNWIND_USER 9 19 ··· 22 12 23 13 int unwind_user_faultable(struct unwind_stacktrace *trace); 24 14 15 + int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func); 16 + int unwind_deferred_request(struct unwind_work *work, u64 *cookie); 17 + void unwind_deferred_cancel(struct unwind_work *work); 18 + 25 19 static __always_inline void unwind_reset_info(void) 26 20 { 21 + if (unlikely(current->unwind_info.id.id)) 22 + current->unwind_info.id.id = 0; 23 + /* 24 + * As unwind_user_faultable() can be called directly and 25 + * depends on nr_entries being cleared on exit to user, 26 + * this needs to be a separate conditional. 27 + */ 27 28 if (unlikely(current->unwind_info.cache)) 28 29 current->unwind_info.cache->nr_entries = 0; 29 30 } ··· 45 24 static inline void unwind_task_free(struct task_struct *task) {} 46 25 47 26 static inline int unwind_user_faultable(struct unwind_stacktrace *trace) { return -ENOSYS; } 27 + static inline int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func) { return -ENOSYS; } 28 + static inline int unwind_deferred_request(struct unwind_work *work, u64 *timestamp) { return -ENOSYS; } 29 + static inline void unwind_deferred_cancel(struct unwind_work *work) {} 48 30 49 31 static inline void unwind_reset_info(void) {} 50 32
+24
include/linux/unwind_deferred_types.h
··· 7 7 unsigned long entries[]; 8 8 }; 9 9 10 + /* 11 + * The unwind_task_id is a unique identifier that maps to a user space 12 + * stacktrace. It is generated the first time a deferred user space 13 + * stacktrace is requested after a task has entered the kerenl and 14 + * is cleared to zero when it exits. The mapped id will be a non-zero 15 + * number. 16 + * 17 + * To simplify the generation of the 64 bit number, 32 bits will be 18 + * the CPU it was generated on, and the other 32 bits will be a per 19 + * cpu counter that gets incremented by two every time a new identifier 20 + * is generated. The LSB will always be set to keep the value 21 + * from being zero. 22 + */ 23 + union unwind_task_id { 24 + struct { 25 + u32 cpu; 26 + u32 cnt; 27 + }; 28 + u64 id; 29 + }; 30 + 10 31 struct unwind_task_info { 11 32 struct unwind_cache *cache; 33 + struct callback_head work; 34 + union unwind_task_id id; 35 + int pending; 12 36 }; 13 37 14 38 #endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
+155 -1
kernel/unwind/deferred.c
··· 2 2 /* 3 3 * Deferred user space unwinding 4 4 */ 5 + #include <linux/sched/task_stack.h> 6 + #include <linux/unwind_deferred.h> 7 + #include <linux/sched/clock.h> 8 + #include <linux/task_work.h> 5 9 #include <linux/kernel.h> 6 10 #include <linux/sched.h> 7 11 #include <linux/sizes.h> 8 12 #include <linux/slab.h> 9 - #include <linux/unwind_deferred.h> 13 + #include <linux/mm.h> 10 14 11 15 /* Make the cache fit in a 4K page */ 12 16 #define UNWIND_MAX_ENTRIES \ 13 17 ((SZ_4K - sizeof(struct unwind_cache)) / sizeof(long)) 18 + 19 + /* Guards adding to and reading the list of callbacks */ 20 + static DEFINE_MUTEX(callback_mutex); 21 + static LIST_HEAD(callbacks); 22 + 23 + /* 24 + * This is a unique percpu identifier for a given task entry context. 25 + * Conceptually, it's incremented every time the CPU enters the kernel from 26 + * user space, so that each "entry context" on the CPU gets a unique ID. In 27 + * reality, as an optimization, it's only incremented on demand for the first 28 + * deferred unwind request after a given entry-from-user. 29 + * 30 + * It's combined with the CPU id to make a systemwide-unique "context cookie". 31 + */ 32 + static DEFINE_PER_CPU(u32, unwind_ctx_ctr); 33 + 34 + /* 35 + * The context cookie is a unique identifier that is assigned to a user 36 + * space stacktrace. As the user space stacktrace remains the same while 37 + * the task is in the kernel, the cookie is an identifier for the stacktrace. 38 + * Although it is possible for the stacktrace to get another cookie if another 39 + * request is made after the cookie was cleared and before reentering user 40 + * space. 41 + */ 42 + static u64 get_cookie(struct unwind_task_info *info) 43 + { 44 + u32 cnt = 1; 45 + u32 old = 0; 46 + 47 + if (info->id.cpu) 48 + return info->id.id; 49 + 50 + /* LSB is always set to ensure 0 is an invalid value */ 51 + cnt |= __this_cpu_read(unwind_ctx_ctr) + 2; 52 + if (try_cmpxchg(&info->id.cnt, &old, cnt)) { 53 + /* Update the per cpu counter */ 54 + __this_cpu_write(unwind_ctx_ctr, cnt); 55 + } 56 + /* Interrupts are disabled, the CPU will always be same */ 57 + info->id.cpu = smp_processor_id() + 1; /* Must be non zero */ 58 + 59 + return info->id.id; 60 + } 14 61 15 62 /** 16 63 * unwind_user_faultable - Produce a user stacktrace in faultable context ··· 109 62 return 0; 110 63 } 111 64 65 + static void unwind_deferred_task_work(struct callback_head *head) 66 + { 67 + struct unwind_task_info *info = container_of(head, struct unwind_task_info, work); 68 + struct unwind_stacktrace trace; 69 + struct unwind_work *work; 70 + u64 cookie; 71 + 72 + if (WARN_ON_ONCE(!info->pending)) 73 + return; 74 + 75 + /* Allow work to come in again */ 76 + WRITE_ONCE(info->pending, 0); 77 + 78 + /* 79 + * From here on out, the callback must always be called, even if it's 80 + * just an empty trace. 81 + */ 82 + trace.nr = 0; 83 + trace.entries = NULL; 84 + 85 + unwind_user_faultable(&trace); 86 + 87 + cookie = info->id.id; 88 + 89 + guard(mutex)(&callback_mutex); 90 + list_for_each_entry(work, &callbacks, list) { 91 + work->func(work, &trace, cookie); 92 + } 93 + } 94 + 95 + /** 96 + * unwind_deferred_request - Request a user stacktrace on task kernel exit 97 + * @work: Unwind descriptor requesting the trace 98 + * @cookie: The cookie of the first request made for this task 99 + * 100 + * Schedule a user space unwind to be done in task work before exiting the 101 + * kernel. 102 + * 103 + * The returned @cookie output is the generated cookie of the very first 104 + * request for a user space stacktrace for this task since it entered the 105 + * kernel. It can be from a request by any caller of this infrastructure. 106 + * Its value will also be passed to the callback function. It can be 107 + * used to stitch kernel and user stack traces together in post-processing. 108 + * 109 + * It's valid to call this function multiple times for the same @work within 110 + * the same task entry context. Each call will return the same cookie 111 + * while the task hasn't left the kernel. If the callback is not pending 112 + * because it has already been previously called for the same entry context, 113 + * it will be called again with the same stack trace and cookie. 114 + * 115 + * Return: 1 if the the callback was already queued. 116 + * 0 if the callback successfully was queued. 117 + * Negative if there's an error. 118 + * @cookie holds the cookie of the first request by any user 119 + */ 120 + int unwind_deferred_request(struct unwind_work *work, u64 *cookie) 121 + { 122 + struct unwind_task_info *info = &current->unwind_info; 123 + int ret; 124 + 125 + *cookie = 0; 126 + 127 + if (WARN_ON_ONCE(in_nmi())) 128 + return -EINVAL; 129 + 130 + if ((current->flags & (PF_KTHREAD | PF_EXITING)) || 131 + !user_mode(task_pt_regs(current))) 132 + return -EINVAL; 133 + 134 + guard(irqsave)(); 135 + 136 + *cookie = get_cookie(info); 137 + 138 + /* callback already pending? */ 139 + if (info->pending) 140 + return 1; 141 + 142 + /* The work has been claimed, now schedule it. */ 143 + ret = task_work_add(current, &info->work, TWA_RESUME); 144 + if (WARN_ON_ONCE(ret)) 145 + return ret; 146 + 147 + info->pending = 1; 148 + return 0; 149 + } 150 + 151 + void unwind_deferred_cancel(struct unwind_work *work) 152 + { 153 + if (!work) 154 + return; 155 + 156 + guard(mutex)(&callback_mutex); 157 + list_del(&work->list); 158 + } 159 + 160 + int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func) 161 + { 162 + memset(work, 0, sizeof(*work)); 163 + 164 + guard(mutex)(&callback_mutex); 165 + list_add(&work->list, &callbacks); 166 + work->func = func; 167 + return 0; 168 + } 169 + 112 170 void unwind_task_init(struct task_struct *task) 113 171 { 114 172 struct unwind_task_info *info = &task->unwind_info; 115 173 116 174 memset(info, 0, sizeof(*info)); 175 + init_task_work(&info->work, unwind_deferred_task_work); 117 176 } 118 177 119 178 void unwind_task_free(struct task_struct *task) ··· 227 74 struct unwind_task_info *info = &task->unwind_info; 228 75 229 76 kfree(info->cache); 77 + task_work_cancel(task, &info->work); 230 78 }