Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'trace-v7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

Pull tracing updates from Steven Rostedt:
"User visible changes:

- Add an entry into MAINTAINERS file for RUST versions of code

There's now RUST code for tracing and static branches. To
differentiate that code from the C code, add entries in for the
RUST version (with "[RUST]" around it) so that the right
maintainers get notified on changes.

- New bitmask-list option added to tracefs

When this is set, bitmasks in trace event are not displayed as hex
numbers, but instead as lists: e.g. 0-5,7,9 instead of 0000015f

- New show_event_filters file in tracefs

Instead of having to search all events/*/*/filter for any active
filters enabled in the trace instance, the file show_event_filters
will list them so that there's only one file that needs to be
examined to see if any filters are active.

- New show_event_triggers file in tracefs

Instead of having to search all events/*/*/trigger for any active
triggers enabled in the trace instance, the file
show_event_triggers will list them so that there's only one file
that needs to be examined to see if any triggers are active.

- Have traceoff_on_warning disable trace pintk buffer too

Recently recording of trace_printk() could go to other trace
instances instead of the top level instance. But if
traceoff_on_warning triggers, it doesn't stop the buffer with
trace_printk() and that data can easily be lost by being
overwritten. Have traceoff_on_warning also disable the instance
that has trace_printk() being written to it.

- Update the hist_debug file to show what function the field uses

When CONFIG_HIST_TRIGGERS_DEBUG is enabled, a hist_debug file
exists for every event. This displays the internal data of any
histogram enabled for that event. But it is lacking the function
that is called to process one of its fields. This is very useful
information that was missing when debugging histograms.

- Up the histogram stack size from 16 to 31

Stack traces can be used as keys for event histograms. Currently
the size of the stack that is stored is limited to just 16 entries.
But the storage space in the histogram is 256 bytes, meaning that
it can store up to 31 entries (plus one for the count of entries).
Instead of letting that space go to waste, up the limit from 16 to
31. This makes the keys much more useful.

- Fix permissions of per CPU file buffer_size_kb

The per CPU file of buffer_size_kb was incorrectly set to read only
in a previous cleanup. It should be writable.

- Reset "last_boot_info" if the persistent buffer is cleared

The last_boot_info shows address information of a persistent ring
buffer if it contains data from a previous boot. It is cleared when
recording starts again, but it is not cleared when the buffer is
reset. The data is useless after a reset so clear it on reset too.

Internal changes:

- A change was made to allow tracepoint callbacks to have preemption
enabled, and instead be protected by SRCU. This required some
updates to the callbacks for perf and BPF.

perf needed to disable preemption directly in its callback because
it expects preemption disabled in the later code.

BPF needed to disable migration, as its code expects to run
completely on the same CPU.

- Have irq_work wake up other CPU if current CPU is "isolated"

When there's a waiter waiting on ring buffer data and a new event
happens, an irq work is triggered to wake up that waiter. This is
noisy on isolated CPUs (running NO_HZ_FULL). Trigger an IPI to a
house keeping CPU instead.

- Use proper free of trigger_data instead of open coding it in.

- Remove redundant call of event_trigger_reset_filter()

It was called immediately in a function that was called right after
it.

- Workqueue cleanups

- Report errors if tracing_update_buffers() were to fail.

- Make the enum update workqueue generic for other parts of tracing

On boot up, a work queue is created to convert enum names into
their numbers in the trace event format files. This work queue can
also be used for other aspects of tracing that takes some time and
shouldn't be called by the init call code.

The blk_trace initialization takes a bit of time. Have the
initialization code moved to the new tracing generic work queue
function.

- Skip kprobe boot event creation call if there's no kprobes defined
on cmdline

The kprobe initialization to set up kprobes if they are defined on
the cmdline requires taking the event_mutex lock. This can be held
by other tracing code doing initialization for a long time. Since
kprobes added to the kernel command line need to be setup
immediately, as they may be tracing early initialization code, they
cannot be postponed in a work queue and must be setup in the
initcall code.

If there's no kprobe on the kernel cmdline, there's no reason to
take the mutex and slow down the boot up code waiting to get the
lock only to find out there's nothing to do. Simply exit out early
if there's no kprobes on the kernel cmdline.

If there are kprobes on the cmdline, then someone cares more about
tracing over the speed of boot up.

- Clean up the trigger code a bit

- Move code out of trace.c and into their own files

trace.c is now over 11,000 lines of code and has become more
difficult to maintain. Start splitting it up so that related code
is in their own files.

Move all the trace_printk() related code into trace_printk.c.

Move the __always_inline stack functions into trace.h.

Move the pid filtering code into a new trace_pid.c file.

- Better define the max latency and snapshot code

The latency tracers have a "max latency" buffer that is a copy of
the main buffer and gets swapped with it when a new high latency is
detected. This keeps the trace up to the highest latency around
where this max_latency buffer is never written to. It is only used
to save the last max latency trace.

A while ago a snapshot feature was added to tracefs to allow user
space to perform the same logic. It could also enable events to
trigger a "snapshot" if one of their fields hit a new high. This
was built on top of the latency max_latency buffer logic.

Because snapshots came later, they were dependent on the latency
tracers to be enabled. In reality, the latency tracers depend on
the snapshot code and not the other way around. It was just that
they came first.

Restructure the code and the kconfigs to have the latency tracers
depend on snapshot code instead. This actually simplifies the logic
a bit and allows to disable more when the latency tracers are not
defined and the snapshot code is.

- Fix a "false sharing" in the hwlat tracer code

The loop to search for latency in hardware was using a variable
that could be changed by user space for each sample. If the user
change this variable, it could cause a bus contention, and reading
that variable can show up as a large latency in the trace causing a
false positive. Read this variable at the start of the sample with
a READ_ONCE() into a local variable and keep the code from sharing
cache lines with readers.

- Fix function graph tracer static branch optimization code

When only one tracer is defined for function graph tracing, it uses
a static branch to call that tracer directly. When another tracer
is added, it goes into loop logic to call all the registered
callbacks.

The code was incorrect when going back to one tracer and never
re-enabled the static branch again to do the optimization code.

- And other small fixes and cleanups"

* tag 'trace-v7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (46 commits)
function_graph: Restore direct mode when callbacks drop to one
tracing: Fix indentation of return statement in print_trace_fmt()
tracing: Reset last_boot_info if ring buffer is reset
tracing: Fix to set write permission to per-cpu buffer_size_kb
tracing: Fix false sharing in hwlat get_sample()
tracing: Move d_max_latency out of CONFIG_FSNOTIFY protection
tracing: Better separate SNAPSHOT and MAX_TRACE options
tracing: Add tracer_uses_snapshot() helper to remove #ifdefs
tracing: Rename trace_array field max_buffer to snapshot_buffer
tracing: Move pid filtering into trace_pid.c
tracing: Move trace_printk functions out of trace.c and into trace_printk.c
tracing: Use system_state in trace_printk_init_buffers()
tracing: Have trace_printk functions use flags instead of using global_trace
tracing: Make tracing_update_buffers() take NULL for global_trace
tracing: Make printk_trace global for tracing system
tracing: Move ftrace_trace_stack() out of trace.c and into trace.h
tracing: Move __trace_buffer_{un}lock_*() functions to trace.h
tracing: Make tracing_selftest_running global to the tracing subsystem
tracing: Make tracing_disabled global for tracing system
tracing: Clean up use of trace_create_maxlat_file()
...

+1392 -1077
+25
Documentation/trace/ftrace.rst
··· 684 684 685 685 See events.rst for more information. 686 686 687 + show_event_filters: 688 + 689 + A list of events that have filters. This shows the 690 + system/event pair along with the filter that is attached to 691 + the event. 692 + 693 + See events.rst for more information. 694 + 695 + show_event_triggers: 696 + 697 + A list of events that have triggers. This shows the 698 + system/event pair along with the trigger that is attached to 699 + the event. 700 + 701 + See events.rst for more information. 702 + 687 703 available_events: 688 704 689 705 A list of events that can be enabled in tracing. ··· 1305 1289 hashed pointer value instead of real address. 1306 1290 This will be useful if you want to find out which hashed 1307 1291 value is corresponding to the real value in trace log. 1292 + 1293 + bitmask-list 1294 + When enabled, bitmasks are displayed as a human-readable list of 1295 + ranges (e.g., 0,2-5,7) using the printk "%*pbl" format specifier. 1296 + When disabled (the default), bitmasks are displayed in the 1297 + traditional hexadecimal bitmap representation. The list format is 1298 + particularly useful for tracing CPU masks and other large bitmasks 1299 + where individual bit positions are more meaningful than their 1300 + hexadecimal encoding. 1308 1301 1309 1302 record-cmd 1310 1303 When any event or tracer is enabled, a hook is enabled
+15
MAINTAINERS
··· 25250 25250 M: Peter Zijlstra <peterz@infradead.org> 25251 25251 M: Josh Poimboeuf <jpoimboe@kernel.org> 25252 25252 M: Jason Baron <jbaron@akamai.com> 25253 + M: Alice Ryhl <aliceryhl@google.com> 25253 25254 R: Steven Rostedt <rostedt@goodmis.org> 25254 25255 R: Ard Biesheuvel <ardb@kernel.org> 25255 25256 S: Supported ··· 25262 25261 F: include/linux/static_call*.h 25263 25262 F: kernel/jump_label.c 25264 25263 F: kernel/static_call*.c 25264 + F: rust/helpers/jump_label.c 25265 + F: rust/kernel/generated_arch_static_branch_asm.rs.S 25266 + F: rust/kernel/jump_label.rs 25265 25267 25266 25268 STI AUDIO (ASoC) DRIVERS 25267 25269 M: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> ··· 26730 26726 F: scripts/tracing/ 26731 26727 F: scripts/tracepoint-update.c 26732 26728 F: tools/testing/selftests/ftrace/ 26729 + 26730 + TRACING [RUST] 26731 + M: Alice Ryhl <aliceryhl@google.com> 26732 + M: Steven Rostedt <rostedt@goodmis.org> 26733 + R: Masami Hiramatsu <mhiramat@kernel.org> 26734 + R: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> 26735 + L: linux-trace-kernel@vger.kernel.org 26736 + L: rust-for-linux@vger.kernel.org 26737 + S: Maintained 26738 + T: git git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git 26739 + F: rust/kernel/tracepoint.rs 26733 26740 26734 26741 TRACING MMIO ACCESSES (MMIOTRACE) 26735 26742 M: Steven Rostedt <rostedt@goodmis.org>
+4 -4
include/linux/trace_events.h
··· 38 38 *symbol_array); 39 39 #endif 40 40 41 - const char *trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr, 41 + struct trace_iterator; 42 + struct trace_event; 43 + 44 + const char *trace_print_bitmask_seq(struct trace_iterator *iter, void *bitmask_ptr, 42 45 unsigned int bitmask_size); 43 46 44 47 const char *trace_print_hex_seq(struct trace_seq *p, ··· 56 53 trace_print_hex_dump_seq(struct trace_seq *p, const char *prefix_str, 57 54 int prefix_type, int rowsize, int groupsize, 58 55 const void *buf, size_t len, bool ascii); 59 - 60 - struct trace_iterator; 61 - struct trace_event; 62 56 63 57 int trace_raw_output_prep(struct trace_iterator *iter, 64 58 struct trace_event *event);
+11 -1
include/linux/trace_seq.h
··· 114 114 extern int trace_seq_path(struct trace_seq *s, const struct path *path); 115 115 116 116 extern void trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, 117 - int nmaskbits); 117 + int nmaskbits); 118 + 119 + extern void trace_seq_bitmask_list(struct trace_seq *s, 120 + const unsigned long *maskp, 121 + int nmaskbits); 118 122 119 123 extern int trace_seq_hex_dump(struct trace_seq *s, const char *prefix_str, 120 124 int prefix_type, int rowsize, int groupsize, ··· 138 134 static inline void 139 135 trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, 140 136 int nmaskbits) 137 + { 138 + } 139 + 140 + static inline void 141 + trace_seq_bitmask_list(struct trace_seq *s, const unsigned long *maskp, 142 + int nmaskbits) 141 143 { 142 144 } 143 145
+5 -4
include/linux/tracepoint.h
··· 108 108 * An alternative is to use the following for batch reclaim associated 109 109 * with a given tracepoint: 110 110 * 111 - * - tracepoint_is_faultable() == false: call_rcu() 111 + * - tracepoint_is_faultable() == false: call_srcu() 112 112 * - tracepoint_is_faultable() == true: call_rcu_tasks_trace() 113 113 */ 114 114 #ifdef CONFIG_TRACEPOINTS 115 + extern struct srcu_struct tracepoint_srcu; 115 116 static inline void tracepoint_synchronize_unregister(void) 116 117 { 117 118 synchronize_rcu_tasks_trace(); 118 - synchronize_rcu(); 119 + synchronize_srcu(&tracepoint_srcu); 119 120 } 120 121 static inline bool tracepoint_is_faultable(struct tracepoint *tp) 121 122 { ··· 276 275 return static_branch_unlikely(&__tracepoint_##name.key);\ 277 276 } 278 277 279 - #define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ 278 + #define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ 280 279 __DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \ 281 280 static inline void __do_trace_##name(proto) \ 282 281 { \ 283 282 TRACEPOINT_CHECK(name) \ 284 283 if (cond) { \ 285 - guard(preempt_notrace)(); \ 284 + guard(srcu_fast_notrace)(&tracepoint_srcu); \ 286 285 __DO_TRACE_CALL(name, TP_ARGS(args)); \ 287 286 } \ 288 287 } \
+2 -2
include/trace/perf.h
··· 71 71 u64 __count __attribute__((unused)); \ 72 72 struct task_struct *__task __attribute__((unused)); \ 73 73 \ 74 + guard(preempt_notrace)(); \ 74 75 do_perf_trace_##call(__data, args); \ 75 76 } 76 77 ··· 86 85 struct task_struct *__task __attribute__((unused)); \ 87 86 \ 88 87 might_fault(); \ 89 - preempt_disable_notrace(); \ 88 + guard(preempt_notrace)(); \ 90 89 do_perf_trace_##call(__data, args); \ 91 - preempt_enable_notrace(); \ 92 90 } 93 91 94 92 /*
+2 -2
include/trace/stages/stage3_trace_output.h
··· 39 39 void *__bitmask = __get_dynamic_array(field); \ 40 40 unsigned int __bitmask_size; \ 41 41 __bitmask_size = __get_dynamic_array_len(field); \ 42 - trace_print_bitmask_seq(p, __bitmask, __bitmask_size); \ 42 + trace_print_bitmask_seq(iter, __bitmask, __bitmask_size); \ 43 43 }) 44 44 45 45 #undef __get_cpumask ··· 51 51 void *__bitmask = __get_rel_dynamic_array(field); \ 52 52 unsigned int __bitmask_size; \ 53 53 __bitmask_size = __get_rel_dynamic_array_len(field); \ 54 - trace_print_bitmask_seq(p, __bitmask, __bitmask_size); \ 54 + trace_print_bitmask_seq(iter, __bitmask, __bitmask_size); \ 55 55 }) 56 56 57 57 #undef __get_rel_cpumask
+2 -2
include/trace/trace_events.h
··· 436 436 static notrace void \ 437 437 trace_event_raw_event_##call(void *__data, proto) \ 438 438 { \ 439 + guard(preempt_notrace)(); \ 439 440 do_trace_event_raw_event_##call(__data, args); \ 440 441 } 441 442 ··· 448 447 trace_event_raw_event_##call(void *__data, proto) \ 449 448 { \ 450 449 might_fault(); \ 451 - preempt_disable_notrace(); \ 450 + guard(preempt_notrace)(); \ 452 451 do_trace_event_raw_event_##call(__data, args); \ 453 - preempt_enable_notrace(); \ 454 452 } 455 453 456 454 /*
+2 -1
kernel/rcu/srcutree.c
··· 789 789 struct srcu_data *sdp; 790 790 791 791 /* NMI-unsafe use in NMI is a bad sign, as is multi-bit read_flavor values. */ 792 - WARN_ON_ONCE((read_flavor != SRCU_READ_FLAVOR_NMI) && in_nmi()); 792 + WARN_ON_ONCE(read_flavor != SRCU_READ_FLAVOR_NMI && 793 + read_flavor != SRCU_READ_FLAVOR_FAST && in_nmi()); 793 794 WARN_ON_ONCE(read_flavor & (read_flavor - 1)); 794 795 795 796 sdp = raw_cpu_ptr(ssp->sda);
+4 -4
kernel/trace/Kconfig
··· 136 136 137 137 config TRACER_MAX_TRACE 138 138 bool 139 + select TRACER_SNAPSHOT 139 140 140 141 config TRACE_CLOCK 141 142 bool ··· 426 425 select GENERIC_TRACER 427 426 select TRACER_MAX_TRACE 428 427 select RING_BUFFER_ALLOW_SWAP 429 - select TRACER_SNAPSHOT 430 428 select TRACER_SNAPSHOT_PER_CPU_SWAP 431 429 help 432 430 This option measures the time spent in irqs-off critical ··· 448 448 select GENERIC_TRACER 449 449 select TRACER_MAX_TRACE 450 450 select RING_BUFFER_ALLOW_SWAP 451 - select TRACER_SNAPSHOT 452 451 select TRACER_SNAPSHOT_PER_CPU_SWAP 453 452 select TRACE_PREEMPT_TOGGLE 454 453 help ··· 469 470 select GENERIC_TRACER 470 471 select CONTEXT_SWITCH_TRACER 471 472 select TRACER_MAX_TRACE 472 - select TRACER_SNAPSHOT 473 473 help 474 474 This tracer tracks the latency of the highest priority task 475 475 to be scheduled in, starting from the point it has woken up. ··· 618 620 619 621 config TRACER_SNAPSHOT 620 622 bool "Create a snapshot trace buffer" 621 - select TRACER_MAX_TRACE 622 623 help 623 624 Allow tracing users to take snapshot of the current buffer using the 624 625 ftrace interface, e.g.: 625 626 626 627 echo 1 > /sys/kernel/tracing/snapshot 627 628 cat snapshot 629 + 630 + Note, the latency tracers select this option. To disable it, 631 + all the latency tracers need to be disabled. 628 632 629 633 config TRACER_SNAPSHOT_PER_CPU_SWAP 630 634 bool "Allow snapshot to swap per CPU"
+1
kernel/trace/Makefile
··· 68 68 obj-$(CONFIG_TRACING) += trace_seq.o 69 69 obj-$(CONFIG_TRACING) += trace_stat.o 70 70 obj-$(CONFIG_TRACING) += trace_printk.o 71 + obj-$(CONFIG_TRACING) += trace_pid.o 71 72 obj-$(CONFIG_TRACING) += pid_list.o 72 73 obj-$(CONFIG_TRACING_MAP) += tracing_map.o 73 74 obj-$(CONFIG_PREEMPTIRQ_DELAY_TEST) += preemptirq_delay_test.o
+22 -1
kernel/trace/blktrace.c
··· 1832 1832 .funcs = &trace_blk_event_funcs, 1833 1833 }; 1834 1834 1835 - static int __init init_blk_tracer(void) 1835 + static struct work_struct blktrace_works __initdata; 1836 + 1837 + static int __init __init_blk_tracer(void) 1836 1838 { 1837 1839 if (!register_trace_event(&trace_blk_event)) { 1838 1840 pr_warn("Warning: could not register block events\n"); ··· 1852 1850 BUILD_BUG_ON(__alignof__(struct blk_io_trace2) % __alignof__(long)); 1853 1851 1854 1852 return 0; 1853 + } 1854 + 1855 + static void __init blktrace_works_func(struct work_struct *work) 1856 + { 1857 + __init_blk_tracer(); 1858 + } 1859 + 1860 + static int __init init_blk_tracer(void) 1861 + { 1862 + int ret = 0; 1863 + 1864 + if (trace_init_wq) { 1865 + INIT_WORK(&blktrace_works, blktrace_works_func); 1866 + queue_work(trace_init_wq, &blktrace_works); 1867 + } else { 1868 + ret = __init_blk_tracer(); 1869 + } 1870 + 1871 + return ret; 1855 1872 } 1856 1873 1857 1874 device_initcall(init_blk_tracer);
+2 -3
kernel/trace/bpf_trace.c
··· 2076 2076 struct bpf_run_ctx *old_run_ctx; 2077 2077 struct bpf_trace_run_ctx run_ctx; 2078 2078 2079 - cant_sleep(); 2079 + rcu_read_lock_dont_migrate(); 2080 2080 if (unlikely(!bpf_prog_get_recursion_context(prog))) { 2081 2081 bpf_prog_inc_misses_counter(prog); 2082 2082 goto out; ··· 2085 2085 run_ctx.bpf_cookie = link->cookie; 2086 2086 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); 2087 2087 2088 - rcu_read_lock(); 2089 2088 (void) bpf_prog_run(prog, args); 2090 - rcu_read_unlock(); 2091 2089 2092 2090 bpf_reset_run_ctx(old_run_ctx); 2093 2091 out: 2094 2092 bpf_prog_put_recursion_context(prog); 2093 + rcu_read_unlock_migrate(); 2095 2094 } 2096 2095 2097 2096 #define UNPACK(...) __VA_ARGS__
+1 -1
kernel/trace/fgraph.c
··· 1303 1303 static_call_update(fgraph_func, func); 1304 1304 static_call_update(fgraph_retfunc, retfunc); 1305 1305 if (enable_branch) 1306 - static_branch_disable(&fgraph_do_direct); 1306 + static_branch_enable(&fgraph_do_direct); 1307 1307 } 1308 1308 1309 1309 static void ftrace_graph_disable_direct(bool disable_branch)
+4 -3
kernel/trace/ftrace.c
··· 1147 1147 }; 1148 1148 1149 1149 #define ENTRY_SIZE sizeof(struct dyn_ftrace) 1150 + #define ENTRIES_PER_PAGE_GROUP(order) ((PAGE_SIZE << (order)) / ENTRY_SIZE) 1150 1151 1151 1152 static struct ftrace_page *ftrace_pages_start; 1152 1153 static struct ftrace_page *ftrace_pages; ··· 3874 3873 *num_pages += 1 << order; 3875 3874 ftrace_number_of_groups++; 3876 3875 3877 - cnt = (PAGE_SIZE << order) / ENTRY_SIZE; 3876 + cnt = ENTRIES_PER_PAGE_GROUP(order); 3878 3877 pg->order = order; 3879 3878 3880 3879 if (cnt > count) ··· 7669 7668 long skip; 7670 7669 7671 7670 /* Count the number of entries unused and compare it to skipped. */ 7672 - pg_remaining = (PAGE_SIZE << pg->order) / ENTRY_SIZE - pg->index; 7671 + pg_remaining = ENTRIES_PER_PAGE_GROUP(pg->order) - pg->index; 7673 7672 7674 7673 if (!WARN(skipped < pg_remaining, "Extra allocated pages for ftrace")) { 7675 7674 ··· 7677 7676 7678 7677 for (pg = pg_unuse; pg && skip > 0; pg = pg->next) { 7679 7678 remaining += 1 << pg->order; 7680 - skip -= (PAGE_SIZE << pg->order) / ENTRY_SIZE; 7679 + skip -= ENTRIES_PER_PAGE_GROUP(pg->order); 7681 7680 } 7682 7681 7683 7682 pages -= remaining;
+21 -3
kernel/trace/ring_buffer.c
··· 4 4 * 5 5 * Copyright (C) 2008 Steven Rostedt <srostedt@redhat.com> 6 6 */ 7 + #include <linux/sched/isolation.h> 7 8 #include <linux/trace_recursion.h> 8 9 #include <linux/trace_events.h> 9 10 #include <linux/ring_buffer.h> ··· 4014 4013 rb_end_commit(cpu_buffer); 4015 4014 } 4016 4015 4016 + static bool 4017 + rb_irq_work_queue(struct rb_irq_work *irq_work) 4018 + { 4019 + int cpu; 4020 + 4021 + /* irq_work_queue_on() is not NMI-safe */ 4022 + if (unlikely(in_nmi())) 4023 + return irq_work_queue(&irq_work->work); 4024 + 4025 + /* 4026 + * If CPU isolation is not active, cpu is always the current 4027 + * CPU, and the following is equivallent to irq_work_queue(). 4028 + */ 4029 + cpu = housekeeping_any_cpu(HK_TYPE_KERNEL_NOISE); 4030 + return irq_work_queue_on(&irq_work->work, cpu); 4031 + } 4032 + 4017 4033 static __always_inline void 4018 4034 rb_wakeups(struct trace_buffer *buffer, struct ring_buffer_per_cpu *cpu_buffer) 4019 4035 { 4020 4036 if (buffer->irq_work.waiters_pending) { 4021 4037 buffer->irq_work.waiters_pending = false; 4022 4038 /* irq_work_queue() supplies it's own memory barriers */ 4023 - irq_work_queue(&buffer->irq_work.work); 4039 + rb_irq_work_queue(&buffer->irq_work); 4024 4040 } 4025 4041 4026 4042 if (cpu_buffer->irq_work.waiters_pending) { 4027 4043 cpu_buffer->irq_work.waiters_pending = false; 4028 4044 /* irq_work_queue() supplies it's own memory barriers */ 4029 - irq_work_queue(&cpu_buffer->irq_work.work); 4045 + rb_irq_work_queue(&cpu_buffer->irq_work); 4030 4046 } 4031 4047 4032 4048 if (cpu_buffer->last_pages_touch == local_read(&cpu_buffer->pages_touched)) ··· 4063 4045 cpu_buffer->irq_work.wakeup_full = true; 4064 4046 cpu_buffer->irq_work.full_waiters_pending = false; 4065 4047 /* irq_work_queue() supplies it's own memory barriers */ 4066 - irq_work_queue(&cpu_buffer->irq_work.work); 4048 + rb_irq_work_queue(&cpu_buffer->irq_work); 4067 4049 } 4068 4050 4069 4051 #ifdef CONFIG_RING_BUFFER_RECORD_RECURSION
+139 -919
kernel/trace/trace.c
··· 67 67 * insertions into the ring-buffer such as trace_printk could occurred 68 68 * at the same time, giving false positive or negative results. 69 69 */ 70 - static bool __read_mostly tracing_selftest_running; 70 + bool __read_mostly tracing_selftest_running; 71 71 72 72 /* 73 73 * If boot-time tracing including tracers/events via kernel cmdline ··· 83 83 } 84 84 } 85 85 #else 86 - #define tracing_selftest_running 0 87 86 #define tracing_selftest_disabled 0 88 87 #endif 89 88 ··· 113 114 * of the tracer is successful. But that is the only place that sets 114 115 * this back to zero. 115 116 */ 116 - static int tracing_disabled = 1; 117 + int tracing_disabled = 1; 117 118 118 119 cpumask_var_t __read_mostly tracing_buffer_mask; 119 120 ··· 534 535 .trace_flags = TRACE_DEFAULT_FLAGS, 535 536 }; 536 537 537 - static struct trace_array *printk_trace = &global_trace; 538 + struct trace_array *printk_trace = &global_trace; 538 539 539 540 /* List of trace_arrays interested in the top level trace_marker */ 540 541 static LIST_HEAD(marker_copies); 541 - 542 - static __always_inline bool printk_binsafe(struct trace_array *tr) 543 - { 544 - /* 545 - * The binary format of traceprintk can cause a crash if used 546 - * by a buffer from another boot. Force the use of the 547 - * non binary version of trace_printk if the trace_printk 548 - * buffer is a boot mapped ring buffer. 549 - */ 550 - return !(tr->flags & TRACE_ARRAY_FL_BOOT); 551 - } 552 542 553 543 static void update_printk_trace(struct trace_array *tr) 554 544 { ··· 635 647 return -ENODEV; 636 648 637 649 return 0; 638 - } 639 - 640 - /** 641 - * trace_find_filtered_pid - check if a pid exists in a filtered_pid list 642 - * @filtered_pids: The list of pids to check 643 - * @search_pid: The PID to find in @filtered_pids 644 - * 645 - * Returns true if @search_pid is found in @filtered_pids, and false otherwise. 646 - */ 647 - bool 648 - trace_find_filtered_pid(struct trace_pid_list *filtered_pids, pid_t search_pid) 649 - { 650 - return trace_pid_list_is_set(filtered_pids, search_pid); 651 - } 652 - 653 - /** 654 - * trace_ignore_this_task - should a task be ignored for tracing 655 - * @filtered_pids: The list of pids to check 656 - * @filtered_no_pids: The list of pids not to be traced 657 - * @task: The task that should be ignored if not filtered 658 - * 659 - * Checks if @task should be traced or not from @filtered_pids. 660 - * Returns true if @task should *NOT* be traced. 661 - * Returns false if @task should be traced. 662 - */ 663 - bool 664 - trace_ignore_this_task(struct trace_pid_list *filtered_pids, 665 - struct trace_pid_list *filtered_no_pids, 666 - struct task_struct *task) 667 - { 668 - /* 669 - * If filtered_no_pids is not empty, and the task's pid is listed 670 - * in filtered_no_pids, then return true. 671 - * Otherwise, if filtered_pids is empty, that means we can 672 - * trace all tasks. If it has content, then only trace pids 673 - * within filtered_pids. 674 - */ 675 - 676 - return (filtered_pids && 677 - !trace_find_filtered_pid(filtered_pids, task->pid)) || 678 - (filtered_no_pids && 679 - trace_find_filtered_pid(filtered_no_pids, task->pid)); 680 - } 681 - 682 - /** 683 - * trace_filter_add_remove_task - Add or remove a task from a pid_list 684 - * @pid_list: The list to modify 685 - * @self: The current task for fork or NULL for exit 686 - * @task: The task to add or remove 687 - * 688 - * If adding a task, if @self is defined, the task is only added if @self 689 - * is also included in @pid_list. This happens on fork and tasks should 690 - * only be added when the parent is listed. If @self is NULL, then the 691 - * @task pid will be removed from the list, which would happen on exit 692 - * of a task. 693 - */ 694 - void trace_filter_add_remove_task(struct trace_pid_list *pid_list, 695 - struct task_struct *self, 696 - struct task_struct *task) 697 - { 698 - if (!pid_list) 699 - return; 700 - 701 - /* For forks, we only add if the forking task is listed */ 702 - if (self) { 703 - if (!trace_find_filtered_pid(pid_list, self->pid)) 704 - return; 705 - } 706 - 707 - /* "self" is set for forks, and NULL for exits */ 708 - if (self) 709 - trace_pid_list_set(pid_list, task->pid); 710 - else 711 - trace_pid_list_clear(pid_list, task->pid); 712 - } 713 - 714 - /** 715 - * trace_pid_next - Used for seq_file to get to the next pid of a pid_list 716 - * @pid_list: The pid list to show 717 - * @v: The last pid that was shown (+1 the actual pid to let zero be displayed) 718 - * @pos: The position of the file 719 - * 720 - * This is used by the seq_file "next" operation to iterate the pids 721 - * listed in a trace_pid_list structure. 722 - * 723 - * Returns the pid+1 as we want to display pid of zero, but NULL would 724 - * stop the iteration. 725 - */ 726 - void *trace_pid_next(struct trace_pid_list *pid_list, void *v, loff_t *pos) 727 - { 728 - long pid = (unsigned long)v; 729 - unsigned int next; 730 - 731 - (*pos)++; 732 - 733 - /* pid already is +1 of the actual previous bit */ 734 - if (trace_pid_list_next(pid_list, pid, &next) < 0) 735 - return NULL; 736 - 737 - pid = next; 738 - 739 - /* Return pid + 1 to allow zero to be represented */ 740 - return (void *)(pid + 1); 741 - } 742 - 743 - /** 744 - * trace_pid_start - Used for seq_file to start reading pid lists 745 - * @pid_list: The pid list to show 746 - * @pos: The position of the file 747 - * 748 - * This is used by seq_file "start" operation to start the iteration 749 - * of listing pids. 750 - * 751 - * Returns the pid+1 as we want to display pid of zero, but NULL would 752 - * stop the iteration. 753 - */ 754 - void *trace_pid_start(struct trace_pid_list *pid_list, loff_t *pos) 755 - { 756 - unsigned long pid; 757 - unsigned int first; 758 - loff_t l = 0; 759 - 760 - if (trace_pid_list_first(pid_list, &first) < 0) 761 - return NULL; 762 - 763 - pid = first; 764 - 765 - /* Return pid + 1 so that zero can be the exit value */ 766 - for (pid++; pid && l < *pos; 767 - pid = (unsigned long)trace_pid_next(pid_list, (void *)pid, &l)) 768 - ; 769 - return (void *)pid; 770 - } 771 - 772 - /** 773 - * trace_pid_show - show the current pid in seq_file processing 774 - * @m: The seq_file structure to write into 775 - * @v: A void pointer of the pid (+1) value to display 776 - * 777 - * Can be directly used by seq_file operations to display the current 778 - * pid value. 779 - */ 780 - int trace_pid_show(struct seq_file *m, void *v) 781 - { 782 - unsigned long pid = (unsigned long)v - 1; 783 - 784 - seq_printf(m, "%lu\n", pid); 785 - return 0; 786 - } 787 - 788 - /* 128 should be much more than enough */ 789 - #define PID_BUF_SIZE 127 790 - 791 - int trace_pid_write(struct trace_pid_list *filtered_pids, 792 - struct trace_pid_list **new_pid_list, 793 - const char __user *ubuf, size_t cnt) 794 - { 795 - struct trace_pid_list *pid_list; 796 - struct trace_parser parser; 797 - unsigned long val; 798 - int nr_pids = 0; 799 - ssize_t read = 0; 800 - ssize_t ret; 801 - loff_t pos; 802 - pid_t pid; 803 - 804 - if (trace_parser_get_init(&parser, PID_BUF_SIZE + 1)) 805 - return -ENOMEM; 806 - 807 - /* 808 - * Always recreate a new array. The write is an all or nothing 809 - * operation. Always create a new array when adding new pids by 810 - * the user. If the operation fails, then the current list is 811 - * not modified. 812 - */ 813 - pid_list = trace_pid_list_alloc(); 814 - if (!pid_list) { 815 - trace_parser_put(&parser); 816 - return -ENOMEM; 817 - } 818 - 819 - if (filtered_pids) { 820 - /* copy the current bits to the new max */ 821 - ret = trace_pid_list_first(filtered_pids, &pid); 822 - while (!ret) { 823 - ret = trace_pid_list_set(pid_list, pid); 824 - if (ret < 0) 825 - goto out; 826 - 827 - ret = trace_pid_list_next(filtered_pids, pid + 1, &pid); 828 - nr_pids++; 829 - } 830 - } 831 - 832 - ret = 0; 833 - while (cnt > 0) { 834 - 835 - pos = 0; 836 - 837 - ret = trace_get_user(&parser, ubuf, cnt, &pos); 838 - if (ret < 0) 839 - break; 840 - 841 - read += ret; 842 - ubuf += ret; 843 - cnt -= ret; 844 - 845 - if (!trace_parser_loaded(&parser)) 846 - break; 847 - 848 - ret = -EINVAL; 849 - if (kstrtoul(parser.buffer, 0, &val)) 850 - break; 851 - 852 - pid = (pid_t)val; 853 - 854 - if (trace_pid_list_set(pid_list, pid) < 0) { 855 - ret = -1; 856 - break; 857 - } 858 - nr_pids++; 859 - 860 - trace_parser_clear(&parser); 861 - ret = 0; 862 - } 863 - out: 864 - trace_parser_put(&parser); 865 - 866 - if (ret < 0) { 867 - trace_pid_list_free(pid_list); 868 - return ret; 869 - } 870 - 871 - if (!nr_pids) { 872 - /* Cleared the list of pids */ 873 - trace_pid_list_free(pid_list); 874 - pid_list = NULL; 875 - } 876 - 877 - *new_pid_list = pid_list; 878 - 879 - return read; 880 650 } 881 651 882 652 static u64 buffer_ftrace_now(struct array_buffer *buf, int cpu) ··· 779 1033 780 1034 #endif 781 1035 782 - #ifdef CONFIG_STACKTRACE 783 - static void __ftrace_trace_stack(struct trace_array *tr, 784 - struct trace_buffer *buffer, 785 - unsigned int trace_ctx, 786 - int skip, struct pt_regs *regs); 787 - static inline void ftrace_trace_stack(struct trace_array *tr, 788 - struct trace_buffer *buffer, 789 - unsigned int trace_ctx, 790 - int skip, struct pt_regs *regs); 791 - 792 - #else 793 - static inline void __ftrace_trace_stack(struct trace_array *tr, 794 - struct trace_buffer *buffer, 795 - unsigned int trace_ctx, 796 - int skip, struct pt_regs *regs) 797 - { 798 - } 799 - static inline void ftrace_trace_stack(struct trace_array *tr, 800 - struct trace_buffer *buffer, 801 - unsigned long trace_ctx, 802 - int skip, struct pt_regs *regs) 803 - { 804 - } 805 - 806 - #endif 807 - 808 - static __always_inline void 809 - trace_event_setup(struct ring_buffer_event *event, 810 - int type, unsigned int trace_ctx) 811 - { 812 - struct trace_entry *ent = ring_buffer_event_data(event); 813 - 814 - tracing_generic_entry_update(ent, type, trace_ctx); 815 - } 816 - 817 - static __always_inline struct ring_buffer_event * 818 - __trace_buffer_lock_reserve(struct trace_buffer *buffer, 819 - int type, 820 - unsigned long len, 821 - unsigned int trace_ctx) 822 - { 823 - struct ring_buffer_event *event; 824 - 825 - event = ring_buffer_lock_reserve(buffer, len); 826 - if (event != NULL) 827 - trace_event_setup(event, type, trace_ctx); 828 - 829 - return event; 830 - } 831 - 832 1036 void tracer_tracing_on(struct trace_array *tr) 833 1037 { 834 1038 if (tr->array_buffer.buffer) ··· 806 1110 } 807 1111 EXPORT_SYMBOL_GPL(tracing_on); 808 1112 809 - 810 - static __always_inline void 811 - __buffer_unlock_commit(struct trace_buffer *buffer, struct ring_buffer_event *event) 812 - { 813 - __this_cpu_write(trace_taskinfo_save, true); 814 - 815 - /* If this is the temp buffer, we need to commit fully */ 816 - if (this_cpu_read(trace_buffered_event) == event) { 817 - /* Length is in event->array[0] */ 818 - ring_buffer_write(buffer, event->array[0], &event->array[1]); 819 - /* Release the temp buffer */ 820 - this_cpu_dec(trace_buffered_event_cnt); 821 - /* ring_buffer_unlock_commit() enables preemption */ 822 - preempt_enable_notrace(); 823 - } else 824 - ring_buffer_unlock_commit(buffer); 825 - } 826 - 827 - int __trace_array_puts(struct trace_array *tr, unsigned long ip, 828 - const char *str, int size) 829 - { 830 - struct ring_buffer_event *event; 831 - struct trace_buffer *buffer; 832 - struct print_entry *entry; 833 - unsigned int trace_ctx; 834 - int alloc; 835 - 836 - if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 837 - return 0; 838 - 839 - if (unlikely(tracing_selftest_running && tr == &global_trace)) 840 - return 0; 841 - 842 - if (unlikely(tracing_disabled)) 843 - return 0; 844 - 845 - alloc = sizeof(*entry) + size + 2; /* possible \n added */ 846 - 847 - trace_ctx = tracing_gen_ctx(); 848 - buffer = tr->array_buffer.buffer; 849 - guard(ring_buffer_nest)(buffer); 850 - event = __trace_buffer_lock_reserve(buffer, TRACE_PRINT, alloc, 851 - trace_ctx); 852 - if (!event) 853 - return 0; 854 - 855 - entry = ring_buffer_event_data(event); 856 - entry->ip = ip; 857 - 858 - memcpy(&entry->buf, str, size); 859 - 860 - /* Add a newline if necessary */ 861 - if (entry->buf[size - 1] != '\n') { 862 - entry->buf[size] = '\n'; 863 - entry->buf[size + 1] = '\0'; 864 - } else 865 - entry->buf[size] = '\0'; 866 - 867 - __buffer_unlock_commit(buffer, event); 868 - ftrace_trace_stack(tr, buffer, trace_ctx, 4, NULL); 869 - return size; 870 - } 871 - EXPORT_SYMBOL_GPL(__trace_array_puts); 872 - 873 - /** 874 - * __trace_puts - write a constant string into the trace buffer. 875 - * @ip: The address of the caller 876 - * @str: The constant string to write 877 - */ 878 - int __trace_puts(unsigned long ip, const char *str) 879 - { 880 - return __trace_array_puts(printk_trace, ip, str, strlen(str)); 881 - } 882 - EXPORT_SYMBOL_GPL(__trace_puts); 883 - 884 - /** 885 - * __trace_bputs - write the pointer to a constant string into trace buffer 886 - * @ip: The address of the caller 887 - * @str: The constant string to write to the buffer to 888 - */ 889 - int __trace_bputs(unsigned long ip, const char *str) 890 - { 891 - struct trace_array *tr = READ_ONCE(printk_trace); 892 - struct ring_buffer_event *event; 893 - struct trace_buffer *buffer; 894 - struct bputs_entry *entry; 895 - unsigned int trace_ctx; 896 - int size = sizeof(struct bputs_entry); 897 - 898 - if (!printk_binsafe(tr)) 899 - return __trace_puts(ip, str); 900 - 901 - if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 902 - return 0; 903 - 904 - if (unlikely(tracing_selftest_running || tracing_disabled)) 905 - return 0; 906 - 907 - trace_ctx = tracing_gen_ctx(); 908 - buffer = tr->array_buffer.buffer; 909 - 910 - guard(ring_buffer_nest)(buffer); 911 - event = __trace_buffer_lock_reserve(buffer, TRACE_BPUTS, size, 912 - trace_ctx); 913 - if (!event) 914 - return 0; 915 - 916 - entry = ring_buffer_event_data(event); 917 - entry->ip = ip; 918 - entry->str = str; 919 - 920 - __buffer_unlock_commit(buffer, event); 921 - ftrace_trace_stack(tr, buffer, trace_ctx, 4, NULL); 922 - 923 - return 1; 924 - } 925 - EXPORT_SYMBOL_GPL(__trace_bputs); 926 - 927 1113 #ifdef CONFIG_TRACER_SNAPSHOT 928 1114 static void tracing_snapshot_instance_cond(struct trace_array *tr, 929 1115 void *cond_data) 930 1116 { 931 - struct tracer *tracer = tr->current_trace; 932 1117 unsigned long flags; 933 1118 934 1119 if (in_nmi()) { ··· 825 1248 return; 826 1249 } 827 1250 828 - /* Note, snapshot can not be used when the tracer uses it */ 829 - if (tracer->use_max_tr) { 830 - trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n"); 1251 + if (tr->mapped) { 1252 + trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n"); 831 1253 trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 832 1254 return; 833 1255 } 834 1256 835 - if (tr->mapped) { 836 - trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n"); 1257 + /* Note, snapshot can not be used when the tracer uses it */ 1258 + if (tracer_uses_snapshot(tr->current_trace)) { 1259 + trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n"); 837 1260 trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 838 1261 return; 839 1262 } ··· 933 1356 934 1357 /* Make the snapshot buffer have the same order as main buffer */ 935 1358 order = ring_buffer_subbuf_order_get(tr->array_buffer.buffer); 936 - ret = ring_buffer_subbuf_order_set(tr->max_buffer.buffer, order); 1359 + ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order); 937 1360 if (ret < 0) 938 1361 return ret; 939 1362 940 1363 /* allocate spare buffer */ 941 - ret = resize_buffer_duplicate_size(&tr->max_buffer, 1364 + ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 942 1365 &tr->array_buffer, RING_BUFFER_ALL_CPUS); 943 1366 if (ret < 0) 944 1367 return ret; ··· 956 1379 * The max_tr ring buffer has some state (e.g. ring->clock) and 957 1380 * we want preserve it. 958 1381 */ 959 - ring_buffer_subbuf_order_set(tr->max_buffer.buffer, 0); 960 - ring_buffer_resize(tr->max_buffer.buffer, 1, RING_BUFFER_ALL_CPUS); 961 - set_buffer_entries(&tr->max_buffer, 1); 962 - tracing_reset_online_cpus(&tr->max_buffer); 1382 + ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, 0); 1383 + ring_buffer_resize(tr->snapshot_buffer.buffer, 1, RING_BUFFER_ALL_CPUS); 1384 + set_buffer_entries(&tr->snapshot_buffer, 1); 1385 + tracing_reset_online_cpus(&tr->snapshot_buffer); 963 1386 tr->allocated_snapshot = false; 964 1387 } 965 1388 ··· 1075 1498 1076 1499 guard(mutex)(&trace_types_lock); 1077 1500 1078 - if (tr->current_trace->use_max_tr) 1501 + if (tracer_uses_snapshot(tr->current_trace)) 1079 1502 return -EBUSY; 1080 1503 1081 1504 /* ··· 1242 1665 void disable_trace_on_warning(void) 1243 1666 { 1244 1667 if (__disable_trace_on_warning) { 1668 + struct trace_array *tr = READ_ONCE(printk_trace); 1669 + 1245 1670 trace_array_printk_buf(global_trace.array_buffer.buffer, _THIS_IP_, 1246 1671 "Disabling tracing due to warning\n"); 1247 1672 tracing_off(); 1673 + 1674 + /* Disable trace_printk() buffer too */ 1675 + if (tr != &global_trace) { 1676 + trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, 1677 + "Disabling tracing due to warning\n"); 1678 + tracer_tracing_off(tr); 1679 + } 1248 1680 } 1249 1681 } 1250 1682 ··· 1488 1902 unsigned long __read_mostly tracing_thresh; 1489 1903 1490 1904 #ifdef CONFIG_TRACER_MAX_TRACE 1491 - static const struct file_operations tracing_max_lat_fops; 1492 - 1493 1905 #ifdef LATENCY_FS_NOTIFY 1494 - 1495 1906 static struct workqueue_struct *fsnotify_wq; 1496 1907 1497 1908 static void latency_fsnotify_workfn(struct work_struct *work) ··· 1503 1920 struct trace_array *tr = container_of(iwork, struct trace_array, 1504 1921 fsnotify_irqwork); 1505 1922 queue_work(fsnotify_wq, &tr->fsnotify_work); 1506 - } 1507 - 1508 - static void trace_create_maxlat_file(struct trace_array *tr, 1509 - struct dentry *d_tracer) 1510 - { 1511 - INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); 1512 - init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); 1513 - tr->d_max_latency = trace_create_file("tracing_max_latency", 1514 - TRACE_MODE_WRITE, 1515 - d_tracer, tr, 1516 - &tracing_max_lat_fops); 1517 1923 } 1518 1924 1519 1925 __init static int latency_fsnotify_init(void) ··· 1529 1957 */ 1530 1958 irq_work_queue(&tr->fsnotify_irqwork); 1531 1959 } 1960 + #endif /* !LATENCY_FS_NOTIFY */ 1532 1961 1533 - #else /* !LATENCY_FS_NOTIFY */ 1962 + static const struct file_operations tracing_max_lat_fops; 1534 1963 1535 - #define trace_create_maxlat_file(tr, d_tracer) \ 1536 - trace_create_file("tracing_max_latency", TRACE_MODE_WRITE, \ 1537 - d_tracer, tr, &tracing_max_lat_fops) 1538 - 1964 + static void trace_create_maxlat_file(struct trace_array *tr, 1965 + struct dentry *d_tracer) 1966 + { 1967 + #ifdef LATENCY_FS_NOTIFY 1968 + INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); 1969 + init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); 1539 1970 #endif 1971 + tr->d_max_latency = trace_create_file("tracing_max_latency", 1972 + TRACE_MODE_WRITE, 1973 + d_tracer, tr, 1974 + &tracing_max_lat_fops); 1975 + } 1540 1976 1541 1977 /* 1542 1978 * Copy the new maximum trace into the separate maximum-trace ··· 1555 1975 __update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu) 1556 1976 { 1557 1977 struct array_buffer *trace_buf = &tr->array_buffer; 1558 - struct array_buffer *max_buf = &tr->max_buffer; 1559 1978 struct trace_array_cpu *data = per_cpu_ptr(trace_buf->data, cpu); 1979 + struct array_buffer *max_buf = &tr->snapshot_buffer; 1560 1980 struct trace_array_cpu *max_data = per_cpu_ptr(max_buf->data, cpu); 1561 1981 1562 1982 max_buf->cpu = cpu; ··· 1585 2005 tracing_record_cmdline(tsk); 1586 2006 latency_fsnotify(tr); 1587 2007 } 2008 + #else 2009 + static inline void trace_create_maxlat_file(struct trace_array *tr, 2010 + struct dentry *d_tracer) { } 2011 + static inline void __update_max_tr(struct trace_array *tr, 2012 + struct task_struct *tsk, int cpu) { } 2013 + #endif /* CONFIG_TRACER_MAX_TRACE */ 1588 2014 2015 + #ifdef CONFIG_TRACER_SNAPSHOT 1589 2016 /** 1590 2017 * update_max_tr - snapshot all trace buffers from global_trace to max_tr 1591 2018 * @tr: tracer ··· 1622 2035 1623 2036 /* Inherit the recordable setting from array_buffer */ 1624 2037 if (ring_buffer_record_is_set_on(tr->array_buffer.buffer)) 1625 - ring_buffer_record_on(tr->max_buffer.buffer); 2038 + ring_buffer_record_on(tr->snapshot_buffer.buffer); 1626 2039 else 1627 - ring_buffer_record_off(tr->max_buffer.buffer); 2040 + ring_buffer_record_off(tr->snapshot_buffer.buffer); 1628 2041 1629 - #ifdef CONFIG_TRACER_SNAPSHOT 1630 2042 if (tr->cond_snapshot && !tr->cond_snapshot->update(tr, cond_data)) { 1631 2043 arch_spin_unlock(&tr->max_lock); 1632 2044 return; 1633 2045 } 1634 - #endif 1635 - swap(tr->array_buffer.buffer, tr->max_buffer.buffer); 2046 + 2047 + swap(tr->array_buffer.buffer, tr->snapshot_buffer.buffer); 1636 2048 1637 2049 __update_max_tr(tr, tsk, cpu); 1638 2050 ··· 1666 2080 1667 2081 arch_spin_lock(&tr->max_lock); 1668 2082 1669 - ret = ring_buffer_swap_cpu(tr->max_buffer.buffer, tr->array_buffer.buffer, cpu); 2083 + ret = ring_buffer_swap_cpu(tr->snapshot_buffer.buffer, tr->array_buffer.buffer, cpu); 1670 2084 1671 2085 if (ret == -EBUSY) { 1672 2086 /* ··· 1676 2090 * and flag that it failed. 1677 2091 * Another reason is resize is in progress. 1678 2092 */ 1679 - trace_array_printk_buf(tr->max_buffer.buffer, _THIS_IP_, 2093 + trace_array_printk_buf(tr->snapshot_buffer.buffer, _THIS_IP_, 1680 2094 "Failed to swap buffers due to commit or resize in progress\n"); 1681 2095 } 1682 2096 ··· 1685 2099 __update_max_tr(tr, tsk, cpu); 1686 2100 arch_spin_unlock(&tr->max_lock); 1687 2101 } 1688 - 1689 - #endif /* CONFIG_TRACER_MAX_TRACE */ 2102 + #endif /* CONFIG_TRACER_SNAPSHOT */ 1690 2103 1691 2104 struct pipe_wait { 1692 2105 struct trace_iterator *iter; ··· 1718 2133 ret = ring_buffer_wait(iter->array_buffer->buffer, iter->cpu_file, full, 1719 2134 wait_pipe_cond, &pwait); 1720 2135 1721 - #ifdef CONFIG_TRACER_MAX_TRACE 2136 + #ifdef CONFIG_TRACER_SNAPSHOT 1722 2137 /* 1723 2138 * Make sure this is still the snapshot buffer, as if a snapshot were 1724 2139 * to happen, this would now be the main buffer. 1725 2140 */ 1726 2141 if (iter->snapshot) 1727 - iter->array_buffer = &iter->tr->max_buffer; 2142 + iter->array_buffer = &iter->tr->snapshot_buffer; 1728 2143 #endif 1729 2144 return ret; 1730 2145 } ··· 1789 2204 tr->current_trace_flags = type->flags ? : type->default_flags; 1790 2205 1791 2206 #ifdef CONFIG_TRACER_MAX_TRACE 1792 - if (type->use_max_tr) { 2207 + if (tracer_uses_snapshot(type)) { 1793 2208 /* If we expanded the buffers, make sure the max is expanded too */ 1794 2209 if (tr->ring_buffer_expanded) 1795 - ring_buffer_resize(tr->max_buffer.buffer, trace_buf_size, 2210 + ring_buffer_resize(tr->snapshot_buffer.buffer, trace_buf_size, 1796 2211 RING_BUFFER_ALL_CPUS); 1797 2212 tr->allocated_snapshot = true; 1798 2213 } ··· 1814 2229 tracing_reset_online_cpus(&tr->array_buffer); 1815 2230 1816 2231 #ifdef CONFIG_TRACER_MAX_TRACE 1817 - if (type->use_max_tr) { 2232 + if (tracer_uses_snapshot(type)) { 1818 2233 tr->allocated_snapshot = false; 1819 2234 1820 2235 /* Shrink the max buffer again */ 1821 2236 if (tr->ring_buffer_expanded) 1822 - ring_buffer_resize(tr->max_buffer.buffer, 1, 2237 + ring_buffer_resize(tr->snapshot_buffer.buffer, 1, 1823 2238 RING_BUFFER_ALL_CPUS); 1824 2239 } 1825 2240 #endif ··· 2061 2476 continue; 2062 2477 tr->clear_trace = false; 2063 2478 tracing_reset_online_cpus(&tr->array_buffer); 2064 - #ifdef CONFIG_TRACER_MAX_TRACE 2065 - tracing_reset_online_cpus(&tr->max_buffer); 2479 + #ifdef CONFIG_TRACER_SNAPSHOT 2480 + tracing_reset_online_cpus(&tr->snapshot_buffer); 2066 2481 #endif 2067 2482 } 2068 2483 } ··· 2101 2516 if (buffer) 2102 2517 ring_buffer_record_enable(buffer); 2103 2518 2104 - #ifdef CONFIG_TRACER_MAX_TRACE 2105 - buffer = tr->max_buffer.buffer; 2519 + #ifdef CONFIG_TRACER_SNAPSHOT 2520 + buffer = tr->snapshot_buffer.buffer; 2106 2521 if (buffer) 2107 2522 ring_buffer_record_enable(buffer); 2108 2523 #endif ··· 2137 2552 if (buffer) 2138 2553 ring_buffer_record_disable(buffer); 2139 2554 2140 - #ifdef CONFIG_TRACER_MAX_TRACE 2141 - buffer = tr->max_buffer.buffer; 2555 + #ifdef CONFIG_TRACER_SNAPSHOT 2556 + buffer = tr->snapshot_buffer.buffer; 2142 2557 if (buffer) 2143 2558 ring_buffer_record_disable(buffer); 2144 2559 #endif ··· 2586 3001 static DEFINE_PER_CPU(struct ftrace_stacks, ftrace_stacks); 2587 3002 static DEFINE_PER_CPU(int, ftrace_stack_reserve); 2588 3003 2589 - static void __ftrace_trace_stack(struct trace_array *tr, 2590 - struct trace_buffer *buffer, 2591 - unsigned int trace_ctx, 2592 - int skip, struct pt_regs *regs) 3004 + void __ftrace_trace_stack(struct trace_array *tr, 3005 + struct trace_buffer *buffer, 3006 + unsigned int trace_ctx, 3007 + int skip, struct pt_regs *regs) 2593 3008 { 2594 3009 struct ring_buffer_event *event; 2595 3010 unsigned int size, nr_entries; ··· 2670 3085 barrier(); 2671 3086 __this_cpu_dec(ftrace_stack_reserve); 2672 3087 trace_clear_recursion(bit); 2673 - } 2674 - 2675 - static inline void ftrace_trace_stack(struct trace_array *tr, 2676 - struct trace_buffer *buffer, 2677 - unsigned int trace_ctx, 2678 - int skip, struct pt_regs *regs) 2679 - { 2680 - if (!(tr->trace_flags & TRACE_ITER(STACKTRACE))) 2681 - return; 2682 - 2683 - __ftrace_trace_stack(tr, buffer, trace_ctx, skip, regs); 2684 3088 } 2685 3089 2686 3090 void __trace_stack(struct trace_array *tr, unsigned int trace_ctx, ··· 2805 3231 2806 3232 __buffer_unlock_commit(buffer, event); 2807 3233 } 2808 - 2809 - /* created for use with alloc_percpu */ 2810 - struct trace_buffer_struct { 2811 - int nesting; 2812 - char buffer[4][TRACE_BUF_SIZE]; 2813 - }; 2814 - 2815 - static struct trace_buffer_struct __percpu *trace_percpu_buffer; 2816 - 2817 - /* 2818 - * This allows for lockless recording. If we're nested too deeply, then 2819 - * this returns NULL. 2820 - */ 2821 - static char *get_trace_buf(void) 2822 - { 2823 - struct trace_buffer_struct *buffer = this_cpu_ptr(trace_percpu_buffer); 2824 - 2825 - if (!trace_percpu_buffer || buffer->nesting >= 4) 2826 - return NULL; 2827 - 2828 - buffer->nesting++; 2829 - 2830 - /* Interrupts must see nesting incremented before we use the buffer */ 2831 - barrier(); 2832 - return &buffer->buffer[buffer->nesting - 1][0]; 2833 - } 2834 - 2835 - static void put_trace_buf(void) 2836 - { 2837 - /* Don't let the decrement of nesting leak before this */ 2838 - barrier(); 2839 - this_cpu_dec(trace_percpu_buffer->nesting); 2840 - } 2841 - 2842 - static int alloc_percpu_trace_buffer(void) 2843 - { 2844 - struct trace_buffer_struct __percpu *buffers; 2845 - 2846 - if (trace_percpu_buffer) 2847 - return 0; 2848 - 2849 - buffers = alloc_percpu(struct trace_buffer_struct); 2850 - if (MEM_FAIL(!buffers, "Could not allocate percpu trace_printk buffer")) 2851 - return -ENOMEM; 2852 - 2853 - trace_percpu_buffer = buffers; 2854 - return 0; 2855 - } 2856 - 2857 - static int buffers_allocated; 2858 - 2859 - void trace_printk_init_buffers(void) 2860 - { 2861 - if (buffers_allocated) 2862 - return; 2863 - 2864 - if (alloc_percpu_trace_buffer()) 2865 - return; 2866 - 2867 - /* trace_printk() is for debug use only. Don't use it in production. */ 2868 - 2869 - pr_warn("\n"); 2870 - pr_warn("**********************************************************\n"); 2871 - pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); 2872 - pr_warn("** **\n"); 2873 - pr_warn("** trace_printk() being used. Allocating extra memory. **\n"); 2874 - pr_warn("** **\n"); 2875 - pr_warn("** This means that this is a DEBUG kernel and it is **\n"); 2876 - pr_warn("** unsafe for production use. **\n"); 2877 - pr_warn("** **\n"); 2878 - pr_warn("** If you see this message and you are not debugging **\n"); 2879 - pr_warn("** the kernel, report this immediately to your vendor! **\n"); 2880 - pr_warn("** **\n"); 2881 - pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); 2882 - pr_warn("**********************************************************\n"); 2883 - 2884 - /* Expand the buffers to set size */ 2885 - tracing_update_buffers(&global_trace); 2886 - 2887 - buffers_allocated = 1; 2888 - 2889 - /* 2890 - * trace_printk_init_buffers() can be called by modules. 2891 - * If that happens, then we need to start cmdline recording 2892 - * directly here. If the global_trace.buffer is already 2893 - * allocated here, then this was called by module code. 2894 - */ 2895 - if (global_trace.array_buffer.buffer) 2896 - tracing_start_cmdline_record(); 2897 - } 2898 - EXPORT_SYMBOL_GPL(trace_printk_init_buffers); 2899 - 2900 - void trace_printk_start_comm(void) 2901 - { 2902 - /* Start tracing comms if trace printk is set */ 2903 - if (!buffers_allocated) 2904 - return; 2905 - tracing_start_cmdline_record(); 2906 - } 2907 - 2908 - static void trace_printk_start_stop_comm(int enabled) 2909 - { 2910 - if (!buffers_allocated) 2911 - return; 2912 - 2913 - if (enabled) 2914 - tracing_start_cmdline_record(); 2915 - else 2916 - tracing_stop_cmdline_record(); 2917 - } 2918 - 2919 - /** 2920 - * trace_vbprintk - write binary msg to tracing buffer 2921 - * @ip: The address of the caller 2922 - * @fmt: The string format to write to the buffer 2923 - * @args: Arguments for @fmt 2924 - */ 2925 - int trace_vbprintk(unsigned long ip, const char *fmt, va_list args) 2926 - { 2927 - struct ring_buffer_event *event; 2928 - struct trace_buffer *buffer; 2929 - struct trace_array *tr = READ_ONCE(printk_trace); 2930 - struct bprint_entry *entry; 2931 - unsigned int trace_ctx; 2932 - char *tbuffer; 2933 - int len = 0, size; 2934 - 2935 - if (!printk_binsafe(tr)) 2936 - return trace_vprintk(ip, fmt, args); 2937 - 2938 - if (unlikely(tracing_selftest_running || tracing_disabled)) 2939 - return 0; 2940 - 2941 - /* Don't pollute graph traces with trace_vprintk internals */ 2942 - pause_graph_tracing(); 2943 - 2944 - trace_ctx = tracing_gen_ctx(); 2945 - guard(preempt_notrace)(); 2946 - 2947 - tbuffer = get_trace_buf(); 2948 - if (!tbuffer) { 2949 - len = 0; 2950 - goto out_nobuffer; 2951 - } 2952 - 2953 - len = vbin_printf((u32 *)tbuffer, TRACE_BUF_SIZE/sizeof(int), fmt, args); 2954 - 2955 - if (len > TRACE_BUF_SIZE/sizeof(int) || len < 0) 2956 - goto out_put; 2957 - 2958 - size = sizeof(*entry) + sizeof(u32) * len; 2959 - buffer = tr->array_buffer.buffer; 2960 - scoped_guard(ring_buffer_nest, buffer) { 2961 - event = __trace_buffer_lock_reserve(buffer, TRACE_BPRINT, size, 2962 - trace_ctx); 2963 - if (!event) 2964 - goto out_put; 2965 - entry = ring_buffer_event_data(event); 2966 - entry->ip = ip; 2967 - entry->fmt = fmt; 2968 - 2969 - memcpy(entry->buf, tbuffer, sizeof(u32) * len); 2970 - __buffer_unlock_commit(buffer, event); 2971 - ftrace_trace_stack(tr, buffer, trace_ctx, 6, NULL); 2972 - } 2973 - out_put: 2974 - put_trace_buf(); 2975 - 2976 - out_nobuffer: 2977 - unpause_graph_tracing(); 2978 - 2979 - return len; 2980 - } 2981 - EXPORT_SYMBOL_GPL(trace_vbprintk); 2982 - 2983 - static __printf(3, 0) 2984 - int __trace_array_vprintk(struct trace_buffer *buffer, 2985 - unsigned long ip, const char *fmt, va_list args) 2986 - { 2987 - struct ring_buffer_event *event; 2988 - int len = 0, size; 2989 - struct print_entry *entry; 2990 - unsigned int trace_ctx; 2991 - char *tbuffer; 2992 - 2993 - if (tracing_disabled) 2994 - return 0; 2995 - 2996 - /* Don't pollute graph traces with trace_vprintk internals */ 2997 - pause_graph_tracing(); 2998 - 2999 - trace_ctx = tracing_gen_ctx(); 3000 - guard(preempt_notrace)(); 3001 - 3002 - 3003 - tbuffer = get_trace_buf(); 3004 - if (!tbuffer) { 3005 - len = 0; 3006 - goto out_nobuffer; 3007 - } 3008 - 3009 - len = vscnprintf(tbuffer, TRACE_BUF_SIZE, fmt, args); 3010 - 3011 - size = sizeof(*entry) + len + 1; 3012 - scoped_guard(ring_buffer_nest, buffer) { 3013 - event = __trace_buffer_lock_reserve(buffer, TRACE_PRINT, size, 3014 - trace_ctx); 3015 - if (!event) 3016 - goto out; 3017 - entry = ring_buffer_event_data(event); 3018 - entry->ip = ip; 3019 - 3020 - memcpy(&entry->buf, tbuffer, len + 1); 3021 - __buffer_unlock_commit(buffer, event); 3022 - ftrace_trace_stack(printk_trace, buffer, trace_ctx, 6, NULL); 3023 - } 3024 - out: 3025 - put_trace_buf(); 3026 - 3027 - out_nobuffer: 3028 - unpause_graph_tracing(); 3029 - 3030 - return len; 3031 - } 3032 - 3033 - int trace_array_vprintk(struct trace_array *tr, 3034 - unsigned long ip, const char *fmt, va_list args) 3035 - { 3036 - if (tracing_selftest_running && tr == &global_trace) 3037 - return 0; 3038 - 3039 - return __trace_array_vprintk(tr->array_buffer.buffer, ip, fmt, args); 3040 - } 3041 - 3042 - /** 3043 - * trace_array_printk - Print a message to a specific instance 3044 - * @tr: The instance trace_array descriptor 3045 - * @ip: The instruction pointer that this is called from. 3046 - * @fmt: The format to print (printf format) 3047 - * 3048 - * If a subsystem sets up its own instance, they have the right to 3049 - * printk strings into their tracing instance buffer using this 3050 - * function. Note, this function will not write into the top level 3051 - * buffer (use trace_printk() for that), as writing into the top level 3052 - * buffer should only have events that can be individually disabled. 3053 - * trace_printk() is only used for debugging a kernel, and should not 3054 - * be ever incorporated in normal use. 3055 - * 3056 - * trace_array_printk() can be used, as it will not add noise to the 3057 - * top level tracing buffer. 3058 - * 3059 - * Note, trace_array_init_printk() must be called on @tr before this 3060 - * can be used. 3061 - */ 3062 - int trace_array_printk(struct trace_array *tr, 3063 - unsigned long ip, const char *fmt, ...) 3064 - { 3065 - int ret; 3066 - va_list ap; 3067 - 3068 - if (!tr) 3069 - return -ENOENT; 3070 - 3071 - /* This is only allowed for created instances */ 3072 - if (tr == &global_trace) 3073 - return 0; 3074 - 3075 - if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 3076 - return 0; 3077 - 3078 - va_start(ap, fmt); 3079 - ret = trace_array_vprintk(tr, ip, fmt, ap); 3080 - va_end(ap); 3081 - return ret; 3082 - } 3083 - EXPORT_SYMBOL_GPL(trace_array_printk); 3084 - 3085 - /** 3086 - * trace_array_init_printk - Initialize buffers for trace_array_printk() 3087 - * @tr: The trace array to initialize the buffers for 3088 - * 3089 - * As trace_array_printk() only writes into instances, they are OK to 3090 - * have in the kernel (unlike trace_printk()). This needs to be called 3091 - * before trace_array_printk() can be used on a trace_array. 3092 - */ 3093 - int trace_array_init_printk(struct trace_array *tr) 3094 - { 3095 - if (!tr) 3096 - return -ENOENT; 3097 - 3098 - /* This is only allowed for created instances */ 3099 - if (tr == &global_trace) 3100 - return -EINVAL; 3101 - 3102 - return alloc_percpu_trace_buffer(); 3103 - } 3104 - EXPORT_SYMBOL_GPL(trace_array_init_printk); 3105 - 3106 - int trace_array_printk_buf(struct trace_buffer *buffer, 3107 - unsigned long ip, const char *fmt, ...) 3108 - { 3109 - int ret; 3110 - va_list ap; 3111 - 3112 - if (!(printk_trace->trace_flags & TRACE_ITER(PRINTK))) 3113 - return 0; 3114 - 3115 - va_start(ap, fmt); 3116 - ret = __trace_array_vprintk(buffer, ip, fmt, ap); 3117 - va_end(ap); 3118 - return ret; 3119 - } 3120 - 3121 - int trace_vprintk(unsigned long ip, const char *fmt, va_list args) 3122 - { 3123 - return trace_array_vprintk(printk_trace, ip, fmt, args); 3124 - } 3125 - EXPORT_SYMBOL_GPL(trace_vprintk); 3126 3234 3127 3235 static void trace_iterator_increment(struct trace_iterator *iter) 3128 3236 { ··· 3242 3986 } 3243 3987 mutex_unlock(&trace_types_lock); 3244 3988 3245 - #ifdef CONFIG_TRACER_MAX_TRACE 3246 - if (iter->snapshot && iter->trace->use_max_tr) 3989 + if (iter->snapshot && tracer_uses_snapshot(iter->trace)) 3247 3990 return ERR_PTR(-EBUSY); 3248 - #endif 3249 3991 3250 3992 if (*pos != iter->pos) { 3251 3993 iter->ent = NULL; ··· 3282 4028 { 3283 4029 struct trace_iterator *iter = m->private; 3284 4030 3285 - #ifdef CONFIG_TRACER_MAX_TRACE 3286 - if (iter->snapshot && iter->trace->use_max_tr) 4031 + if (iter->snapshot && tracer_uses_snapshot(iter->trace)) 3287 4032 return; 3288 - #endif 3289 4033 3290 4034 trace_access_unlock(iter->cpu_file); 3291 4035 trace_event_read_unlock(); ··· 3537 4285 /* ftrace and system call events are still OK */ 3538 4286 if ((event->type > __TRACE_LAST_TYPE) && 3539 4287 !is_syscall_event(event)) 3540 - return print_event_fields(iter, event); 4288 + return print_event_fields(iter, event); 3541 4289 } 3542 4290 return event->funcs->trace(iter, sym_flags, event); 3543 4291 } ··· 3760 4508 "# MAY BE MISSING FUNCTION EVENTS\n"); 3761 4509 } 3762 4510 3763 - #ifdef CONFIG_TRACER_MAX_TRACE 4511 + #ifdef CONFIG_TRACER_SNAPSHOT 3764 4512 static void show_snapshot_main_help(struct seq_file *m) 3765 4513 { 3766 4514 seq_puts(m, "# echo 0 > snapshot : Clears and frees snapshot buffer\n" ··· 3938 4686 3939 4687 iter->tr = tr; 3940 4688 3941 - #ifdef CONFIG_TRACER_MAX_TRACE 4689 + #ifdef CONFIG_TRACER_SNAPSHOT 3942 4690 /* Currently only the top directory has a snapshot */ 3943 4691 if (tr->current_trace->print_max || snapshot) 3944 - iter->array_buffer = &tr->max_buffer; 4692 + iter->array_buffer = &tr->snapshot_buffer; 3945 4693 else 3946 4694 #endif 3947 4695 iter->array_buffer = &tr->array_buffer; ··· 4008 4756 4009 4757 filp->private_data = inode->i_private; 4010 4758 return 0; 4011 - } 4012 - 4013 - bool tracing_is_disabled(void) 4014 - { 4015 - return (tracing_disabled) ? true: false; 4016 4759 } 4017 4760 4018 4761 /* ··· 4127 4880 return single_release(inode, file); 4128 4881 } 4129 4882 4883 + static bool update_last_data_if_empty(struct trace_array *tr); 4884 + 4130 4885 static int tracing_open(struct inode *inode, struct file *file) 4131 4886 { 4132 4887 struct trace_array *tr = inode->i_private; ··· 4146 4897 4147 4898 #ifdef CONFIG_TRACER_MAX_TRACE 4148 4899 if (tr->current_trace->print_max) 4149 - trace_buf = &tr->max_buffer; 4900 + trace_buf = &tr->snapshot_buffer; 4150 4901 #endif 4151 4902 4152 4903 if (cpu == RING_BUFFER_ALL_CPUS) 4153 4904 tracing_reset_online_cpus(trace_buf); 4154 4905 else 4155 4906 tracing_reset_cpu(trace_buf, cpu); 4907 + 4908 + update_last_data_if_empty(tr); 4156 4909 } 4157 4910 4158 4911 if (file->f_mode & FMODE_READ) { ··· 4179 4928 static bool 4180 4929 trace_ok_for_array(struct tracer *t, struct trace_array *tr) 4181 4930 { 4182 - #ifdef CONFIG_TRACER_SNAPSHOT 4183 4931 /* arrays with mapped buffer range do not have snapshots */ 4184 - if (tr->range_addr_start && t->use_max_tr) 4932 + if (tr->range_addr_start && tracer_uses_snapshot(t)) 4185 4933 return false; 4186 - #endif 4187 4934 return (tr->flags & TRACE_ARRAY_FL_GLOBAL) || t->allow_instances; 4188 4935 } 4189 4936 ··· 4358 5109 if (cpumask_test_cpu(cpu, tr->tracing_cpumask) && 4359 5110 !cpumask_test_cpu(cpu, tracing_cpumask_new)) { 4360 5111 ring_buffer_record_disable_cpu(tr->array_buffer.buffer, cpu); 4361 - #ifdef CONFIG_TRACER_MAX_TRACE 4362 - ring_buffer_record_disable_cpu(tr->max_buffer.buffer, cpu); 5112 + #ifdef CONFIG_TRACER_SNAPSHOT 5113 + ring_buffer_record_disable_cpu(tr->snapshot_buffer.buffer, cpu); 4363 5114 #endif 4364 5115 } 4365 5116 if (!cpumask_test_cpu(cpu, tr->tracing_cpumask) && 4366 5117 cpumask_test_cpu(cpu, tracing_cpumask_new)) { 4367 5118 ring_buffer_record_enable_cpu(tr->array_buffer.buffer, cpu); 4368 - #ifdef CONFIG_TRACER_MAX_TRACE 4369 - ring_buffer_record_enable_cpu(tr->max_buffer.buffer, cpu); 5119 + #ifdef CONFIG_TRACER_SNAPSHOT 5120 + ring_buffer_record_enable_cpu(tr->snapshot_buffer.buffer, cpu); 4370 5121 #endif 4371 5122 } 4372 5123 } ··· 4575 5326 4576 5327 case TRACE_ITER(OVERWRITE): 4577 5328 ring_buffer_change_overwrite(tr->array_buffer.buffer, enabled); 4578 - #ifdef CONFIG_TRACER_MAX_TRACE 4579 - ring_buffer_change_overwrite(tr->max_buffer.buffer, enabled); 5329 + #ifdef CONFIG_TRACER_SNAPSHOT 5330 + ring_buffer_change_overwrite(tr->snapshot_buffer.buffer, enabled); 4580 5331 #endif 4581 5332 break; 4582 5333 ··· 5219 5970 int tracer_init(struct tracer *t, struct trace_array *tr) 5220 5971 { 5221 5972 tracing_reset_online_cpus(&tr->array_buffer); 5973 + update_last_data_if_empty(tr); 5222 5974 return t->init(tr); 5223 5975 } 5224 5976 ··· 5240 5990 } 5241 5991 } 5242 5992 5243 - #ifdef CONFIG_TRACER_MAX_TRACE 5993 + #ifdef CONFIG_TRACER_SNAPSHOT 5244 5994 /* resize @tr's buffer to the size of @size_tr's entries */ 5245 5995 static int resize_buffer_duplicate_size(struct array_buffer *trace_buf, 5246 5996 struct array_buffer *size_buf, int cpu_id) ··· 5266 6016 5267 6017 return ret; 5268 6018 } 5269 - #endif /* CONFIG_TRACER_MAX_TRACE */ 6019 + #endif /* CONFIG_TRACER_SNAPSHOT */ 5270 6020 5271 6021 static int __tracing_resize_ring_buffer(struct trace_array *tr, 5272 6022 unsigned long size, int cpu) ··· 5291 6041 if (ret < 0) 5292 6042 goto out_start; 5293 6043 5294 - #ifdef CONFIG_TRACER_MAX_TRACE 6044 + #ifdef CONFIG_TRACER_SNAPSHOT 5295 6045 if (!tr->allocated_snapshot) 5296 6046 goto out; 5297 6047 5298 - ret = ring_buffer_resize(tr->max_buffer.buffer, size, cpu); 6048 + ret = ring_buffer_resize(tr->snapshot_buffer.buffer, size, cpu); 5299 6049 if (ret < 0) { 5300 6050 int r = resize_buffer_duplicate_size(&tr->array_buffer, 5301 6051 &tr->array_buffer, cpu); ··· 5320 6070 goto out_start; 5321 6071 } 5322 6072 5323 - update_buffer_entries(&tr->max_buffer, cpu); 6073 + update_buffer_entries(&tr->snapshot_buffer, cpu); 5324 6074 5325 6075 out: 5326 - #endif /* CONFIG_TRACER_MAX_TRACE */ 6076 + #endif /* CONFIG_TRACER_SNAPSHOT */ 5327 6077 5328 6078 update_buffer_entries(&tr->array_buffer, cpu); 5329 6079 out_start: ··· 5514 6264 { 5515 6265 int ret = 0; 5516 6266 6267 + if (!tr) 6268 + tr = &global_trace; 6269 + 5517 6270 guard(mutex)(&trace_types_lock); 5518 6271 5519 6272 update_last_data(tr); ··· 5551 6298 { 5552 6299 struct tracer *trace = NULL; 5553 6300 struct tracers *t; 5554 - #ifdef CONFIG_TRACER_MAX_TRACE 5555 6301 bool had_max_tr; 5556 - #endif 5557 6302 int ret; 5558 6303 5559 6304 guard(mutex)(&trace_types_lock); ··· 5579 6328 return 0; 5580 6329 5581 6330 #ifdef CONFIG_TRACER_SNAPSHOT 5582 - if (trace->use_max_tr) { 6331 + if (tracer_uses_snapshot(trace)) { 5583 6332 local_irq_disable(); 5584 6333 arch_spin_lock(&tr->max_lock); 5585 6334 ret = tr->cond_snapshot ? -EBUSY : 0; ··· 5611 6360 if (tr->current_trace->reset) 5612 6361 tr->current_trace->reset(tr); 5613 6362 5614 - #ifdef CONFIG_TRACER_MAX_TRACE 5615 - had_max_tr = tr->current_trace->use_max_tr; 6363 + had_max_tr = tracer_uses_snapshot(tr->current_trace); 5616 6364 5617 6365 /* Current trace needs to be nop_trace before synchronize_rcu */ 5618 6366 tr->current_trace = &nop_trace; 5619 6367 tr->current_trace_flags = nop_trace.flags; 5620 6368 5621 - if (had_max_tr && !trace->use_max_tr) { 6369 + if (had_max_tr && !tracer_uses_snapshot(trace)) { 5622 6370 /* 5623 6371 * We need to make sure that the update_max_tr sees that 5624 6372 * current_trace changed to nop_trace to keep it from ··· 5630 6380 tracing_disarm_snapshot(tr); 5631 6381 } 5632 6382 5633 - if (!had_max_tr && trace->use_max_tr) { 6383 + if (!had_max_tr && tracer_uses_snapshot(trace)) { 5634 6384 ret = tracing_arm_snapshot_locked(tr); 5635 6385 if (ret) 5636 6386 return ret; 5637 6387 } 5638 - #else 5639 - tr->current_trace = &nop_trace; 5640 - #endif 5641 6388 5642 6389 tr->current_trace_flags = t->flags ? : t->tracer->flags; 5643 6390 5644 6391 if (trace->init) { 5645 6392 ret = tracer_init(trace, tr); 5646 6393 if (ret) { 5647 - #ifdef CONFIG_TRACER_MAX_TRACE 5648 - if (trace->use_max_tr) 6394 + if (tracer_uses_snapshot(trace)) 5649 6395 tracing_disarm_snapshot(tr); 5650 - #endif 5651 6396 tr->current_trace_flags = nop_trace.flags; 5652 6397 return ret; 5653 6398 } ··· 6847 7602 unsigned long ip; 6848 7603 char *buf; 6849 7604 6850 - if (tracing_disabled) 7605 + if (unlikely(tracing_disabled)) 6851 7606 return -EINVAL; 6852 7607 6853 7608 if (!(tr->trace_flags & TRACE_ITER(MARKERS))) ··· 6927 7682 ssize_t written = -ENODEV; 6928 7683 char *buf; 6929 7684 6930 - if (tracing_disabled) 7685 + if (unlikely(tracing_disabled)) 6931 7686 return -EINVAL; 6932 7687 6933 7688 if (!(tr->trace_flags & TRACE_ITER(MARKERS))) ··· 7028 7783 */ 7029 7784 tracing_reset_online_cpus(&tr->array_buffer); 7030 7785 7031 - #ifdef CONFIG_TRACER_MAX_TRACE 7032 - if (tr->max_buffer.buffer) 7033 - ring_buffer_set_clock(tr->max_buffer.buffer, trace_clocks[i].func); 7034 - tracing_reset_online_cpus(&tr->max_buffer); 7786 + #ifdef CONFIG_TRACER_SNAPSHOT 7787 + if (tr->snapshot_buffer.buffer) 7788 + ring_buffer_set_clock(tr->snapshot_buffer.buffer, trace_clocks[i].func); 7789 + tracing_reset_online_cpus(&tr->snapshot_buffer); 7035 7790 #endif 7791 + update_last_data_if_empty(tr); 7036 7792 7037 7793 if (tr->scratch && !(tr->flags & TRACE_ARRAY_FL_LAST_BOOT)) { 7038 7794 struct trace_scratch *tscratch = tr->scratch; ··· 7126 7880 return ring_buffer_event_time_stamp(buffer, rbe); 7127 7881 } 7128 7882 7129 - /* 7130 - * Set or disable using the per CPU trace_buffer_event when possible. 7131 - */ 7132 - int tracing_set_filter_buffering(struct trace_array *tr, bool set) 7133 - { 7134 - guard(mutex)(&trace_types_lock); 7135 - 7136 - if (set && tr->no_filter_buffering_ref++) 7137 - return 0; 7138 - 7139 - if (!set) { 7140 - if (WARN_ON_ONCE(!tr->no_filter_buffering_ref)) 7141 - return -EINVAL; 7142 - 7143 - --tr->no_filter_buffering_ref; 7144 - } 7145 - 7146 - return 0; 7147 - } 7148 - 7149 7883 struct ftrace_buffer_info { 7150 7884 struct trace_iterator iter; 7151 7885 void *spare; ··· 7164 7938 ret = 0; 7165 7939 7166 7940 iter->tr = tr; 7167 - iter->array_buffer = &tr->max_buffer; 7941 + iter->array_buffer = &tr->snapshot_buffer; 7168 7942 iter->cpu_file = tracing_get_cpu(inode); 7169 7943 m->private = iter; 7170 7944 file->private_data = m; ··· 7201 7975 7202 7976 guard(mutex)(&trace_types_lock); 7203 7977 7204 - if (tr->current_trace->use_max_tr) 7978 + if (tracer_uses_snapshot(tr->current_trace)) 7205 7979 return -EBUSY; 7206 7980 7207 7981 local_irq_disable(); ··· 7227 8001 return -EINVAL; 7228 8002 #endif 7229 8003 if (tr->allocated_snapshot) 7230 - ret = resize_buffer_duplicate_size(&tr->max_buffer, 8004 + ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 7231 8005 &tr->array_buffer, iter->cpu_file); 7232 8006 7233 8007 ret = tracing_arm_snapshot_locked(tr); ··· 7248 8022 default: 7249 8023 if (tr->allocated_snapshot) { 7250 8024 if (iter->cpu_file == RING_BUFFER_ALL_CPUS) 7251 - tracing_reset_online_cpus(&tr->max_buffer); 8025 + tracing_reset_online_cpus(&tr->snapshot_buffer); 7252 8026 else 7253 - tracing_reset_cpu(&tr->max_buffer, iter->cpu_file); 8027 + tracing_reset_cpu(&tr->snapshot_buffer, iter->cpu_file); 7254 8028 } 7255 8029 break; 7256 8030 } ··· 7300 8074 7301 8075 info = filp->private_data; 7302 8076 7303 - if (info->iter.trace->use_max_tr) { 8077 + if (tracer_uses_snapshot(info->iter.trace)) { 7304 8078 tracing_buffers_release(inode, filp); 7305 8079 return -EBUSY; 7306 8080 } 7307 8081 7308 8082 info->iter.snapshot = true; 7309 - info->iter.array_buffer = &info->iter.tr->max_buffer; 8083 + info->iter.array_buffer = &info->iter.tr->snapshot_buffer; 7310 8084 7311 8085 return ret; 7312 8086 } ··· 7856 8630 if (!count) 7857 8631 return 0; 7858 8632 7859 - #ifdef CONFIG_TRACER_MAX_TRACE 7860 - if (iter->snapshot && iter->tr->current_trace->use_max_tr) 8633 + if (iter->snapshot && tracer_uses_snapshot(iter->tr->current_trace)) 7861 8634 return -EBUSY; 7862 - #endif 7863 8635 7864 8636 page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer); 7865 8637 ··· 8041 8817 int entries, i; 8042 8818 ssize_t ret = 0; 8043 8819 8044 - #ifdef CONFIG_TRACER_MAX_TRACE 8045 - if (iter->snapshot && iter->tr->current_trace->use_max_tr) 8820 + if (iter->snapshot && tracer_uses_snapshot(iter->tr->current_trace)) 8046 8821 return -EBUSY; 8047 - #endif 8048 8822 8049 8823 page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer); 8050 8824 if (*ppos & (page_size - 1)) ··· 8176 8954 return 0; 8177 8955 } 8178 8956 8179 - #ifdef CONFIG_TRACER_MAX_TRACE 8957 + #ifdef CONFIG_TRACER_SNAPSHOT 8180 8958 static int get_snapshot_map(struct trace_array *tr) 8181 8959 { 8182 8960 int err = 0; ··· 8619 9397 trace_create_cpu_file("stats", TRACE_MODE_READ, d_cpu, 8620 9398 tr, cpu, &tracing_stats_fops); 8621 9399 8622 - trace_create_cpu_file("buffer_size_kb", TRACE_MODE_READ, d_cpu, 9400 + trace_create_cpu_file("buffer_size_kb", TRACE_MODE_WRITE, d_cpu, 8623 9401 tr, cpu, &tracing_entries_fops); 8624 9402 8625 9403 if (tr->range_addr_start) ··· 9180 9958 if (ret) 9181 9959 goto out; 9182 9960 9183 - #ifdef CONFIG_TRACER_MAX_TRACE 9961 + #ifdef CONFIG_TRACER_SNAPSHOT 9184 9962 9185 9963 if (!tr->allocated_snapshot) 9186 9964 goto out_max; 9187 9965 9188 - ret = ring_buffer_subbuf_order_set(tr->max_buffer.buffer, order); 9966 + ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order); 9189 9967 if (ret) { 9190 9968 /* Put back the old order */ 9191 9969 cnt = ring_buffer_subbuf_order_set(tr->array_buffer.buffer, old_order); ··· 9401 10179 if (ret) 9402 10180 return ret; 9403 10181 9404 - #ifdef CONFIG_TRACER_MAX_TRACE 10182 + #ifdef CONFIG_TRACER_SNAPSHOT 9405 10183 /* Fix mapped buffer trace arrays do not have snapshot buffers */ 9406 10184 if (tr->range_addr_start) 9407 10185 return 0; 9408 10186 9409 - ret = allocate_trace_buffer(tr, &tr->max_buffer, 10187 + ret = allocate_trace_buffer(tr, &tr->snapshot_buffer, 9410 10188 allocate_snapshot ? size : 1); 9411 10189 if (MEM_FAIL(ret, "Failed to allocate trace buffer\n")) { 9412 10190 free_trace_buffer(&tr->array_buffer); ··· 9428 10206 free_trace_buffer(&tr->array_buffer); 9429 10207 kfree(tr->module_delta); 9430 10208 9431 - #ifdef CONFIG_TRACER_MAX_TRACE 9432 - free_trace_buffer(&tr->max_buffer); 10209 + #ifdef CONFIG_TRACER_SNAPSHOT 10210 + free_trace_buffer(&tr->snapshot_buffer); 9433 10211 #endif 9434 10212 } 9435 10213 ··· 9570 10348 tr->syscall_buf_sz = global_trace.syscall_buf_sz; 9571 10349 9572 10350 tr->max_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; 9573 - #ifdef CONFIG_TRACER_MAX_TRACE 10351 + #ifdef CONFIG_TRACER_SNAPSHOT 9574 10352 spin_lock_init(&tr->snapshot_trigger_lock); 9575 10353 #endif 9576 10354 tr->current_trace = &nop_trace; ··· 9895 10673 9896 10674 create_trace_options_dir(tr); 9897 10675 9898 - #ifdef CONFIG_TRACER_MAX_TRACE 9899 10676 trace_create_maxlat_file(tr, d_tracer); 9900 - #endif 9901 10677 9902 10678 if (ftrace_create_function_files(tr, d_tracer)) 9903 10679 MEM_FAIL(1, "Could not allocate function filter files"); ··· 9994 10774 extern struct trace_eval_map *__start_ftrace_eval_maps[]; 9995 10775 extern struct trace_eval_map *__stop_ftrace_eval_maps[]; 9996 10776 9997 - static struct workqueue_struct *eval_map_wq __initdata; 10777 + struct workqueue_struct *trace_init_wq __initdata; 9998 10778 static struct work_struct eval_map_work __initdata; 9999 10779 static struct work_struct tracerfs_init_work __initdata; 10000 10780 ··· 10010 10790 { 10011 10791 INIT_WORK(&eval_map_work, eval_map_work_func); 10012 10792 10013 - eval_map_wq = alloc_workqueue("eval_map_wq", WQ_UNBOUND, 0); 10014 - if (!eval_map_wq) { 10015 - pr_err("Unable to allocate eval_map_wq\n"); 10793 + trace_init_wq = alloc_workqueue("trace_init_wq", WQ_UNBOUND, 0); 10794 + if (!trace_init_wq) { 10795 + pr_err("Unable to allocate trace_init_wq\n"); 10016 10796 /* Do work here */ 10017 10797 eval_map_work_func(&eval_map_work); 10018 10798 return -ENOMEM; 10019 10799 } 10020 10800 10021 - queue_work(eval_map_wq, &eval_map_work); 10801 + queue_work(trace_init_wq, &eval_map_work); 10022 10802 return 0; 10023 10803 } 10024 10804 ··· 10027 10807 static int __init trace_eval_sync(void) 10028 10808 { 10029 10809 /* Make sure the eval map updates are finished */ 10030 - if (eval_map_wq) 10031 - destroy_workqueue(eval_map_wq); 10810 + if (trace_init_wq) 10811 + destroy_workqueue(trace_init_wq); 10032 10812 return 0; 10033 10813 } 10034 10814 ··· 10189 10969 if (ret) 10190 10970 return 0; 10191 10971 10192 - if (eval_map_wq) { 10972 + if (trace_init_wq) { 10193 10973 INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func); 10194 - queue_work(eval_map_wq, &tracerfs_init_work); 10974 + queue_work(trace_init_wq, &tracerfs_init_work); 10195 10975 } else { 10196 10976 tracer_init_tracefs_work_func(NULL); 10197 10977 } ··· 10524 11304 return done; 10525 11305 } 10526 11306 10527 - #ifdef CONFIG_TRACER_MAX_TRACE 11307 + #ifdef CONFIG_TRACER_SNAPSHOT 10528 11308 __init static bool tr_needs_alloc_snapshot(const char *name) 10529 11309 { 10530 11310 char *test; ··· 10714 11494 } 10715 11495 } else { 10716 11496 /* Only non mapped buffers have snapshot buffers */ 10717 - if (IS_ENABLED(CONFIG_TRACER_MAX_TRACE)) 11497 + if (IS_ENABLED(CONFIG_TRACER_SNAPSHOT)) 10718 11498 do_allocate_snapshot(name); 10719 11499 } 10720 11500 ··· 10841 11621 global_trace.current_trace_flags = nop_trace.flags; 10842 11622 10843 11623 global_trace.max_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; 10844 - #ifdef CONFIG_TRACER_MAX_TRACE 11624 + #ifdef CONFIG_TRACER_SNAPSHOT 10845 11625 spin_lock_init(&global_trace.snapshot_trigger_lock); 10846 11626 #endif 10847 11627 ftrace_init_global_array_ops(&global_trace); ··· 10909 11689 10910 11690 void __init ftrace_boot_snapshot(void) 10911 11691 { 10912 - #ifdef CONFIG_TRACER_MAX_TRACE 11692 + #ifdef CONFIG_TRACER_SNAPSHOT 10913 11693 struct trace_array *tr; 10914 11694 10915 11695 if (!snapshot_at_boot)
+113 -18
kernel/trace/trace.h
··· 131 131 132 132 #define FAULT_STRING "(fault)" 133 133 134 - #define HIST_STACKTRACE_DEPTH 16 134 + #define HIST_STACKTRACE_DEPTH 31 135 135 #define HIST_STACKTRACE_SIZE (HIST_STACKTRACE_DEPTH * sizeof(unsigned long)) 136 136 #define HIST_STACKTRACE_SKIP 5 137 137 ··· 332 332 struct list_head list; 333 333 char *name; 334 334 struct array_buffer array_buffer; 335 - #ifdef CONFIG_TRACER_MAX_TRACE 335 + #ifdef CONFIG_TRACER_SNAPSHOT 336 336 /* 337 - * The max_buffer is used to snapshot the trace when a maximum 337 + * The snapshot_buffer is used to snapshot the trace when a maximum 338 338 * latency is reached, or when the user initiates a snapshot. 339 339 * Some tracers will use this to store a maximum trace while 340 340 * it continues examining live traces. 341 341 * 342 - * The buffers for the max_buffer are set up the same as the array_buffer 343 - * When a snapshot is taken, the buffer of the max_buffer is swapped 344 - * with the buffer of the array_buffer and the buffers are reset for 345 - * the array_buffer so the tracing can continue. 342 + * The buffers for the snapshot_buffer are set up the same as the 343 + * array_buffer. When a snapshot is taken, the buffer of the 344 + * snapshot_buffer is swapped with the buffer of the array_buffer 345 + * and the buffers are reset for the array_buffer so the tracing can 346 + * continue. 346 347 */ 347 - struct array_buffer max_buffer; 348 + struct array_buffer snapshot_buffer; 348 349 bool allocated_snapshot; 349 350 spinlock_t snapshot_trigger_lock; 350 351 unsigned int snapshot; 352 + #ifdef CONFIG_TRACER_MAX_TRACE 351 353 unsigned long max_latency; 352 - #ifdef CONFIG_FSNOTIFY 353 354 struct dentry *d_max_latency; 355 + #ifdef CONFIG_FSNOTIFY 354 356 struct work_struct fsnotify_work; 355 357 struct irq_work fsnotify_irqwork; 356 - #endif 357 - #endif 358 + #endif /* CONFIG_FSNOTIFY */ 359 + #endif /* CONFIG_TRACER_MAX_TRACE */ 360 + #endif /* CONFIG_TRACER_SNAPSHOT */ 361 + 358 362 /* The below is for memory mapped ring buffer */ 359 363 unsigned int mapped; 360 364 unsigned long range_addr_start; ··· 384 380 * 385 381 * It is also used in other places outside the update_max_tr 386 382 * so it needs to be defined outside of the 387 - * CONFIG_TRACER_MAX_TRACE. 383 + * CONFIG_TRACER_SNAPSHOT. 388 384 */ 389 385 arch_spinlock_t max_lock; 390 386 #ifdef CONFIG_FTRACE_SYSCALLS ··· 483 479 extern struct trace_array *trace_array_find_get(const char *instance); 484 480 485 481 extern u64 tracing_event_time_stamp(struct trace_buffer *buffer, struct ring_buffer_event *rbe); 486 - extern int tracing_set_filter_buffering(struct trace_array *tr, bool set); 487 482 extern int tracing_set_clock(struct trace_array *tr, const char *clockstr); 488 483 489 484 extern bool trace_clock_in_ns(struct trace_array *tr); 490 485 491 486 extern unsigned long trace_adjust_address(struct trace_array *tr, unsigned long addr); 487 + 488 + extern struct trace_array *printk_trace; 492 489 493 490 /* 494 491 * The global tracer (top) should be the first trace array added, ··· 666 661 return iter->buffer_iter ? iter->buffer_iter[cpu] : NULL; 667 662 } 668 663 664 + extern int tracing_disabled; 665 + 669 666 int tracer_init(struct tracer *t, struct trace_array *tr); 670 667 int tracing_is_enabled(void); 671 668 void tracing_reset_online_cpus(struct array_buffer *buf); ··· 679 672 int tracing_open_file_tr(struct inode *inode, struct file *filp); 680 673 int tracing_release_file_tr(struct inode *inode, struct file *filp); 681 674 int tracing_single_release_file_tr(struct inode *inode, struct file *filp); 682 - bool tracing_is_disabled(void); 683 675 bool tracer_tracing_is_on(struct trace_array *tr); 684 676 void tracer_tracing_on(struct trace_array *tr); 685 677 void tracer_tracing_off(struct trace_array *tr); ··· 778 772 extern unsigned long nsecs_to_usecs(unsigned long nsecs); 779 773 780 774 extern unsigned long tracing_thresh; 775 + extern struct workqueue_struct *trace_init_wq __initdata; 781 776 782 777 /* PID filtering */ 783 778 ··· 797 790 struct trace_pid_list **new_pid_list, 798 791 const char __user *ubuf, size_t cnt); 799 792 800 - #ifdef CONFIG_TRACER_MAX_TRACE 793 + #ifdef CONFIG_TRACER_SNAPSHOT 801 794 void update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu, 802 795 void *cond_data); 803 796 void update_max_tr_single(struct trace_array *tr, 804 797 struct task_struct *tsk, int cpu); 805 798 806 - #ifdef CONFIG_FSNOTIFY 807 - #define LATENCY_FS_NOTIFY 799 + #if defined(CONFIG_TRACER_MAX_TRACE) && defined(CONFIG_FSNOTIFY) 800 + # define LATENCY_FS_NOTIFY 808 801 #endif 809 - #endif /* CONFIG_TRACER_MAX_TRACE */ 810 802 811 803 #ifdef LATENCY_FS_NOTIFY 812 804 void latency_fsnotify(struct trace_array *tr); 813 805 #else 814 806 static inline void latency_fsnotify(struct trace_array *tr) { } 815 807 #endif 808 + #endif /* CONFIG_TRACER_SNAPSHOT */ 816 809 817 810 #ifdef CONFIG_STACKTRACE 818 811 void __trace_stack(struct trace_array *tr, unsigned int trace_ctx, int skip); ··· 822 815 { 823 816 } 824 817 #endif /* CONFIG_STACKTRACE */ 818 + 819 + #ifdef CONFIG_TRACER_MAX_TRACE 820 + static inline bool tracer_uses_snapshot(struct tracer *tracer) 821 + { 822 + return tracer->use_max_tr; 823 + } 824 + #else 825 + static inline bool tracer_uses_snapshot(struct tracer *tracer) 826 + { 827 + return false; 828 + } 829 + #endif 825 830 826 831 void trace_last_func_repeats(struct trace_array *tr, 827 832 struct trace_func_repeats *last_info, ··· 884 865 struct trace_array *tr); 885 866 extern int trace_selftest_startup_branch(struct tracer *trace, 886 867 struct trace_array *tr); 868 + extern bool __read_mostly tracing_selftest_running; 887 869 /* 888 870 * Tracer data references selftest functions that only occur 889 871 * on boot up. These can be __init functions. Thus, when selftests ··· 897 877 } 898 878 /* Tracers are seldom changed. Optimize when selftests are disabled. */ 899 879 #define __tracer_data __read_mostly 880 + #define tracing_selftest_running 0 900 881 #endif /* CONFIG_FTRACE_STARTUP_TEST */ 901 882 902 883 extern void *head_page(struct trace_array_cpu *data); ··· 1435 1414 C(COPY_MARKER, "copy_trace_marker"), \ 1436 1415 C(PAUSE_ON_TRACE, "pause-on-trace"), \ 1437 1416 C(HASH_PTR, "hash-ptr"), /* Print hashed pointer */ \ 1417 + C(BITMASK_LIST, "bitmask-list"), \ 1438 1418 FUNCTION_FLAGS \ 1439 1419 FGRAPH_FLAGS \ 1440 1420 STACK_FLAGS \ ··· 1588 1566 char *trace_user_fault_read(struct trace_user_buf_info *tinfo, 1589 1567 const char __user *ptr, size_t size, 1590 1568 trace_user_buf_copy copy_func, void *data); 1569 + 1570 + static __always_inline void 1571 + trace_event_setup(struct ring_buffer_event *event, 1572 + int type, unsigned int trace_ctx) 1573 + { 1574 + struct trace_entry *ent = ring_buffer_event_data(event); 1575 + 1576 + tracing_generic_entry_update(ent, type, trace_ctx); 1577 + } 1578 + 1579 + static __always_inline struct ring_buffer_event * 1580 + __trace_buffer_lock_reserve(struct trace_buffer *buffer, 1581 + int type, 1582 + unsigned long len, 1583 + unsigned int trace_ctx) 1584 + { 1585 + struct ring_buffer_event *event; 1586 + 1587 + event = ring_buffer_lock_reserve(buffer, len); 1588 + if (event != NULL) 1589 + trace_event_setup(event, type, trace_ctx); 1590 + 1591 + return event; 1592 + } 1593 + 1594 + static __always_inline void 1595 + __buffer_unlock_commit(struct trace_buffer *buffer, struct ring_buffer_event *event) 1596 + { 1597 + __this_cpu_write(trace_taskinfo_save, true); 1598 + 1599 + /* If this is the temp buffer, we need to commit fully */ 1600 + if (this_cpu_read(trace_buffered_event) == event) { 1601 + /* Length is in event->array[0] */ 1602 + ring_buffer_write(buffer, event->array[0], &event->array[1]); 1603 + /* Release the temp buffer */ 1604 + this_cpu_dec(trace_buffered_event_cnt); 1605 + /* ring_buffer_unlock_commit() enables preemption */ 1606 + preempt_enable_notrace(); 1607 + } else 1608 + ring_buffer_unlock_commit(buffer); 1609 + } 1591 1610 1592 1611 static inline void 1593 1612 __trace_event_discard_commit(struct trace_buffer *buffer, ··· 2150 2087 2151 2088 void trace_printk_control(bool enabled); 2152 2089 void trace_printk_start_comm(void); 2090 + void trace_printk_start_stop_comm(int enabled); 2153 2091 int trace_keep_overwrite(struct tracer *tracer, u64 mask, int set); 2154 2092 int set_tracer_flag(struct trace_array *tr, u64 mask, int enabled); 2155 2093 ··· 2300 2236 if (*name == ':' || *name == '.' || *name == '*') 2301 2237 *name = '_'; 2302 2238 } 2239 + 2240 + #ifdef CONFIG_STACKTRACE 2241 + void __ftrace_trace_stack(struct trace_array *tr, 2242 + struct trace_buffer *buffer, 2243 + unsigned int trace_ctx, 2244 + int skip, struct pt_regs *regs); 2245 + 2246 + static __always_inline void ftrace_trace_stack(struct trace_array *tr, 2247 + struct trace_buffer *buffer, 2248 + unsigned int trace_ctx, 2249 + int skip, struct pt_regs *regs) 2250 + { 2251 + if (!(tr->trace_flags & TRACE_ITER(STACKTRACE))) 2252 + return; 2253 + 2254 + __ftrace_trace_stack(tr, buffer, trace_ctx, skip, regs); 2255 + } 2256 + #else 2257 + static inline void __ftrace_trace_stack(struct trace_array *tr, 2258 + struct trace_buffer *buffer, 2259 + unsigned int trace_ctx, 2260 + int skip, struct pt_regs *regs) 2261 + { 2262 + } 2263 + static inline void ftrace_trace_stack(struct trace_array *tr, 2264 + struct trace_buffer *buffer, 2265 + unsigned long trace_ctx, 2266 + int skip, struct pt_regs *regs) 2267 + { 2268 + } 2269 + #endif 2303 2270 2304 2271 /* 2305 2272 * This is a generic way to read and write a u64 value from a file in tracefs.
+157 -6
kernel/trace/trace_events.c
··· 649 649 } 650 650 EXPORT_SYMBOL_GPL(trace_event_ignore_this_pid); 651 651 652 + /** 653 + * trace_event_buffer_reserve - reserve space on the ring buffer for an event 654 + * @fbuffer: information about how to save the event 655 + * @trace_file: the instance file descriptor for the event 656 + * @len: The length of the event 657 + * 658 + * The @fbuffer has information about the ring buffer and data will 659 + * be added to it to be used by the call to trace_event_buffer_commit(). 660 + * The @trace_file is the desrciptor with information about the status 661 + * of the given event for a specific trace_array instance. 662 + * The @len is the length of data to save for the event. 663 + * 664 + * Returns a pointer to the data on the ring buffer or NULL if the 665 + * event was not reserved (event was filtered, too big, or the buffer 666 + * simply was disabled for write). 667 + */ 652 668 void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer, 653 669 struct trace_event_file *trace_file, 654 670 unsigned long len) ··· 1678 1662 mutex_unlock(&event_mutex); 1679 1663 } 1680 1664 1665 + static int get_call_len(struct trace_event_call *call) 1666 + { 1667 + int len; 1668 + 1669 + /* Get the length of "<system>:<event>" */ 1670 + len = strlen(call->class->system) + 1; 1671 + len += strlen(trace_event_name(call)); 1672 + 1673 + /* Set the index to 32 bytes to separate event from data */ 1674 + return len >= 32 ? 1 : 32 - len; 1675 + } 1676 + 1677 + /** 1678 + * t_show_filters - seq_file callback to display active event filters 1679 + * @m: The seq_file interface for formatted output 1680 + * @v: The current trace_event_file being iterated 1681 + * 1682 + * Identifies and prints active filters for the current event file in the 1683 + * iteration. If a filter is applied to the current event and, if so, 1684 + * prints the system name, event name, and the filter string. 1685 + */ 1686 + static int t_show_filters(struct seq_file *m, void *v) 1687 + { 1688 + struct trace_event_file *file = v; 1689 + struct trace_event_call *call = file->event_call; 1690 + struct event_filter *filter; 1691 + int len; 1692 + 1693 + guard(rcu)(); 1694 + filter = rcu_dereference(file->filter); 1695 + if (!filter || !filter->filter_string) 1696 + return 0; 1697 + 1698 + len = get_call_len(call); 1699 + 1700 + seq_printf(m, "%s:%s%*.s%s\n", call->class->system, 1701 + trace_event_name(call), len, "", filter->filter_string); 1702 + 1703 + return 0; 1704 + } 1705 + 1706 + /** 1707 + * t_show_triggers - seq_file callback to display active event triggers 1708 + * @m: The seq_file interface for formatted output 1709 + * @v: The current trace_event_file being iterated 1710 + * 1711 + * Iterates through the trigger list of the current event file and prints 1712 + * each active trigger's configuration using its associated print 1713 + * operation. 1714 + */ 1715 + static int t_show_triggers(struct seq_file *m, void *v) 1716 + { 1717 + struct trace_event_file *file = v; 1718 + struct trace_event_call *call = file->event_call; 1719 + struct event_trigger_data *data; 1720 + int len; 1721 + 1722 + /* 1723 + * The event_mutex is held by t_start(), protecting the 1724 + * file->triggers list traversal. 1725 + */ 1726 + if (list_empty(&file->triggers)) 1727 + return 0; 1728 + 1729 + len = get_call_len(call); 1730 + 1731 + list_for_each_entry_rcu(data, &file->triggers, list) { 1732 + seq_printf(m, "%s:%s%*.s", call->class->system, 1733 + trace_event_name(call), len, ""); 1734 + 1735 + data->cmd_ops->print(m, data); 1736 + } 1737 + 1738 + return 0; 1739 + } 1740 + 1681 1741 #ifdef CONFIG_MODULES 1682 1742 static int s_show(struct seq_file *m, void *v) 1683 1743 { ··· 2268 2176 struct event_subsystem *system = NULL; 2269 2177 int ret; 2270 2178 2271 - if (tracing_is_disabled()) 2179 + if (unlikely(tracing_disabled)) 2272 2180 return -ENODEV; 2273 2181 2274 2182 /* Make sure the system still exists */ ··· 2581 2489 2582 2490 static int ftrace_event_avail_open(struct inode *inode, struct file *file); 2583 2491 static int ftrace_event_set_open(struct inode *inode, struct file *file); 2492 + static int ftrace_event_show_filters_open(struct inode *inode, struct file *file); 2493 + static int ftrace_event_show_triggers_open(struct inode *inode, struct file *file); 2584 2494 static int ftrace_event_set_pid_open(struct inode *inode, struct file *file); 2585 2495 static int ftrace_event_set_npid_open(struct inode *inode, struct file *file); 2586 2496 static int ftrace_event_release(struct inode *inode, struct file *file); ··· 2599 2505 .next = s_next, 2600 2506 .show = s_show, 2601 2507 .stop = s_stop, 2508 + }; 2509 + 2510 + static const struct seq_operations show_show_event_filters_seq_ops = { 2511 + .start = t_start, 2512 + .next = t_next, 2513 + .show = t_show_filters, 2514 + .stop = t_stop, 2515 + }; 2516 + 2517 + static const struct seq_operations show_show_event_triggers_seq_ops = { 2518 + .start = t_start, 2519 + .next = t_next, 2520 + .show = t_show_triggers, 2521 + .stop = t_stop, 2602 2522 }; 2603 2523 2604 2524 static const struct seq_operations show_set_pid_seq_ops = { ··· 2642 2534 .write = ftrace_event_write, 2643 2535 .llseek = seq_lseek, 2644 2536 .release = ftrace_event_release, 2537 + }; 2538 + 2539 + static const struct file_operations ftrace_show_event_filters_fops = { 2540 + .open = ftrace_event_show_filters_open, 2541 + .read = seq_read, 2542 + .llseek = seq_lseek, 2543 + .release = seq_release, 2544 + }; 2545 + 2546 + static const struct file_operations ftrace_show_event_triggers_fops = { 2547 + .open = ftrace_event_show_triggers_open, 2548 + .read = seq_read, 2549 + .llseek = seq_lseek, 2550 + .release = seq_release, 2645 2551 }; 2646 2552 2647 2553 static const struct file_operations ftrace_set_event_pid_fops = { ··· 2800 2678 if (ret < 0) 2801 2679 trace_array_put(tr); 2802 2680 return ret; 2681 + } 2682 + 2683 + /** 2684 + * ftrace_event_show_filters_open - open interface for set_event_filters 2685 + * @inode: The inode of the file 2686 + * @file: The file being opened 2687 + * 2688 + * Connects the set_event_filters file to the sequence operations 2689 + * required to iterate over and display active event filters. 2690 + */ 2691 + static int 2692 + ftrace_event_show_filters_open(struct inode *inode, struct file *file) 2693 + { 2694 + return ftrace_event_open(inode, file, &show_show_event_filters_seq_ops); 2695 + } 2696 + 2697 + /** 2698 + * ftrace_event_show_triggers_open - open interface for show_event_triggers 2699 + * @inode: The inode of the file 2700 + * @file: The file being opened 2701 + * 2702 + * Connects the show_event_triggers file to the sequence operations 2703 + * required to iterate over and display active event triggers. 2704 + */ 2705 + static int 2706 + ftrace_event_show_triggers_open(struct inode *inode, struct file *file) 2707 + { 2708 + return ftrace_event_open(inode, file, &show_show_event_triggers_seq_ops); 2803 2709 } 2804 2710 2805 2711 static int ··· 4113 3963 EXPORT_SYMBOL_GPL(trace_put_event_file); 4114 3964 4115 3965 #ifdef CONFIG_DYNAMIC_FTRACE 4116 - 4117 - /* Avoid typos */ 4118 - #define ENABLE_EVENT_STR "enable_event" 4119 - #define DISABLE_EVENT_STR "disable_event" 4120 - 4121 3966 struct event_probe_data { 4122 3967 struct trace_event_file *file; 4123 3968 unsigned long count; ··· 4544 4399 tr, &ftrace_set_event_fops); 4545 4400 if (!entry) 4546 4401 return -ENOMEM; 4402 + 4403 + trace_create_file("show_event_filters", TRACE_MODE_READ, parent, tr, 4404 + &ftrace_show_event_filters_fops); 4405 + 4406 + trace_create_file("show_event_triggers", TRACE_MODE_READ, parent, tr, 4407 + &ftrace_show_event_triggers_fops); 4547 4408 4548 4409 nr_entries = ARRAY_SIZE(events_entries); 4549 4410
+1 -1
kernel/trace/trace_events_filter.c
··· 1375 1375 struct filter_head *filter_list = container_of(rhp, struct filter_head, rcu); 1376 1376 1377 1377 INIT_RCU_WORK(&filter_list->rwork, free_filter_list_work); 1378 - queue_rcu_work(system_wq, &filter_list->rwork); 1378 + queue_rcu_work(system_dfl_wq, &filter_list->rwork); 1379 1379 } 1380 1380 1381 1381 /*
+66 -35
kernel/trace/trace_events_hist.c
··· 105 105 FIELD_OP_MULT, 106 106 }; 107 107 108 + #define FIELD_FUNCS \ 109 + C(NOP, "nop"), \ 110 + C(VAR_REF, "var_ref"), \ 111 + C(COUNTER, "counter"), \ 112 + C(CONST, "const"), \ 113 + C(LOG2, "log2"), \ 114 + C(BUCKET, "bucket"), \ 115 + C(TIMESTAMP, "timestamp"), \ 116 + C(CPU, "cpu"), \ 117 + C(COMM, "comm"), \ 118 + C(STRING, "string"), \ 119 + C(DYNSTRING, "dynstring"), \ 120 + C(RELDYNSTRING, "reldynstring"), \ 121 + C(PSTRING, "pstring"), \ 122 + C(S64, "s64"), \ 123 + C(U64, "u64"), \ 124 + C(S32, "s32"), \ 125 + C(U32, "u32"), \ 126 + C(S16, "s16"), \ 127 + C(U16, "u16"), \ 128 + C(S8, "s8"), \ 129 + C(U8, "u8"), \ 130 + C(UMINUS, "uminus"), \ 131 + C(MINUS, "minus"), \ 132 + C(PLUS, "plus"), \ 133 + C(DIV, "div"), \ 134 + C(MULT, "mult"), \ 135 + C(DIV_POWER2, "div_power2"), \ 136 + C(DIV_NOT_POWER2, "div_not_power2"), \ 137 + C(DIV_MULT_SHIFT, "div_mult_shift"), \ 138 + C(EXECNAME, "execname"), \ 139 + C(STACK, "stack"), 140 + 141 + #undef C 142 + #define C(a, b) HIST_FIELD_FN_##a 143 + 108 144 enum hist_field_fn { 109 - HIST_FIELD_FN_NOP, 110 - HIST_FIELD_FN_VAR_REF, 111 - HIST_FIELD_FN_COUNTER, 112 - HIST_FIELD_FN_CONST, 113 - HIST_FIELD_FN_LOG2, 114 - HIST_FIELD_FN_BUCKET, 115 - HIST_FIELD_FN_TIMESTAMP, 116 - HIST_FIELD_FN_CPU, 117 - HIST_FIELD_FN_COMM, 118 - HIST_FIELD_FN_STRING, 119 - HIST_FIELD_FN_DYNSTRING, 120 - HIST_FIELD_FN_RELDYNSTRING, 121 - HIST_FIELD_FN_PSTRING, 122 - HIST_FIELD_FN_S64, 123 - HIST_FIELD_FN_U64, 124 - HIST_FIELD_FN_S32, 125 - HIST_FIELD_FN_U32, 126 - HIST_FIELD_FN_S16, 127 - HIST_FIELD_FN_U16, 128 - HIST_FIELD_FN_S8, 129 - HIST_FIELD_FN_U8, 130 - HIST_FIELD_FN_UMINUS, 131 - HIST_FIELD_FN_MINUS, 132 - HIST_FIELD_FN_PLUS, 133 - HIST_FIELD_FN_DIV, 134 - HIST_FIELD_FN_MULT, 135 - HIST_FIELD_FN_DIV_POWER2, 136 - HIST_FIELD_FN_DIV_NOT_POWER2, 137 - HIST_FIELD_FN_DIV_MULT_SHIFT, 138 - HIST_FIELD_FN_EXECNAME, 139 - HIST_FIELD_FN_STACK, 145 + FIELD_FUNCS 140 146 }; 141 147 142 148 /* ··· 3163 3157 u64 var_val; 3164 3158 3165 3159 /* Make sure stacktrace can fit in the string variable length */ 3166 - BUILD_BUG_ON((HIST_STACKTRACE_DEPTH + 1) * sizeof(long) >= STR_VAR_LEN_MAX); 3160 + BUILD_BUG_ON((HIST_STACKTRACE_DEPTH + 1) * sizeof(long) > STR_VAR_LEN_MAX); 3167 3161 3168 3162 for (i = 0, j = field_var_str_start; i < n_field_vars; i++) { 3169 3163 struct field_var *field_var = field_vars[i]; ··· 5860 5854 }; 5861 5855 5862 5856 #ifdef CONFIG_HIST_TRIGGERS_DEBUG 5857 + 5858 + #undef C 5859 + #define C(a, b) b 5860 + 5861 + static const char * const field_funcs[] = { FIELD_FUNCS }; 5862 + 5863 5863 static void hist_field_debug_show_flags(struct seq_file *m, 5864 5864 unsigned long flags) 5865 5865 { ··· 5930 5918 seq_printf(m, " type: %s\n", field->type); 5931 5919 seq_printf(m, " size: %u\n", field->size); 5932 5920 seq_printf(m, " is_signed: %u\n", field->is_signed); 5921 + seq_printf(m, " function: hist_field_%s()\n", field_funcs[field->fn_num]); 5933 5922 5934 5923 return 0; 5935 5924 } ··· 6531 6518 return updated; 6532 6519 } 6533 6520 6521 + /* 6522 + * Set or disable using the per CPU trace_buffer_event when possible. 6523 + */ 6524 + static int tracing_set_filter_buffering(struct trace_array *tr, bool set) 6525 + { 6526 + guard(mutex)(&trace_types_lock); 6527 + 6528 + if (set && tr->no_filter_buffering_ref++) 6529 + return 0; 6530 + 6531 + if (!set) { 6532 + if (WARN_ON_ONCE(!tr->no_filter_buffering_ref)) 6533 + return -EINVAL; 6534 + 6535 + --tr->no_filter_buffering_ref; 6536 + } 6537 + 6538 + return 0; 6539 + } 6540 + 6534 6541 static int hist_register_trigger(char *glob, 6535 6542 struct event_trigger_data *data, 6536 6543 struct trace_event_file *file) ··· 6940 6907 out_unreg: 6941 6908 event_trigger_unregister(cmd_ops, file, glob+1, trigger_data); 6942 6909 out_free: 6943 - event_trigger_reset_filter(cmd_ops, trigger_data); 6944 - 6945 6910 remove_hist_vars(hist_data); 6946 6911 6947 - kfree(trigger_data); 6912 + trigger_data_free(trigger_data); 6948 6913 6949 6914 destroy_hist_data(hist_data); 6950 6915 goto out;
+3 -3
kernel/trace/trace_events_synth.c
··· 499 499 return len; 500 500 } 501 501 502 - static notrace void trace_event_raw_event_synth(void *__data, 503 - u64 *var_ref_vals, 504 - unsigned int *var_ref_idx) 502 + static void trace_event_raw_event_synth(void *__data, 503 + u64 *var_ref_vals, 504 + unsigned int *var_ref_idx) 505 505 { 506 506 unsigned int i, n_u64, val_idx, len, data_size = 0; 507 507 struct trace_event_file *trace_file = __data;
+28 -42
kernel/trace/trace_events_trigger.c
··· 1347 1347 { 1348 1348 struct trace_event_file *file = data->private_data; 1349 1349 1350 - if (file) { 1351 - if (tracer_tracing_is_on(file->tr)) 1352 - return; 1353 - 1354 - tracer_tracing_on(file->tr); 1355 - return; 1356 - } 1357 - 1358 - if (tracing_is_on()) 1350 + if (WARN_ON_ONCE(!file)) 1359 1351 return; 1360 1352 1361 - tracing_on(); 1353 + if (tracer_tracing_is_on(file->tr)) 1354 + return; 1355 + 1356 + tracer_tracing_on(file->tr); 1362 1357 } 1363 1358 1364 1359 static bool ··· 1363 1368 { 1364 1369 struct trace_event_file *file = data->private_data; 1365 1370 1366 - if (file) { 1367 - if (tracer_tracing_is_on(file->tr)) 1368 - return false; 1369 - } else { 1370 - if (tracing_is_on()) 1371 - return false; 1372 - } 1371 + if (WARN_ON_ONCE(!file)) 1372 + return false; 1373 + 1374 + if (tracer_tracing_is_on(file->tr)) 1375 + return false; 1373 1376 1374 1377 if (!data->count) 1375 1378 return false; ··· 1385 1392 { 1386 1393 struct trace_event_file *file = data->private_data; 1387 1394 1388 - if (file) { 1389 - if (!tracer_tracing_is_on(file->tr)) 1390 - return; 1391 - 1392 - tracer_tracing_off(file->tr); 1393 - return; 1394 - } 1395 - 1396 - if (!tracing_is_on()) 1395 + if (WARN_ON_ONCE(!file)) 1397 1396 return; 1398 1397 1399 - tracing_off(); 1398 + if (!tracer_tracing_is_on(file->tr)) 1399 + return; 1400 + 1401 + tracer_tracing_off(file->tr); 1400 1402 } 1401 1403 1402 1404 static bool ··· 1401 1413 { 1402 1414 struct trace_event_file *file = data->private_data; 1403 1415 1404 - if (file) { 1405 - if (!tracer_tracing_is_on(file->tr)) 1406 - return false; 1407 - } else { 1408 - if (!tracing_is_on()) 1409 - return false; 1410 - } 1416 + if (WARN_ON_ONCE(!file)) 1417 + return false; 1418 + 1419 + if (!tracer_tracing_is_on(file->tr)) 1420 + return false; 1411 1421 1412 1422 if (!data->count) 1413 1423 return false; ··· 1467 1481 { 1468 1482 struct trace_event_file *file = data->private_data; 1469 1483 1470 - if (file) 1471 - tracing_snapshot_instance(file->tr); 1472 - else 1473 - tracing_snapshot(); 1484 + if (WARN_ON_ONCE(!file)) 1485 + return; 1486 + 1487 + tracing_snapshot_instance(file->tr); 1474 1488 } 1475 1489 1476 1490 static int ··· 1556 1570 { 1557 1571 struct trace_event_file *file = data->private_data; 1558 1572 1559 - if (file) 1560 - __trace_stack(file->tr, tracing_gen_ctx_dec(), STACK_SKIP); 1561 - else 1562 - trace_dump_stack(STACK_SKIP); 1573 + if (WARN_ON_ONCE(!file)) 1574 + return; 1575 + 1576 + __trace_stack(file->tr, tracing_gen_ctx_dec(), STACK_SKIP); 1563 1577 } 1564 1578 1565 1579 static int
+7 -8
kernel/trace/trace_hwlat.c
··· 102 102 /* keep the global state somewhere. */ 103 103 static struct hwlat_data { 104 104 105 - struct mutex lock; /* protect changes */ 105 + struct mutex lock; /* protect changes */ 106 106 107 - u64 count; /* total since reset */ 107 + atomic64_t count; /* total since reset */ 108 108 109 109 u64 sample_window; /* total sampling window (on+off) */ 110 110 u64 sample_width; /* active sampling portion of window */ ··· 193 193 * get_sample - sample the CPU TSC and look for likely hardware latencies 194 194 * 195 195 * Used to repeatedly capture the CPU TSC (or similar), looking for potential 196 - * hardware-induced latency. Called with interrupts disabled and with 197 - * hwlat_data.lock held. 196 + * hardware-induced latency. Called with interrupts disabled. 198 197 */ 199 198 static int get_sample(void) 200 199 { ··· 203 204 time_type start, t1, t2, last_t2; 204 205 s64 diff, outer_diff, total, last_total = 0; 205 206 u64 sample = 0; 207 + u64 sample_width = READ_ONCE(hwlat_data.sample_width); 206 208 u64 thresh = tracing_thresh; 207 209 u64 outer_sample = 0; 208 210 int ret = -1; ··· 267 267 if (diff > sample) 268 268 sample = diff; /* only want highest value */ 269 269 270 - } while (total <= hwlat_data.sample_width); 270 + } while (total <= sample_width); 271 271 272 272 barrier(); /* finish the above in the view for NMIs */ 273 273 trace_hwlat_callback_enabled = false; ··· 285 285 if (kdata->nmi_total_ts) 286 286 do_div(kdata->nmi_total_ts, NSEC_PER_USEC); 287 287 288 - hwlat_data.count++; 289 - s.seqnum = hwlat_data.count; 288 + s.seqnum = atomic64_inc_return(&hwlat_data.count); 290 289 s.duration = sample; 291 290 s.outer_duration = outer_sample; 292 291 s.nmi_total_ts = kdata->nmi_total_ts; ··· 831 832 832 833 hwlat_trace = tr; 833 834 834 - hwlat_data.count = 0; 835 + atomic64_set(&hwlat_data.count, 0); 835 836 tr->max_latency = 0; 836 837 save_tracing_thresh = tracing_thresh; 837 838
+5 -1
kernel/trace/trace_kprobe.c
··· 2048 2048 trace_create_file("kprobe_profile", TRACE_MODE_READ, 2049 2049 NULL, NULL, &kprobe_profile_ops); 2050 2050 2051 + /* If no 'kprobe_event=' cmd is provided, return directly. */ 2052 + if (kprobe_boot_events_buf[0] == '\0') 2053 + return 0; 2054 + 2051 2055 setup_boot_kprobe_events(); 2052 2056 2053 2057 return 0; ··· 2083 2079 struct trace_kprobe *tk; 2084 2080 struct trace_event_file *file; 2085 2081 2086 - if (tracing_is_disabled()) 2082 + if (unlikely(tracing_disabled)) 2087 2083 return -ENODEV; 2088 2084 2089 2085 if (tracing_selftest_disabled)
+27 -3
kernel/trace/trace_output.c
··· 194 194 EXPORT_SYMBOL(trace_print_symbols_seq_u64); 195 195 #endif 196 196 197 + /** 198 + * trace_print_bitmask_seq - print a bitmask to a sequence buffer 199 + * @iter: The trace iterator for the current event instance 200 + * @bitmask_ptr: The pointer to the bitmask data 201 + * @bitmask_size: The size of the bitmask in bytes 202 + * 203 + * Prints a bitmask into a sequence buffer as either a hex string or a 204 + * human-readable range list, depending on the instance's "bitmask-list" 205 + * trace option. The bitmask is formatted into the iterator's temporary 206 + * scratchpad rather than the primary sequence buffer. This avoids 207 + * duplication and pointer-collision issues when the returned string is 208 + * processed by a "%s" specifier in a TP_printk() macro. 209 + * 210 + * Returns a pointer to the formatted string within the temporary buffer. 211 + */ 197 212 const char * 198 - trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr, 213 + trace_print_bitmask_seq(struct trace_iterator *iter, void *bitmask_ptr, 199 214 unsigned int bitmask_size) 200 215 { 201 - const char *ret = trace_seq_buffer_ptr(p); 216 + struct trace_seq *p = &iter->tmp_seq; 217 + const struct trace_array *tr = iter->tr; 218 + const char *ret; 202 219 203 - trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8); 220 + trace_seq_init(p); 221 + ret = trace_seq_buffer_ptr(p); 222 + 223 + if (tr->trace_flags & TRACE_ITER(BITMASK_LIST)) 224 + trace_seq_bitmask_list(p, bitmask_ptr, bitmask_size * 8); 225 + else 226 + trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8); 227 + 204 228 trace_seq_putc(p, 0); 205 229 206 230 return ret;
+246
kernel/trace/trace_pid.c
··· 1 + // SPDX-License-Identifier: GPL-2.0 2 + 3 + #include "trace.h" 4 + 5 + /** 6 + * trace_find_filtered_pid - check if a pid exists in a filtered_pid list 7 + * @filtered_pids: The list of pids to check 8 + * @search_pid: The PID to find in @filtered_pids 9 + * 10 + * Returns true if @search_pid is found in @filtered_pids, and false otherwise. 11 + */ 12 + bool 13 + trace_find_filtered_pid(struct trace_pid_list *filtered_pids, pid_t search_pid) 14 + { 15 + return trace_pid_list_is_set(filtered_pids, search_pid); 16 + } 17 + 18 + /** 19 + * trace_ignore_this_task - should a task be ignored for tracing 20 + * @filtered_pids: The list of pids to check 21 + * @filtered_no_pids: The list of pids not to be traced 22 + * @task: The task that should be ignored if not filtered 23 + * 24 + * Checks if @task should be traced or not from @filtered_pids. 25 + * Returns true if @task should *NOT* be traced. 26 + * Returns false if @task should be traced. 27 + */ 28 + bool 29 + trace_ignore_this_task(struct trace_pid_list *filtered_pids, 30 + struct trace_pid_list *filtered_no_pids, 31 + struct task_struct *task) 32 + { 33 + /* 34 + * If filtered_no_pids is not empty, and the task's pid is listed 35 + * in filtered_no_pids, then return true. 36 + * Otherwise, if filtered_pids is empty, that means we can 37 + * trace all tasks. If it has content, then only trace pids 38 + * within filtered_pids. 39 + */ 40 + 41 + return (filtered_pids && 42 + !trace_find_filtered_pid(filtered_pids, task->pid)) || 43 + (filtered_no_pids && 44 + trace_find_filtered_pid(filtered_no_pids, task->pid)); 45 + } 46 + 47 + /** 48 + * trace_filter_add_remove_task - Add or remove a task from a pid_list 49 + * @pid_list: The list to modify 50 + * @self: The current task for fork or NULL for exit 51 + * @task: The task to add or remove 52 + * 53 + * If adding a task, if @self is defined, the task is only added if @self 54 + * is also included in @pid_list. This happens on fork and tasks should 55 + * only be added when the parent is listed. If @self is NULL, then the 56 + * @task pid will be removed from the list, which would happen on exit 57 + * of a task. 58 + */ 59 + void trace_filter_add_remove_task(struct trace_pid_list *pid_list, 60 + struct task_struct *self, 61 + struct task_struct *task) 62 + { 63 + if (!pid_list) 64 + return; 65 + 66 + /* For forks, we only add if the forking task is listed */ 67 + if (self) { 68 + if (!trace_find_filtered_pid(pid_list, self->pid)) 69 + return; 70 + } 71 + 72 + /* "self" is set for forks, and NULL for exits */ 73 + if (self) 74 + trace_pid_list_set(pid_list, task->pid); 75 + else 76 + trace_pid_list_clear(pid_list, task->pid); 77 + } 78 + 79 + /** 80 + * trace_pid_next - Used for seq_file to get to the next pid of a pid_list 81 + * @pid_list: The pid list to show 82 + * @v: The last pid that was shown (+1 the actual pid to let zero be displayed) 83 + * @pos: The position of the file 84 + * 85 + * This is used by the seq_file "next" operation to iterate the pids 86 + * listed in a trace_pid_list structure. 87 + * 88 + * Returns the pid+1 as we want to display pid of zero, but NULL would 89 + * stop the iteration. 90 + */ 91 + void *trace_pid_next(struct trace_pid_list *pid_list, void *v, loff_t *pos) 92 + { 93 + long pid = (unsigned long)v; 94 + unsigned int next; 95 + 96 + (*pos)++; 97 + 98 + /* pid already is +1 of the actual previous bit */ 99 + if (trace_pid_list_next(pid_list, pid, &next) < 0) 100 + return NULL; 101 + 102 + pid = next; 103 + 104 + /* Return pid + 1 to allow zero to be represented */ 105 + return (void *)(pid + 1); 106 + } 107 + 108 + /** 109 + * trace_pid_start - Used for seq_file to start reading pid lists 110 + * @pid_list: The pid list to show 111 + * @pos: The position of the file 112 + * 113 + * This is used by seq_file "start" operation to start the iteration 114 + * of listing pids. 115 + * 116 + * Returns the pid+1 as we want to display pid of zero, but NULL would 117 + * stop the iteration. 118 + */ 119 + void *trace_pid_start(struct trace_pid_list *pid_list, loff_t *pos) 120 + { 121 + unsigned long pid; 122 + unsigned int first; 123 + loff_t l = 0; 124 + 125 + if (trace_pid_list_first(pid_list, &first) < 0) 126 + return NULL; 127 + 128 + pid = first; 129 + 130 + /* Return pid + 1 so that zero can be the exit value */ 131 + for (pid++; pid && l < *pos; 132 + pid = (unsigned long)trace_pid_next(pid_list, (void *)pid, &l)) 133 + ; 134 + return (void *)pid; 135 + } 136 + 137 + /** 138 + * trace_pid_show - show the current pid in seq_file processing 139 + * @m: The seq_file structure to write into 140 + * @v: A void pointer of the pid (+1) value to display 141 + * 142 + * Can be directly used by seq_file operations to display the current 143 + * pid value. 144 + */ 145 + int trace_pid_show(struct seq_file *m, void *v) 146 + { 147 + unsigned long pid = (unsigned long)v - 1; 148 + 149 + seq_printf(m, "%lu\n", pid); 150 + return 0; 151 + } 152 + 153 + /* 128 should be much more than enough */ 154 + #define PID_BUF_SIZE 127 155 + 156 + int trace_pid_write(struct trace_pid_list *filtered_pids, 157 + struct trace_pid_list **new_pid_list, 158 + const char __user *ubuf, size_t cnt) 159 + { 160 + struct trace_pid_list *pid_list; 161 + struct trace_parser parser; 162 + unsigned long val; 163 + int nr_pids = 0; 164 + ssize_t read = 0; 165 + ssize_t ret; 166 + loff_t pos; 167 + pid_t pid; 168 + 169 + if (trace_parser_get_init(&parser, PID_BUF_SIZE + 1)) 170 + return -ENOMEM; 171 + 172 + /* 173 + * Always recreate a new array. The write is an all or nothing 174 + * operation. Always create a new array when adding new pids by 175 + * the user. If the operation fails, then the current list is 176 + * not modified. 177 + */ 178 + pid_list = trace_pid_list_alloc(); 179 + if (!pid_list) { 180 + trace_parser_put(&parser); 181 + return -ENOMEM; 182 + } 183 + 184 + if (filtered_pids) { 185 + /* copy the current bits to the new max */ 186 + ret = trace_pid_list_first(filtered_pids, &pid); 187 + while (!ret) { 188 + ret = trace_pid_list_set(pid_list, pid); 189 + if (ret < 0) 190 + goto out; 191 + 192 + ret = trace_pid_list_next(filtered_pids, pid + 1, &pid); 193 + nr_pids++; 194 + } 195 + } 196 + 197 + ret = 0; 198 + while (cnt > 0) { 199 + 200 + pos = 0; 201 + 202 + ret = trace_get_user(&parser, ubuf, cnt, &pos); 203 + if (ret < 0) 204 + break; 205 + 206 + read += ret; 207 + ubuf += ret; 208 + cnt -= ret; 209 + 210 + if (!trace_parser_loaded(&parser)) 211 + break; 212 + 213 + ret = -EINVAL; 214 + if (kstrtoul(parser.buffer, 0, &val)) 215 + break; 216 + 217 + pid = (pid_t)val; 218 + 219 + if (trace_pid_list_set(pid_list, pid) < 0) { 220 + ret = -1; 221 + break; 222 + } 223 + nr_pids++; 224 + 225 + trace_parser_clear(&parser); 226 + ret = 0; 227 + } 228 + out: 229 + trace_parser_put(&parser); 230 + 231 + if (ret < 0) { 232 + trace_pid_list_free(pid_list); 233 + return ret; 234 + } 235 + 236 + if (!nr_pids) { 237 + /* Cleared the list of pids */ 238 + trace_pid_list_free(pid_list); 239 + pid_list = NULL; 240 + } 241 + 242 + *new_pid_list = pid_list; 243 + 244 + return read; 245 + } 246 +
+430
kernel/trace/trace_printk.c
··· 376 376 .release = seq_release, 377 377 }; 378 378 379 + static __always_inline bool printk_binsafe(struct trace_array *tr) 380 + { 381 + /* 382 + * The binary format of traceprintk can cause a crash if used 383 + * by a buffer from another boot. Force the use of the 384 + * non binary version of trace_printk if the trace_printk 385 + * buffer is a boot mapped ring buffer. 386 + */ 387 + return !(tr->flags & TRACE_ARRAY_FL_BOOT); 388 + } 389 + 390 + int __trace_array_puts(struct trace_array *tr, unsigned long ip, 391 + const char *str, int size) 392 + { 393 + struct ring_buffer_event *event; 394 + struct trace_buffer *buffer; 395 + struct print_entry *entry; 396 + unsigned int trace_ctx; 397 + int alloc; 398 + 399 + if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 400 + return 0; 401 + 402 + if (unlikely(tracing_selftest_running && 403 + (tr->flags & TRACE_ARRAY_FL_GLOBAL))) 404 + return 0; 405 + 406 + if (unlikely(tracing_disabled)) 407 + return 0; 408 + 409 + alloc = sizeof(*entry) + size + 2; /* possible \n added */ 410 + 411 + trace_ctx = tracing_gen_ctx(); 412 + buffer = tr->array_buffer.buffer; 413 + guard(ring_buffer_nest)(buffer); 414 + event = __trace_buffer_lock_reserve(buffer, TRACE_PRINT, alloc, 415 + trace_ctx); 416 + if (!event) 417 + return 0; 418 + 419 + entry = ring_buffer_event_data(event); 420 + entry->ip = ip; 421 + 422 + memcpy(&entry->buf, str, size); 423 + 424 + /* Add a newline if necessary */ 425 + if (entry->buf[size - 1] != '\n') { 426 + entry->buf[size] = '\n'; 427 + entry->buf[size + 1] = '\0'; 428 + } else 429 + entry->buf[size] = '\0'; 430 + 431 + __buffer_unlock_commit(buffer, event); 432 + ftrace_trace_stack(tr, buffer, trace_ctx, 4, NULL); 433 + return size; 434 + } 435 + EXPORT_SYMBOL_GPL(__trace_array_puts); 436 + 437 + /** 438 + * __trace_puts - write a constant string into the trace buffer. 439 + * @ip: The address of the caller 440 + * @str: The constant string to write 441 + */ 442 + int __trace_puts(unsigned long ip, const char *str) 443 + { 444 + return __trace_array_puts(printk_trace, ip, str, strlen(str)); 445 + } 446 + EXPORT_SYMBOL_GPL(__trace_puts); 447 + 448 + /** 449 + * __trace_bputs - write the pointer to a constant string into trace buffer 450 + * @ip: The address of the caller 451 + * @str: The constant string to write to the buffer to 452 + */ 453 + int __trace_bputs(unsigned long ip, const char *str) 454 + { 455 + struct trace_array *tr = READ_ONCE(printk_trace); 456 + struct ring_buffer_event *event; 457 + struct trace_buffer *buffer; 458 + struct bputs_entry *entry; 459 + unsigned int trace_ctx; 460 + int size = sizeof(struct bputs_entry); 461 + 462 + if (!printk_binsafe(tr)) 463 + return __trace_puts(ip, str); 464 + 465 + if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 466 + return 0; 467 + 468 + if (unlikely(tracing_selftest_running || tracing_disabled)) 469 + return 0; 470 + 471 + trace_ctx = tracing_gen_ctx(); 472 + buffer = tr->array_buffer.buffer; 473 + 474 + guard(ring_buffer_nest)(buffer); 475 + event = __trace_buffer_lock_reserve(buffer, TRACE_BPUTS, size, 476 + trace_ctx); 477 + if (!event) 478 + return 0; 479 + 480 + entry = ring_buffer_event_data(event); 481 + entry->ip = ip; 482 + entry->str = str; 483 + 484 + __buffer_unlock_commit(buffer, event); 485 + ftrace_trace_stack(tr, buffer, trace_ctx, 4, NULL); 486 + 487 + return 1; 488 + } 489 + EXPORT_SYMBOL_GPL(__trace_bputs); 490 + 491 + /* created for use with alloc_percpu */ 492 + struct trace_buffer_struct { 493 + int nesting; 494 + char buffer[4][TRACE_BUF_SIZE]; 495 + }; 496 + 497 + static struct trace_buffer_struct __percpu *trace_percpu_buffer; 498 + 499 + /* 500 + * This allows for lockless recording. If we're nested too deeply, then 501 + * this returns NULL. 502 + */ 503 + static char *get_trace_buf(void) 504 + { 505 + struct trace_buffer_struct *buffer = this_cpu_ptr(trace_percpu_buffer); 506 + 507 + if (!trace_percpu_buffer || buffer->nesting >= 4) 508 + return NULL; 509 + 510 + buffer->nesting++; 511 + 512 + /* Interrupts must see nesting incremented before we use the buffer */ 513 + barrier(); 514 + return &buffer->buffer[buffer->nesting - 1][0]; 515 + } 516 + 517 + static void put_trace_buf(void) 518 + { 519 + /* Don't let the decrement of nesting leak before this */ 520 + barrier(); 521 + this_cpu_dec(trace_percpu_buffer->nesting); 522 + } 523 + 524 + static int alloc_percpu_trace_buffer(void) 525 + { 526 + struct trace_buffer_struct __percpu *buffers; 527 + 528 + if (trace_percpu_buffer) 529 + return 0; 530 + 531 + buffers = alloc_percpu(struct trace_buffer_struct); 532 + if (MEM_FAIL(!buffers, "Could not allocate percpu trace_printk buffer")) 533 + return -ENOMEM; 534 + 535 + trace_percpu_buffer = buffers; 536 + return 0; 537 + } 538 + 539 + static int buffers_allocated; 540 + 541 + void trace_printk_init_buffers(void) 542 + { 543 + if (buffers_allocated) 544 + return; 545 + 546 + if (alloc_percpu_trace_buffer()) 547 + return; 548 + 549 + /* trace_printk() is for debug use only. Don't use it in production. */ 550 + 551 + pr_warn("\n"); 552 + pr_warn("**********************************************************\n"); 553 + pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); 554 + pr_warn("** **\n"); 555 + pr_warn("** trace_printk() being used. Allocating extra memory. **\n"); 556 + pr_warn("** **\n"); 557 + pr_warn("** This means that this is a DEBUG kernel and it is **\n"); 558 + pr_warn("** unsafe for production use. **\n"); 559 + pr_warn("** **\n"); 560 + pr_warn("** If you see this message and you are not debugging **\n"); 561 + pr_warn("** the kernel, report this immediately to your vendor! **\n"); 562 + pr_warn("** **\n"); 563 + pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); 564 + pr_warn("**********************************************************\n"); 565 + 566 + /* Expand the buffers to set size */ 567 + if (tracing_update_buffers(NULL) < 0) 568 + pr_err("Failed to expand tracing buffers for trace_printk() calls\n"); 569 + else 570 + buffers_allocated = 1; 571 + 572 + /* 573 + * trace_printk_init_buffers() can be called by modules. 574 + * If that happens, then we need to start cmdline recording 575 + * directly here. 576 + */ 577 + if (system_state == SYSTEM_RUNNING) 578 + tracing_start_cmdline_record(); 579 + } 580 + EXPORT_SYMBOL_GPL(trace_printk_init_buffers); 581 + 582 + void trace_printk_start_comm(void) 583 + { 584 + /* Start tracing comms if trace printk is set */ 585 + if (!buffers_allocated) 586 + return; 587 + tracing_start_cmdline_record(); 588 + } 589 + 590 + void trace_printk_start_stop_comm(int enabled) 591 + { 592 + if (!buffers_allocated) 593 + return; 594 + 595 + if (enabled) 596 + tracing_start_cmdline_record(); 597 + else 598 + tracing_stop_cmdline_record(); 599 + } 600 + 601 + /** 602 + * trace_vbprintk - write binary msg to tracing buffer 603 + * @ip: The address of the caller 604 + * @fmt: The string format to write to the buffer 605 + * @args: Arguments for @fmt 606 + */ 607 + int trace_vbprintk(unsigned long ip, const char *fmt, va_list args) 608 + { 609 + struct ring_buffer_event *event; 610 + struct trace_buffer *buffer; 611 + struct trace_array *tr = READ_ONCE(printk_trace); 612 + struct bprint_entry *entry; 613 + unsigned int trace_ctx; 614 + char *tbuffer; 615 + int len = 0, size; 616 + 617 + if (!printk_binsafe(tr)) 618 + return trace_vprintk(ip, fmt, args); 619 + 620 + if (unlikely(tracing_selftest_running || tracing_disabled)) 621 + return 0; 622 + 623 + /* Don't pollute graph traces with trace_vprintk internals */ 624 + pause_graph_tracing(); 625 + 626 + trace_ctx = tracing_gen_ctx(); 627 + guard(preempt_notrace)(); 628 + 629 + tbuffer = get_trace_buf(); 630 + if (!tbuffer) { 631 + len = 0; 632 + goto out_nobuffer; 633 + } 634 + 635 + len = vbin_printf((u32 *)tbuffer, TRACE_BUF_SIZE/sizeof(int), fmt, args); 636 + 637 + if (len > TRACE_BUF_SIZE/sizeof(int) || len < 0) 638 + goto out_put; 639 + 640 + size = sizeof(*entry) + sizeof(u32) * len; 641 + buffer = tr->array_buffer.buffer; 642 + scoped_guard(ring_buffer_nest, buffer) { 643 + event = __trace_buffer_lock_reserve(buffer, TRACE_BPRINT, size, 644 + trace_ctx); 645 + if (!event) 646 + goto out_put; 647 + entry = ring_buffer_event_data(event); 648 + entry->ip = ip; 649 + entry->fmt = fmt; 650 + 651 + memcpy(entry->buf, tbuffer, sizeof(u32) * len); 652 + __buffer_unlock_commit(buffer, event); 653 + ftrace_trace_stack(tr, buffer, trace_ctx, 6, NULL); 654 + } 655 + out_put: 656 + put_trace_buf(); 657 + 658 + out_nobuffer: 659 + unpause_graph_tracing(); 660 + 661 + return len; 662 + } 663 + EXPORT_SYMBOL_GPL(trace_vbprintk); 664 + 665 + static __printf(3, 0) 666 + int __trace_array_vprintk(struct trace_buffer *buffer, 667 + unsigned long ip, const char *fmt, va_list args) 668 + { 669 + struct ring_buffer_event *event; 670 + int len = 0, size; 671 + struct print_entry *entry; 672 + unsigned int trace_ctx; 673 + char *tbuffer; 674 + 675 + if (unlikely(tracing_disabled)) 676 + return 0; 677 + 678 + /* Don't pollute graph traces with trace_vprintk internals */ 679 + pause_graph_tracing(); 680 + 681 + trace_ctx = tracing_gen_ctx(); 682 + guard(preempt_notrace)(); 683 + 684 + 685 + tbuffer = get_trace_buf(); 686 + if (!tbuffer) { 687 + len = 0; 688 + goto out_nobuffer; 689 + } 690 + 691 + len = vscnprintf(tbuffer, TRACE_BUF_SIZE, fmt, args); 692 + 693 + size = sizeof(*entry) + len + 1; 694 + scoped_guard(ring_buffer_nest, buffer) { 695 + event = __trace_buffer_lock_reserve(buffer, TRACE_PRINT, size, 696 + trace_ctx); 697 + if (!event) 698 + goto out; 699 + entry = ring_buffer_event_data(event); 700 + entry->ip = ip; 701 + 702 + memcpy(&entry->buf, tbuffer, len + 1); 703 + __buffer_unlock_commit(buffer, event); 704 + ftrace_trace_stack(printk_trace, buffer, trace_ctx, 6, NULL); 705 + } 706 + out: 707 + put_trace_buf(); 708 + 709 + out_nobuffer: 710 + unpause_graph_tracing(); 711 + 712 + return len; 713 + } 714 + 715 + int trace_array_vprintk(struct trace_array *tr, 716 + unsigned long ip, const char *fmt, va_list args) 717 + { 718 + if (tracing_selftest_running && (tr->flags & TRACE_ARRAY_FL_GLOBAL)) 719 + return 0; 720 + 721 + return __trace_array_vprintk(tr->array_buffer.buffer, ip, fmt, args); 722 + } 723 + 724 + /** 725 + * trace_array_printk - Print a message to a specific instance 726 + * @tr: The instance trace_array descriptor 727 + * @ip: The instruction pointer that this is called from. 728 + * @fmt: The format to print (printf format) 729 + * 730 + * If a subsystem sets up its own instance, they have the right to 731 + * printk strings into their tracing instance buffer using this 732 + * function. Note, this function will not write into the top level 733 + * buffer (use trace_printk() for that), as writing into the top level 734 + * buffer should only have events that can be individually disabled. 735 + * trace_printk() is only used for debugging a kernel, and should not 736 + * be ever incorporated in normal use. 737 + * 738 + * trace_array_printk() can be used, as it will not add noise to the 739 + * top level tracing buffer. 740 + * 741 + * Note, trace_array_init_printk() must be called on @tr before this 742 + * can be used. 743 + */ 744 + int trace_array_printk(struct trace_array *tr, 745 + unsigned long ip, const char *fmt, ...) 746 + { 747 + int ret; 748 + va_list ap; 749 + 750 + if (!tr) 751 + return -ENOENT; 752 + 753 + /* This is only allowed for created instances */ 754 + if (tr->flags & TRACE_ARRAY_FL_GLOBAL) 755 + return 0; 756 + 757 + if (!(tr->trace_flags & TRACE_ITER(PRINTK))) 758 + return 0; 759 + 760 + va_start(ap, fmt); 761 + ret = trace_array_vprintk(tr, ip, fmt, ap); 762 + va_end(ap); 763 + return ret; 764 + } 765 + EXPORT_SYMBOL_GPL(trace_array_printk); 766 + 767 + /** 768 + * trace_array_init_printk - Initialize buffers for trace_array_printk() 769 + * @tr: The trace array to initialize the buffers for 770 + * 771 + * As trace_array_printk() only writes into instances, they are OK to 772 + * have in the kernel (unlike trace_printk()). This needs to be called 773 + * before trace_array_printk() can be used on a trace_array. 774 + */ 775 + int trace_array_init_printk(struct trace_array *tr) 776 + { 777 + if (!tr) 778 + return -ENOENT; 779 + 780 + /* This is only allowed for created instances */ 781 + if (tr->flags & TRACE_ARRAY_FL_GLOBAL) 782 + return -EINVAL; 783 + 784 + return alloc_percpu_trace_buffer(); 785 + } 786 + EXPORT_SYMBOL_GPL(trace_array_init_printk); 787 + 788 + int trace_array_printk_buf(struct trace_buffer *buffer, 789 + unsigned long ip, const char *fmt, ...) 790 + { 791 + int ret; 792 + va_list ap; 793 + 794 + if (!(printk_trace->trace_flags & TRACE_ITER(PRINTK))) 795 + return 0; 796 + 797 + va_start(ap, fmt); 798 + ret = __trace_array_vprintk(buffer, ip, fmt, ap); 799 + va_end(ap); 800 + return ret; 801 + } 802 + 803 + int trace_vprintk(unsigned long ip, const char *fmt, va_list args) 804 + { 805 + return trace_array_vprintk(printk_trace, ip, fmt, args); 806 + } 807 + EXPORT_SYMBOL_GPL(trace_vprintk); 808 + 379 809 static __init int init_trace_printk_function_export(void) 380 810 { 381 811 int ret;
+5 -5
kernel/trace/trace_selftest.c
··· 1225 1225 /* check both trace buffers */ 1226 1226 ret = trace_test_buffer(&tr->array_buffer, NULL); 1227 1227 if (!ret) 1228 - ret = trace_test_buffer(&tr->max_buffer, &count); 1228 + ret = trace_test_buffer(&tr->snapshot_buffer, &count); 1229 1229 trace->reset(tr); 1230 1230 tracing_start(); 1231 1231 ··· 1287 1287 /* check both trace buffers */ 1288 1288 ret = trace_test_buffer(&tr->array_buffer, NULL); 1289 1289 if (!ret) 1290 - ret = trace_test_buffer(&tr->max_buffer, &count); 1290 + ret = trace_test_buffer(&tr->snapshot_buffer, &count); 1291 1291 trace->reset(tr); 1292 1292 tracing_start(); 1293 1293 ··· 1355 1355 if (ret) 1356 1356 goto out; 1357 1357 1358 - ret = trace_test_buffer(&tr->max_buffer, &count); 1358 + ret = trace_test_buffer(&tr->snapshot_buffer, &count); 1359 1359 if (ret) 1360 1360 goto out; 1361 1361 ··· 1385 1385 if (ret) 1386 1386 goto out; 1387 1387 1388 - ret = trace_test_buffer(&tr->max_buffer, &count); 1388 + ret = trace_test_buffer(&tr->snapshot_buffer, &count); 1389 1389 1390 1390 if (!ret && !count) { 1391 1391 printk(KERN_CONT ".. no entries found .."); ··· 1513 1513 /* check both trace buffers */ 1514 1514 ret = trace_test_buffer(&tr->array_buffer, NULL); 1515 1515 if (!ret) 1516 - ret = trace_test_buffer(&tr->max_buffer, &count); 1516 + ret = trace_test_buffer(&tr->snapshot_buffer, &count); 1517 1517 1518 1518 1519 1519 trace->reset(tr);
+28 -1
kernel/trace/trace_seq.c
··· 106 106 * Writes a ASCII representation of a bitmask string into @s. 107 107 */ 108 108 void trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, 109 - int nmaskbits) 109 + int nmaskbits) 110 110 { 111 111 unsigned int save_len = s->seq.len; 112 112 ··· 123 123 } 124 124 } 125 125 EXPORT_SYMBOL_GPL(trace_seq_bitmask); 126 + 127 + /** 128 + * trace_seq_bitmask_list - write a bitmask array in its list representation 129 + * @s: trace sequence descriptor 130 + * @maskp: points to an array of unsigned longs that represent a bitmask 131 + * @nmaskbits: The number of bits that are valid in @maskp 132 + * 133 + * Writes a list representation (e.g., 0-3,5-7) of a bitmask string into @s. 134 + */ 135 + void trace_seq_bitmask_list(struct trace_seq *s, const unsigned long *maskp, 136 + int nmaskbits) 137 + { 138 + unsigned int save_len = s->seq.len; 139 + 140 + if (s->full) 141 + return; 142 + 143 + __trace_seq_init(s); 144 + 145 + seq_buf_printf(&s->seq, "%*pbl", nmaskbits, maskp); 146 + 147 + if (unlikely(seq_buf_has_overflowed(&s->seq))) { 148 + s->seq.len = save_len; 149 + s->full = 1; 150 + } 151 + } 152 + EXPORT_SYMBOL_GPL(trace_seq_bitmask_list); 126 153 127 154 /** 128 155 * trace_seq_vprintf - sequence printing of trace information
+14 -4
kernel/tracepoint.c
··· 34 34 35 35 struct tp_transition_snapshot { 36 36 unsigned long rcu; 37 + unsigned long srcu_gp; 37 38 bool ongoing; 38 39 }; 40 + 41 + DEFINE_SRCU_FAST(tracepoint_srcu); 42 + EXPORT_SYMBOL_GPL(tracepoint_srcu); 39 43 40 44 /* Protected by tracepoints_mutex */ 41 45 static struct tp_transition_snapshot tp_transition_snapshot[_NR_TP_TRANSITION_SYNC]; ··· 50 46 51 47 /* Keep the latest get_state snapshot. */ 52 48 snapshot->rcu = get_state_synchronize_rcu(); 49 + snapshot->srcu_gp = start_poll_synchronize_srcu(&tracepoint_srcu); 53 50 snapshot->ongoing = true; 54 51 } 55 52 ··· 61 56 if (!snapshot->ongoing) 62 57 return; 63 58 cond_synchronize_rcu(snapshot->rcu); 59 + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) 60 + synchronize_srcu(&tracepoint_srcu); 64 61 snapshot->ongoing = false; 65 62 } 66 63 ··· 119 112 struct tp_probes *tp_probes = container_of(old, 120 113 struct tp_probes, probes[0]); 121 114 122 - if (tracepoint_is_faultable(tp)) 123 - call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); 124 - else 125 - call_rcu(&tp_probes->rcu, rcu_free_old_probes); 115 + if (tracepoint_is_faultable(tp)) { 116 + call_rcu_tasks_trace(&tp_probes->rcu, 117 + rcu_free_old_probes); 118 + } else { 119 + call_srcu(&tracepoint_srcu, &tp_probes->rcu, 120 + rcu_free_old_probes); 121 + } 126 122 } 127 123 } 128 124