Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'trace-v7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

Pull tracing updates from Steven Rostedt:

- Fix printf format warning for bprintf

sunrpc uses a trace_printk() that triggers a printf warning during
the compile. Move the __printf() attribute around for when debugging
is not enabled the warning will go away

- Remove redundant check for EVENT_FILE_FL_FREED in
event_filter_write()

The FREED flag is checked in the call to event_file_file() and then
checked again right afterward, which is unneeded

- Clean up event_file_file() and event_file_data() helpers

These helper functions played a different role in the past, but now
with eventfs, the READ_ONCE() isn't needed. Simplify the code a bit
and also add a warning to event_file_data() if the file or its data
is not present

- Remove updating file->private_data in tracing open

All access to the file private data is handled by the helper
functions, which do not use file->private_data. Stop updating it on
open

- Show ENUM names in function arguments via BTF in function tracing

When showing the function arguments when func-args option is set for
function tracing, if one of the arguments is found to be an enum,
show the name of the enum instead of its number

- Add new trace_call__##name() API for tracepoints

Tracepoints are enabled via static_branch() blocks, where when not
enabled, there's only a nop that is in the code where the execution
will just skip over it. When tracing is enabled, the nop is converted
to a direct jump to the tracepoint code. Sometimes more calculations
are required to be performed to update the parameters of the
tracepoint. In this case, trace_##name##_enabled() is called which is
a static_branch() that gets enabled only when the tracepoint is
enabled. This allows the extra calculations to also be skipped by the
nop:

if (trace_foo_enabled()) {
x = bar();
trace_foo(x);
}

Where the x=bar() is only performed when foo is enabled. The problem
with this approach is that there's now two static_branch() calls. One
for checking if the tracepoint is enabled, and then again to know if
the tracepoint should be called. The second one is redundant

Introduce trace_call__foo() that will call the foo() tracepoint
directly without doing a static_branch():

if (trace_foo_enabled()) {
x = bar();
trace_call__foo();
}

- Update various locations to use the new trace_call__##name() API

- Move snapshot code out of trace.c

Cleaning up trace.c to not be a "dump all", move the snapshot code
out of it and into a new trace_snapshot.c file

- Clean up some "%*.s" to "%*s"

- Allow boot kernel command line options to be called multiple times

Have options like:

ftrace_filter=foo ftrace_filter=bar ftrace_filter=zoo

Equal to:

ftrace_filter=foo,bar,zoo

- Fix ipi_raise event CPU field to be a CPU field

The ipi_raise target_cpus field is defined as a __bitmask(). There is
now a __cpumask() field definition. Update the field to use that

- Have hist_field_name() use a snprintf() and not a series of strcat()

It's safer to use snprintf() that a series of strcat()

- Fix tracepoint regfunc balancing

A tracepoint can define a "reg" and "unreg" function that gets called
before the tracepoint is enabled, and after it is disabled
respectively. But on error, after the "reg" func is called and the
tracepoint is not enabled, the "unreg" function is not called to tear
down what the "reg" function performed

- Fix output that shows what histograms are enabled

Event variables are displayed incorrectly in the histogram output

Instead of "sched.sched_wakeup.$var", it is showing
"$sched.sched_wakeup.var" where the '$' is in the incorrect location

- Some other simple cleanups

* tag 'trace-v7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (24 commits)
selftests/ftrace: Add test case for fully-qualified variable references
tracing: Fix fully-qualified variable reference printing in histograms
tracepoint: balance regfunc() on func_add() failure in tracepoint_add_func()
tracing: Rebuild full_name on each hist_field_name() call
tracing: Report ipi_raise target CPUs as cpumask
tracing: Remove duplicate latency_fsnotify() stub
tracing: Preserve repeated trace_trigger boot parameters
tracing: Append repeated boot-time tracing parameters
tracing: Remove spurious default precision from show_event_trigger/filter formats
cpufreq: Use trace_call__##name() at guarded tracepoint call sites
tracing: Remove tracing_alloc_snapshot() when snapshot isn't defined
tracing: Move snapshot code out of trace.c and into trace_snapshot.c
mm: damon: Use trace_call__##name() at guarded tracepoint call sites
btrfs: Use trace_call__##name() at guarded tracepoint call sites
spi: Use trace_call__##name() at guarded tracepoint call sites
i2c: Use trace_call__##name() at guarded tracepoint call sites
kernel: Use trace_call__##name() at guarded tracepoint call sites
tracepoint: Add trace_call__##name() API
tracing: trace_mmap.h: fix a kernel-doc warning
tracing: Pretty-print enum parameters in function arguments
...

+1355 -1250
+5 -5
drivers/cpufreq/amd-pstate.c
··· 256 256 if (trace_amd_pstate_epp_perf_enabled()) { 257 257 union perf_cached perf = READ_ONCE(cpudata->perf); 258 258 259 - trace_amd_pstate_epp_perf(cpudata->cpu, 259 + trace_call__amd_pstate_epp_perf(cpudata->cpu, 260 260 perf.highest_perf, 261 261 epp, 262 262 min_perf, ··· 306 306 if (trace_amd_pstate_epp_perf_enabled()) { 307 307 union perf_cached perf = cpudata->perf; 308 308 309 - trace_amd_pstate_epp_perf(cpudata->cpu, perf.highest_perf, 309 + trace_call__amd_pstate_epp_perf(cpudata->cpu, perf.highest_perf, 310 310 epp, 311 311 FIELD_GET(AMD_CPPC_MIN_PERF_MASK, 312 312 cpudata->cppc_req_cached), ··· 420 420 if (trace_amd_pstate_epp_perf_enabled()) { 421 421 union perf_cached perf = cpudata->perf; 422 422 423 - trace_amd_pstate_epp_perf(cpudata->cpu, perf.highest_perf, 423 + trace_call__amd_pstate_epp_perf(cpudata->cpu, perf.highest_perf, 424 424 epp, 425 425 FIELD_GET(AMD_CPPC_MIN_PERF_MASK, 426 426 cpudata->cppc_req_cached), ··· 585 585 if (trace_amd_pstate_epp_perf_enabled()) { 586 586 union perf_cached perf = READ_ONCE(cpudata->perf); 587 587 588 - trace_amd_pstate_epp_perf(cpudata->cpu, 588 + trace_call__amd_pstate_epp_perf(cpudata->cpu, 589 589 perf.highest_perf, 590 590 epp, 591 591 min_perf, ··· 663 663 } 664 664 665 665 if (trace_amd_pstate_perf_enabled() && amd_pstate_sample(cpudata)) { 666 - trace_amd_pstate_perf(min_perf, des_perf, max_perf, cpudata->freq, 666 + trace_call__amd_pstate_perf(min_perf, des_perf, max_perf, cpudata->freq, 667 667 cpudata->cur.mperf, cpudata->cur.aperf, cpudata->cur.tsc, 668 668 cpudata->cpu, fast_switch); 669 669 }
+1 -1
drivers/cpufreq/cpufreq.c
··· 2212 2212 2213 2213 if (trace_cpu_frequency_enabled()) { 2214 2214 for_each_cpu(cpu, policy->cpus) 2215 - trace_cpu_frequency(freq, cpu); 2215 + trace_call__cpu_frequency(freq, cpu); 2216 2216 } 2217 2217 2218 2218 return freq;
+1 -1
drivers/cpufreq/intel_pstate.c
··· 3132 3132 return; 3133 3133 3134 3134 sample = &cpu->sample; 3135 - trace_pstate_sample(trace_type, 3135 + trace_call__pstate_sample(trace_type, 3136 3136 0, 3137 3137 old_pstate, 3138 3138 cpu->pstate.current_pstate,
+1 -1
drivers/i2c/i2c-core-slave.c
··· 89 89 int ret = client->slave_cb(client, event, val); 90 90 91 91 if (trace_i2c_slave_enabled()) 92 - trace_i2c_slave(client, event, val, ret); 92 + trace_call__i2c_slave(client, event, val, ret); 93 93 94 94 return ret; 95 95 }
+2 -2
drivers/spi/spi-axi-spi-engine.c
··· 953 953 struct spi_transfer *xfer; 954 954 955 955 list_for_each_entry(xfer, &msg->transfers, transfer_list) 956 - trace_spi_transfer_start(msg, xfer); 956 + trace_call__spi_transfer_start(msg, xfer); 957 957 } 958 958 959 959 spin_lock_irqsave(&spi_engine->lock, flags); ··· 987 987 struct spi_transfer *xfer; 988 988 989 989 list_for_each_entry(xfer, &msg->transfers, transfer_list) 990 - trace_spi_transfer_stop(msg, xfer); 990 + trace_call__spi_transfer_stop(msg, xfer); 991 991 } 992 992 993 993 out:
+2 -2
fs/btrfs/extent_map.c
··· 1318 1318 if (trace_btrfs_extent_map_shrinker_scan_enter_enabled()) { 1319 1319 s64 nr = percpu_counter_sum_positive(&fs_info->evictable_extent_maps); 1320 1320 1321 - trace_btrfs_extent_map_shrinker_scan_enter(fs_info, nr); 1321 + trace_call__btrfs_extent_map_shrinker_scan_enter(fs_info, nr); 1322 1322 } 1323 1323 1324 1324 while (ctx.scanned < ctx.nr_to_scan && !btrfs_fs_closing(fs_info)) { ··· 1358 1358 if (trace_btrfs_extent_map_shrinker_scan_exit_enabled()) { 1359 1359 s64 nr = percpu_counter_sum_positive(&fs_info->evictable_extent_maps); 1360 1360 1361 - trace_btrfs_extent_map_shrinker_scan_exit(fs_info, nr_dropped, nr); 1361 + trace_call__btrfs_extent_map_shrinker_scan_exit(fs_info, nr_dropped, nr); 1362 1362 } 1363 1363 1364 1364 atomic64_set(&fs_info->em_shrinker_nr_to_scan, 0);
+2 -2
fs/btrfs/raid56.c
··· 1719 1719 struct raid56_bio_trace_info trace_info = { 0 }; 1720 1720 1721 1721 bio_get_trace_info(rbio, bio, &trace_info); 1722 - trace_raid56_read(rbio, bio, &trace_info); 1722 + trace_call__raid56_read(rbio, bio, &trace_info); 1723 1723 } 1724 1724 submit_bio(bio); 1725 1725 } ··· 2404 2404 struct raid56_bio_trace_info trace_info = { 0 }; 2405 2405 2406 2406 bio_get_trace_info(rbio, bio, &trace_info); 2407 - trace_raid56_write(rbio, bio, &trace_info); 2407 + trace_call__raid56_write(rbio, bio, &trace_info); 2408 2408 } 2409 2409 submit_bio(bio); 2410 2410 }
+1 -1
include/linux/ftrace.h
··· 31 31 #define ARCH_SUPPORTS_FTRACE_OPS 0 32 32 #endif 33 33 34 - #ifdef CONFIG_TRACING 34 + #ifdef CONFIG_TRACER_SNAPSHOT 35 35 extern void ftrace_boot_snapshot(void); 36 36 #else 37 37 static inline void ftrace_boot_snapshot(void) { }
-1
include/linux/trace_printk.h
··· 107 107 __trace_printk(_THIS_IP_, fmt, ##args); \ 108 108 } while (0) 109 109 110 - extern __printf(2, 3) 111 110 int __trace_bprintk(unsigned long ip, const char *fmt, ...); 112 111 113 112 extern __printf(2, 3)
+11
include/linux/tracepoint.h
··· 314 314 WARN_ONCE(!rcu_is_watching(), \ 315 315 "RCU not watching for tracepoint"); \ 316 316 } \ 317 + } \ 318 + static inline void trace_call__##name(proto) \ 319 + { \ 320 + __do_trace_##name(args); \ 317 321 } 318 322 319 323 #define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \ ··· 337 333 WARN_ONCE(!rcu_is_watching(), \ 338 334 "RCU not watching for tracepoint"); \ 339 335 } \ 336 + } \ 337 + static inline void trace_call__##name(proto) \ 338 + { \ 339 + might_fault(); \ 340 + __do_trace_##name(args); \ 340 341 } 341 342 342 343 /* ··· 426 417 #else /* !TRACEPOINTS_ENABLED */ 427 418 #define __DECLARE_TRACE_COMMON(name, proto, args, data_proto) \ 428 419 static inline void trace_##name(proto) \ 420 + { } \ 421 + static inline void trace_call__##name(proto) \ 429 422 { } \ 430 423 static inline int \ 431 424 register_trace_##name(void (*probe)(data_proto), \
+3 -3
include/trace/events/ipi.h
··· 68 68 TP_ARGS(mask, reason), 69 69 70 70 TP_STRUCT__entry( 71 - __bitmask(target_cpus, nr_cpumask_bits) 71 + __cpumask(target_cpus) 72 72 __field(const char *, reason) 73 73 ), 74 74 75 75 TP_fast_assign( 76 - __assign_bitmask(target_cpus, cpumask_bits(mask), nr_cpumask_bits); 76 + __assign_cpumask(target_cpus, cpumask_bits(mask)); 77 77 __entry->reason = reason; 78 78 ), 79 79 80 - TP_printk("target_mask=%s (%s)", __get_bitmask(target_cpus), __entry->reason) 80 + TP_printk("target_mask=%s (%s)", __get_cpumask(target_cpus), __entry->reason) 81 81 ); 82 82 83 83 DECLARE_EVENT_CLASS(ipi_handler,
+1
include/uapi/linux/trace_mmap.h
··· 10 10 * @meta_struct_len: Size of this structure. 11 11 * @subbuf_size: Size of each sub-buffer. 12 12 * @nr_subbufs: Number of subbfs in the ring-buffer, including the reader. 13 + * @reader: The reader composite info structure 13 14 * @reader.lost_events: Number of events lost at the time of the reader swap. 14 15 * @reader.id: subbuf ID of the current reader. ID range [0 : @nr_subbufs - 1] 15 16 * @reader.read: Number of bytes read on the reader subbuf.
+1 -1
kernel/irq_work.c
··· 79 79 static __always_inline void irq_work_raise(struct irq_work *work) 80 80 { 81 81 if (trace_ipi_send_cpu_enabled() && arch_irq_work_has_interrupt()) 82 - trace_ipi_send_cpu(smp_processor_id(), _RET_IP_, work->func); 82 + trace_call__ipi_send_cpu(smp_processor_id(), _RET_IP_, work->func); 83 83 84 84 arch_irq_work_raise(); 85 85 }
+1 -1
kernel/sched/ext.c
··· 5943 5943 vscnprintf(line_buf, sizeof(line_buf), fmt, args); 5944 5944 va_end(args); 5945 5945 5946 - trace_sched_ext_dump(line_buf); 5946 + trace_call__sched_ext_dump(line_buf); 5947 5947 } 5948 5948 #endif 5949 5949 /* @s may be zero sized and seq_buf triggers WARN if so */
+1 -1
kernel/smp.c
··· 408 408 func = CSD_TYPE(csd) == CSD_TYPE_TTWU ? 409 409 sched_ttwu_pending : csd->func; 410 410 411 - trace_csd_queue_cpu(cpu, _RET_IP_, func, csd); 411 + trace_call__csd_queue_cpu(cpu, _RET_IP_, func, csd); 412 412 } 413 413 414 414 /*
+1
kernel/trace/Makefile
··· 69 69 obj-$(CONFIG_TRACING) += trace_stat.o 70 70 obj-$(CONFIG_TRACING) += trace_printk.o 71 71 obj-$(CONFIG_TRACING) += trace_pid.o 72 + obj-$(CONFIG_TRACER_SNAPSHOT) += trace_snapshot.o 72 73 obj-$(CONFIG_TRACING) += pid_list.o 73 74 obj-$(CONFIG_TRACING_MAP) += tracing_map.o 74 75 obj-$(CONFIG_PREEMPTIRQ_DELAY_TEST) += preemptirq_delay_test.o
+8 -4
kernel/trace/ftrace.c
··· 6841 6841 static int __init set_ftrace_notrace(char *str) 6842 6842 { 6843 6843 ftrace_filter_param = true; 6844 - strscpy(ftrace_notrace_buf, str, FTRACE_FILTER_SIZE); 6844 + trace_append_boot_param(ftrace_notrace_buf, str, ',', 6845 + FTRACE_FILTER_SIZE); 6845 6846 return 1; 6846 6847 } 6847 6848 __setup("ftrace_notrace=", set_ftrace_notrace); ··· 6850 6849 static int __init set_ftrace_filter(char *str) 6851 6850 { 6852 6851 ftrace_filter_param = true; 6853 - strscpy(ftrace_filter_buf, str, FTRACE_FILTER_SIZE); 6852 + trace_append_boot_param(ftrace_filter_buf, str, ',', 6853 + FTRACE_FILTER_SIZE); 6854 6854 return 1; 6855 6855 } 6856 6856 __setup("ftrace_filter=", set_ftrace_filter); ··· 6863 6861 6864 6862 static int __init set_graph_function(char *str) 6865 6863 { 6866 - strscpy(ftrace_graph_buf, str, FTRACE_FILTER_SIZE); 6864 + trace_append_boot_param(ftrace_graph_buf, str, ',', 6865 + FTRACE_FILTER_SIZE); 6867 6866 return 1; 6868 6867 } 6869 6868 __setup("ftrace_graph_filter=", set_graph_function); 6870 6869 6871 6870 static int __init set_graph_notrace_function(char *str) 6872 6871 { 6873 - strscpy(ftrace_graph_notrace_buf, str, FTRACE_FILTER_SIZE); 6872 + trace_append_boot_param(ftrace_graph_notrace_buf, str, ',', 6873 + FTRACE_FILTER_SIZE); 6874 6874 return 1; 6875 6875 } 6876 6876 __setup("ftrace_graph_notrace=", set_graph_notrace_function);
+52 -1178
kernel/trace/trace.c
··· 47 47 #include <linux/trace.h> 48 48 #include <linux/sched/clock.h> 49 49 #include <linux/sched/rt.h> 50 - #include <linux/fsnotify.h> 51 50 #include <linux/irq_work.h> 52 51 #include <linux/workqueue.h> 53 52 #include <linux/sort.h> ··· 218 219 static char bootup_tracer_buf[MAX_TRACER_SIZE] __initdata; 219 220 static char *default_bootup_tracer; 220 221 221 - static bool allocate_snapshot; 222 - static bool snapshot_at_boot; 223 - 224 222 static char boot_instance_info[COMMAND_LINE_SIZE] __initdata; 225 223 static int boot_instance_index; 226 224 227 - static char boot_snapshot_info[COMMAND_LINE_SIZE] __initdata; 228 - static int boot_snapshot_index; 225 + /* 226 + * Repeated boot parameters, including Bootconfig array expansions, need 227 + * to stay in the delimiter form that the existing parser consumes. 228 + */ 229 + void __init trace_append_boot_param(char *buf, const char *str, char sep, 230 + int size) 231 + { 232 + int len, needed, str_len; 233 + 234 + if (!*str) 235 + return; 236 + 237 + len = strlen(buf); 238 + str_len = strlen(str); 239 + needed = len + str_len + 1; 240 + 241 + /* For continuation, account for the separator. */ 242 + if (len) 243 + needed++; 244 + if (needed > size) 245 + return; 246 + 247 + if (len) 248 + buf[len++] = sep; 249 + 250 + strscpy(buf + len, str, size - len); 251 + } 229 252 230 253 static int __init set_cmdline_ftrace(char *str) 231 254 { ··· 297 276 } 298 277 __setup("traceoff_on_warning", stop_trace_on_warning); 299 278 300 - static int __init boot_alloc_snapshot(char *str) 301 - { 302 - char *slot = boot_snapshot_info + boot_snapshot_index; 303 - int left = sizeof(boot_snapshot_info) - boot_snapshot_index; 304 - int ret; 305 - 306 - if (str[0] == '=') { 307 - str++; 308 - if (strlen(str) >= left) 309 - return -1; 310 - 311 - ret = snprintf(slot, left, "%s\t", str); 312 - boot_snapshot_index += ret; 313 - } else { 314 - allocate_snapshot = true; 315 - /* We also need the main ring buffer expanded */ 316 - trace_set_ring_buffer_expanded(NULL); 317 - } 318 - return 1; 319 - } 320 - __setup("alloc_snapshot", boot_alloc_snapshot); 321 - 322 - 323 - static int __init boot_snapshot(char *str) 324 - { 325 - snapshot_at_boot = true; 326 - boot_alloc_snapshot(str); 327 - return 1; 328 - } 329 - __setup("ftrace_boot_snapshot", boot_snapshot); 330 - 331 - 332 279 static int __init boot_instance(char *str) 333 280 { 334 281 char *slot = boot_instance_info + boot_instance_index; ··· 318 329 319 330 static int __init set_trace_boot_options(char *str) 320 331 { 321 - strscpy(trace_boot_options_buf, str, MAX_TRACER_SIZE); 332 + trace_append_boot_param(trace_boot_options_buf, str, ',', 333 + MAX_TRACER_SIZE); 322 334 return 1; 323 335 } 324 336 __setup("trace_options=", set_trace_boot_options); ··· 853 863 EXPORT_SYMBOL_GPL(tracing_on); 854 864 855 865 #ifdef CONFIG_TRACER_SNAPSHOT 856 - static void tracing_snapshot_instance_cond(struct trace_array *tr, 857 - void *cond_data) 858 - { 859 - unsigned long flags; 860 - 861 - if (in_nmi()) { 862 - trace_array_puts(tr, "*** SNAPSHOT CALLED FROM NMI CONTEXT ***\n"); 863 - trace_array_puts(tr, "*** snapshot is being ignored ***\n"); 864 - return; 865 - } 866 - 867 - if (!tr->allocated_snapshot) { 868 - trace_array_puts(tr, "*** SNAPSHOT NOT ALLOCATED ***\n"); 869 - trace_array_puts(tr, "*** stopping trace here! ***\n"); 870 - tracer_tracing_off(tr); 871 - return; 872 - } 873 - 874 - if (tr->mapped) { 875 - trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n"); 876 - trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 877 - return; 878 - } 879 - 880 - /* Note, snapshot can not be used when the tracer uses it */ 881 - if (tracer_uses_snapshot(tr->current_trace)) { 882 - trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n"); 883 - trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 884 - return; 885 - } 886 - 887 - local_irq_save(flags); 888 - update_max_tr(tr, current, smp_processor_id(), cond_data); 889 - local_irq_restore(flags); 890 - } 891 - 892 - void tracing_snapshot_instance(struct trace_array *tr) 893 - { 894 - tracing_snapshot_instance_cond(tr, NULL); 895 - } 896 - 897 866 /** 898 867 * tracing_snapshot - take a snapshot of the current buffer. 899 868 * ··· 876 927 EXPORT_SYMBOL_GPL(tracing_snapshot); 877 928 878 929 /** 879 - * tracing_snapshot_cond - conditionally take a snapshot of the current buffer. 880 - * @tr: The tracing instance to snapshot 881 - * @cond_data: The data to be tested conditionally, and possibly saved 882 - * 883 - * This is the same as tracing_snapshot() except that the snapshot is 884 - * conditional - the snapshot will only happen if the 885 - * cond_snapshot.update() implementation receiving the cond_data 886 - * returns true, which means that the trace array's cond_snapshot 887 - * update() operation used the cond_data to determine whether the 888 - * snapshot should be taken, and if it was, presumably saved it along 889 - * with the snapshot. 890 - */ 891 - void tracing_snapshot_cond(struct trace_array *tr, void *cond_data) 892 - { 893 - tracing_snapshot_instance_cond(tr, cond_data); 894 - } 895 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond); 896 - 897 - /** 898 - * tracing_cond_snapshot_data - get the user data associated with a snapshot 899 - * @tr: The tracing instance 900 - * 901 - * When the user enables a conditional snapshot using 902 - * tracing_snapshot_cond_enable(), the user-defined cond_data is saved 903 - * with the snapshot. This accessor is used to retrieve it. 904 - * 905 - * Should not be called from cond_snapshot.update(), since it takes 906 - * the tr->max_lock lock, which the code calling 907 - * cond_snapshot.update() has already done. 908 - * 909 - * Returns the cond_data associated with the trace array's snapshot. 910 - */ 911 - void *tracing_cond_snapshot_data(struct trace_array *tr) 912 - { 913 - void *cond_data = NULL; 914 - 915 - local_irq_disable(); 916 - arch_spin_lock(&tr->max_lock); 917 - 918 - if (tr->cond_snapshot) 919 - cond_data = tr->cond_snapshot->cond_data; 920 - 921 - arch_spin_unlock(&tr->max_lock); 922 - local_irq_enable(); 923 - 924 - return cond_data; 925 - } 926 - EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data); 927 - 928 - static int resize_buffer_duplicate_size(struct array_buffer *trace_buf, 929 - struct array_buffer *size_buf, int cpu_id); 930 - static void set_buffer_entries(struct array_buffer *buf, unsigned long val); 931 - 932 - int tracing_alloc_snapshot_instance(struct trace_array *tr) 933 - { 934 - int order; 935 - int ret; 936 - 937 - if (!tr->allocated_snapshot) { 938 - 939 - /* Make the snapshot buffer have the same order as main buffer */ 940 - order = ring_buffer_subbuf_order_get(tr->array_buffer.buffer); 941 - ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order); 942 - if (ret < 0) 943 - return ret; 944 - 945 - /* allocate spare buffer */ 946 - ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 947 - &tr->array_buffer, RING_BUFFER_ALL_CPUS); 948 - if (ret < 0) 949 - return ret; 950 - 951 - tr->allocated_snapshot = true; 952 - } 953 - 954 - return 0; 955 - } 956 - 957 - static void free_snapshot(struct trace_array *tr) 958 - { 959 - /* 960 - * We don't free the ring buffer. instead, resize it because 961 - * The max_tr ring buffer has some state (e.g. ring->clock) and 962 - * we want preserve it. 963 - */ 964 - ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, 0); 965 - ring_buffer_resize(tr->snapshot_buffer.buffer, 1, RING_BUFFER_ALL_CPUS); 966 - set_buffer_entries(&tr->snapshot_buffer, 1); 967 - tracing_reset_online_cpus(&tr->snapshot_buffer); 968 - tr->allocated_snapshot = false; 969 - } 970 - 971 - static int tracing_arm_snapshot_locked(struct trace_array *tr) 972 - { 973 - int ret; 974 - 975 - lockdep_assert_held(&trace_types_lock); 976 - 977 - spin_lock(&tr->snapshot_trigger_lock); 978 - if (tr->snapshot == UINT_MAX || tr->mapped) { 979 - spin_unlock(&tr->snapshot_trigger_lock); 980 - return -EBUSY; 981 - } 982 - 983 - tr->snapshot++; 984 - spin_unlock(&tr->snapshot_trigger_lock); 985 - 986 - ret = tracing_alloc_snapshot_instance(tr); 987 - if (ret) { 988 - spin_lock(&tr->snapshot_trigger_lock); 989 - tr->snapshot--; 990 - spin_unlock(&tr->snapshot_trigger_lock); 991 - } 992 - 993 - return ret; 994 - } 995 - 996 - int tracing_arm_snapshot(struct trace_array *tr) 997 - { 998 - guard(mutex)(&trace_types_lock); 999 - return tracing_arm_snapshot_locked(tr); 1000 - } 1001 - 1002 - void tracing_disarm_snapshot(struct trace_array *tr) 1003 - { 1004 - spin_lock(&tr->snapshot_trigger_lock); 1005 - if (!WARN_ON(!tr->snapshot)) 1006 - tr->snapshot--; 1007 - spin_unlock(&tr->snapshot_trigger_lock); 1008 - } 1009 - 1010 - /** 1011 930 * tracing_alloc_snapshot - allocate snapshot buffer. 1012 931 * 1013 932 * This only allocates the snapshot buffer if it isn't already ··· 895 1078 896 1079 return ret; 897 1080 } 898 - EXPORT_SYMBOL_GPL(tracing_alloc_snapshot); 899 - 900 - /** 901 - * tracing_snapshot_alloc - allocate and take a snapshot of the current buffer. 902 - * 903 - * This is similar to tracing_snapshot(), but it will allocate the 904 - * snapshot buffer if it isn't already allocated. Use this only 905 - * where it is safe to sleep, as the allocation may sleep. 906 - * 907 - * This causes a swap between the snapshot buffer and the current live 908 - * tracing buffer. You can use this to take snapshots of the live 909 - * trace when some condition is triggered, but continue to trace. 910 - */ 911 - void tracing_snapshot_alloc(void) 912 - { 913 - int ret; 914 - 915 - ret = tracing_alloc_snapshot(); 916 - if (ret < 0) 917 - return; 918 - 919 - tracing_snapshot(); 920 - } 921 - EXPORT_SYMBOL_GPL(tracing_snapshot_alloc); 922 - 923 - /** 924 - * tracing_snapshot_cond_enable - enable conditional snapshot for an instance 925 - * @tr: The tracing instance 926 - * @cond_data: User data to associate with the snapshot 927 - * @update: Implementation of the cond_snapshot update function 928 - * 929 - * Check whether the conditional snapshot for the given instance has 930 - * already been enabled, or if the current tracer is already using a 931 - * snapshot; if so, return -EBUSY, else create a cond_snapshot and 932 - * save the cond_data and update function inside. 933 - * 934 - * Returns 0 if successful, error otherwise. 935 - */ 936 - int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, 937 - cond_update_fn_t update) 938 - { 939 - struct cond_snapshot *cond_snapshot __free(kfree) = 940 - kzalloc_obj(*cond_snapshot); 941 - int ret; 942 - 943 - if (!cond_snapshot) 944 - return -ENOMEM; 945 - 946 - cond_snapshot->cond_data = cond_data; 947 - cond_snapshot->update = update; 948 - 949 - guard(mutex)(&trace_types_lock); 950 - 951 - if (tracer_uses_snapshot(tr->current_trace)) 952 - return -EBUSY; 953 - 954 - /* 955 - * The cond_snapshot can only change to NULL without the 956 - * trace_types_lock. We don't care if we race with it going 957 - * to NULL, but we want to make sure that it's not set to 958 - * something other than NULL when we get here, which we can 959 - * do safely with only holding the trace_types_lock and not 960 - * having to take the max_lock. 961 - */ 962 - if (tr->cond_snapshot) 963 - return -EBUSY; 964 - 965 - ret = tracing_arm_snapshot_locked(tr); 966 - if (ret) 967 - return ret; 968 - 969 - local_irq_disable(); 970 - arch_spin_lock(&tr->max_lock); 971 - tr->cond_snapshot = no_free_ptr(cond_snapshot); 972 - arch_spin_unlock(&tr->max_lock); 973 - local_irq_enable(); 974 - 975 - return 0; 976 - } 977 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable); 978 - 979 - /** 980 - * tracing_snapshot_cond_disable - disable conditional snapshot for an instance 981 - * @tr: The tracing instance 982 - * 983 - * Check whether the conditional snapshot for the given instance is 984 - * enabled; if so, free the cond_snapshot associated with it, 985 - * otherwise return -EINVAL. 986 - * 987 - * Returns 0 if successful, error otherwise. 988 - */ 989 - int tracing_snapshot_cond_disable(struct trace_array *tr) 990 - { 991 - int ret = 0; 992 - 993 - local_irq_disable(); 994 - arch_spin_lock(&tr->max_lock); 995 - 996 - if (!tr->cond_snapshot) 997 - ret = -EINVAL; 998 - else { 999 - kfree(tr->cond_snapshot); 1000 - tr->cond_snapshot = NULL; 1001 - } 1002 - 1003 - arch_spin_unlock(&tr->max_lock); 1004 - local_irq_enable(); 1005 - 1006 - tracing_disarm_snapshot(tr); 1007 - 1008 - return ret; 1009 - } 1010 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable); 1011 1081 #else 1012 1082 void tracing_snapshot(void) 1013 1083 { 1014 1084 WARN_ONCE(1, "Snapshot feature not enabled, but internal snapshot used"); 1015 1085 } 1016 1086 EXPORT_SYMBOL_GPL(tracing_snapshot); 1017 - void tracing_snapshot_cond(struct trace_array *tr, void *cond_data) 1018 - { 1019 - WARN_ONCE(1, "Snapshot feature not enabled, but internal conditional snapshot used"); 1020 - } 1021 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond); 1022 - int tracing_alloc_snapshot(void) 1023 - { 1024 - WARN_ONCE(1, "Snapshot feature not enabled, but snapshot allocation used"); 1025 - return -ENODEV; 1026 - } 1027 - EXPORT_SYMBOL_GPL(tracing_alloc_snapshot); 1028 1087 void tracing_snapshot_alloc(void) 1029 1088 { 1030 1089 /* Give warning */ 1031 1090 tracing_snapshot(); 1032 1091 } 1033 1092 EXPORT_SYMBOL_GPL(tracing_snapshot_alloc); 1034 - void *tracing_cond_snapshot_data(struct trace_array *tr) 1035 - { 1036 - return NULL; 1037 - } 1038 - EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data); 1039 - int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update) 1040 - { 1041 - return -ENODEV; 1042 - } 1043 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable); 1044 - int tracing_snapshot_cond_disable(struct trace_array *tr) 1045 - { 1046 - return false; 1047 - } 1048 - EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable); 1049 - #define free_snapshot(tr) do { } while (0) 1050 - #define tracing_arm_snapshot_locked(tr) ({ -EBUSY; }) 1051 1093 #endif /* CONFIG_TRACER_SNAPSHOT */ 1052 1094 1053 1095 void tracer_tracing_off(struct trace_array *tr) ··· 1219 1543 1220 1544 unsigned long __read_mostly tracing_thresh; 1221 1545 1222 - #ifdef CONFIG_TRACER_MAX_TRACE 1223 - #ifdef LATENCY_FS_NOTIFY 1224 - static struct workqueue_struct *fsnotify_wq; 1225 - 1226 - static void latency_fsnotify_workfn(struct work_struct *work) 1227 - { 1228 - struct trace_array *tr = container_of(work, struct trace_array, 1229 - fsnotify_work); 1230 - fsnotify_inode(tr->d_max_latency->d_inode, FS_MODIFY); 1231 - } 1232 - 1233 - static void latency_fsnotify_workfn_irq(struct irq_work *iwork) 1234 - { 1235 - struct trace_array *tr = container_of(iwork, struct trace_array, 1236 - fsnotify_irqwork); 1237 - queue_work(fsnotify_wq, &tr->fsnotify_work); 1238 - } 1239 - 1240 - __init static int latency_fsnotify_init(void) 1241 - { 1242 - fsnotify_wq = alloc_workqueue("tr_max_lat_wq", 1243 - WQ_UNBOUND | WQ_HIGHPRI, 0); 1244 - if (!fsnotify_wq) { 1245 - pr_err("Unable to allocate tr_max_lat_wq\n"); 1246 - return -ENOMEM; 1247 - } 1248 - return 0; 1249 - } 1250 - 1251 - late_initcall_sync(latency_fsnotify_init); 1252 - 1253 - void latency_fsnotify(struct trace_array *tr) 1254 - { 1255 - if (!fsnotify_wq) 1256 - return; 1257 - /* 1258 - * We cannot call queue_work(&tr->fsnotify_work) from here because it's 1259 - * possible that we are called from __schedule() or do_idle(), which 1260 - * could cause a deadlock. 1261 - */ 1262 - irq_work_queue(&tr->fsnotify_irqwork); 1263 - } 1264 - #endif /* !LATENCY_FS_NOTIFY */ 1265 - 1266 - static const struct file_operations tracing_max_lat_fops; 1267 - 1268 - static void trace_create_maxlat_file(struct trace_array *tr, 1269 - struct dentry *d_tracer) 1270 - { 1271 - #ifdef LATENCY_FS_NOTIFY 1272 - INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); 1273 - init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); 1274 - #endif 1275 - tr->d_max_latency = trace_create_file("tracing_max_latency", 1276 - TRACE_MODE_WRITE, 1277 - d_tracer, tr, 1278 - &tracing_max_lat_fops); 1279 - } 1280 - 1281 - /* 1282 - * Copy the new maximum trace into the separate maximum-trace 1283 - * structure. (this way the maximum trace is permanently saved, 1284 - * for later retrieval via /sys/kernel/tracing/tracing_max_latency) 1285 - */ 1286 - static void 1287 - __update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu) 1288 - { 1289 - struct array_buffer *trace_buf = &tr->array_buffer; 1290 - struct trace_array_cpu *data = per_cpu_ptr(trace_buf->data, cpu); 1291 - struct array_buffer *max_buf = &tr->snapshot_buffer; 1292 - struct trace_array_cpu *max_data = per_cpu_ptr(max_buf->data, cpu); 1293 - 1294 - max_buf->cpu = cpu; 1295 - max_buf->time_start = data->preempt_timestamp; 1296 - 1297 - max_data->saved_latency = tr->max_latency; 1298 - max_data->critical_start = data->critical_start; 1299 - max_data->critical_end = data->critical_end; 1300 - 1301 - strscpy(max_data->comm, tsk->comm); 1302 - max_data->pid = tsk->pid; 1303 - /* 1304 - * If tsk == current, then use current_uid(), as that does not use 1305 - * RCU. The irq tracer can be called out of RCU scope. 1306 - */ 1307 - if (tsk == current) 1308 - max_data->uid = current_uid(); 1309 - else 1310 - max_data->uid = task_uid(tsk); 1311 - 1312 - max_data->nice = tsk->static_prio - 20 - MAX_RT_PRIO; 1313 - max_data->policy = tsk->policy; 1314 - max_data->rt_priority = tsk->rt_priority; 1315 - 1316 - /* record this tasks comm */ 1317 - tracing_record_cmdline(tsk); 1318 - latency_fsnotify(tr); 1319 - } 1320 - #else 1321 - static inline void trace_create_maxlat_file(struct trace_array *tr, 1322 - struct dentry *d_tracer) { } 1323 - static inline void __update_max_tr(struct trace_array *tr, 1324 - struct task_struct *tsk, int cpu) { } 1325 - #endif /* CONFIG_TRACER_MAX_TRACE */ 1326 - 1327 - #ifdef CONFIG_TRACER_SNAPSHOT 1328 - /** 1329 - * update_max_tr - snapshot all trace buffers from global_trace to max_tr 1330 - * @tr: tracer 1331 - * @tsk: the task with the latency 1332 - * @cpu: The cpu that initiated the trace. 1333 - * @cond_data: User data associated with a conditional snapshot 1334 - * 1335 - * Flip the buffers between the @tr and the max_tr and record information 1336 - * about which task was the cause of this latency. 1337 - */ 1338 - void 1339 - update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu, 1340 - void *cond_data) 1341 - { 1342 - if (tr->stop_count) 1343 - return; 1344 - 1345 - WARN_ON_ONCE(!irqs_disabled()); 1346 - 1347 - if (!tr->allocated_snapshot) { 1348 - /* Only the nop tracer should hit this when disabling */ 1349 - WARN_ON_ONCE(tr->current_trace != &nop_trace); 1350 - return; 1351 - } 1352 - 1353 - arch_spin_lock(&tr->max_lock); 1354 - 1355 - /* Inherit the recordable setting from array_buffer */ 1356 - if (ring_buffer_record_is_set_on(tr->array_buffer.buffer)) 1357 - ring_buffer_record_on(tr->snapshot_buffer.buffer); 1358 - else 1359 - ring_buffer_record_off(tr->snapshot_buffer.buffer); 1360 - 1361 - if (tr->cond_snapshot && !tr->cond_snapshot->update(tr, cond_data)) { 1362 - arch_spin_unlock(&tr->max_lock); 1363 - return; 1364 - } 1365 - 1366 - swap(tr->array_buffer.buffer, tr->snapshot_buffer.buffer); 1367 - 1368 - __update_max_tr(tr, tsk, cpu); 1369 - 1370 - arch_spin_unlock(&tr->max_lock); 1371 - 1372 - /* Any waiters on the old snapshot buffer need to wake up */ 1373 - ring_buffer_wake_waiters(tr->array_buffer.buffer, RING_BUFFER_ALL_CPUS); 1374 - } 1375 - 1376 - /** 1377 - * update_max_tr_single - only copy one trace over, and reset the rest 1378 - * @tr: tracer 1379 - * @tsk: task with the latency 1380 - * @cpu: the cpu of the buffer to copy. 1381 - * 1382 - * Flip the trace of a single CPU buffer between the @tr and the max_tr. 1383 - */ 1384 - void 1385 - update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu) 1386 - { 1387 - int ret; 1388 - 1389 - if (tr->stop_count) 1390 - return; 1391 - 1392 - WARN_ON_ONCE(!irqs_disabled()); 1393 - if (!tr->allocated_snapshot) { 1394 - /* Only the nop tracer should hit this when disabling */ 1395 - WARN_ON_ONCE(tr->current_trace != &nop_trace); 1396 - return; 1397 - } 1398 - 1399 - arch_spin_lock(&tr->max_lock); 1400 - 1401 - ret = ring_buffer_swap_cpu(tr->snapshot_buffer.buffer, tr->array_buffer.buffer, cpu); 1402 - 1403 - if (ret == -EBUSY) { 1404 - /* 1405 - * We failed to swap the buffer due to a commit taking 1406 - * place on this CPU. We fail to record, but we reset 1407 - * the max trace buffer (no one writes directly to it) 1408 - * and flag that it failed. 1409 - * Another reason is resize is in progress. 1410 - */ 1411 - trace_array_printk_buf(tr->snapshot_buffer.buffer, _THIS_IP_, 1412 - "Failed to swap buffers due to commit or resize in progress\n"); 1413 - } 1414 - 1415 - WARN_ON_ONCE(ret && ret != -EAGAIN && ret != -EBUSY); 1416 - 1417 - __update_max_tr(tr, tsk, cpu); 1418 - arch_spin_unlock(&tr->max_lock); 1419 - } 1420 - #endif /* CONFIG_TRACER_SNAPSHOT */ 1421 - 1422 1546 struct pipe_wait { 1423 1547 struct trace_iterator *iter; 1424 1548 int wait_index; ··· 1527 2051 return 0; 1528 2052 } 1529 2053 1530 - static void tracing_reset_cpu(struct array_buffer *buf, int cpu) 2054 + void tracing_reset_cpu(struct array_buffer *buf, int cpu) 1531 2055 { 1532 2056 struct trace_buffer *buffer = buf->buffer; 1533 2057 ··· 3292 3816 "# MAY BE MISSING FUNCTION EVENTS\n"); 3293 3817 } 3294 3818 3295 - #ifdef CONFIG_TRACER_SNAPSHOT 3296 - static void show_snapshot_main_help(struct seq_file *m) 3297 - { 3298 - seq_puts(m, "# echo 0 > snapshot : Clears and frees snapshot buffer\n" 3299 - "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n" 3300 - "# Takes a snapshot of the main buffer.\n" 3301 - "# echo 2 > snapshot : Clears snapshot buffer (but does not allocate or free)\n" 3302 - "# (Doesn't have to be '2' works with any number that\n" 3303 - "# is not a '0' or '1')\n"); 3304 - } 3305 - 3306 - static void show_snapshot_percpu_help(struct seq_file *m) 3307 - { 3308 - seq_puts(m, "# echo 0 > snapshot : Invalid for per_cpu snapshot file.\n"); 3309 - #ifdef CONFIG_RING_BUFFER_ALLOW_SWAP 3310 - seq_puts(m, "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n" 3311 - "# Takes a snapshot of the main buffer for this cpu.\n"); 3312 - #else 3313 - seq_puts(m, "# echo 1 > snapshot : Not supported with this kernel.\n" 3314 - "# Must use main snapshot file to allocate.\n"); 3315 - #endif 3316 - seq_puts(m, "# echo 2 > snapshot : Clears this cpu's snapshot buffer (but does not allocate)\n" 3317 - "# (Doesn't have to be '2' works with any number that\n" 3318 - "# is not a '0' or '1')\n"); 3319 - } 3320 - 3321 - static void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter) 3322 - { 3323 - if (iter->tr->allocated_snapshot) 3324 - seq_puts(m, "#\n# * Snapshot is allocated *\n#\n"); 3325 - else 3326 - seq_puts(m, "#\n# * Snapshot is freed *\n#\n"); 3327 - 3328 - seq_puts(m, "# Snapshot commands:\n"); 3329 - if (iter->cpu_file == RING_BUFFER_ALL_CPUS) 3330 - show_snapshot_main_help(m); 3331 - else 3332 - show_snapshot_percpu_help(m); 3333 - } 3334 - #else 3335 - /* Should never be called */ 3336 - static inline void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter) { } 3337 - #endif 3338 - 3339 3819 static int s_show(struct seq_file *m, void *v) 3340 3820 { 3341 3821 struct trace_iterator *iter = v; ··· 3340 3908 return 0; 3341 3909 } 3342 3910 3343 - /* 3344 - * Should be used after trace_array_get(), trace_types_lock 3345 - * ensures that i_cdev was already initialized. 3346 - */ 3347 - int tracing_get_cpu(struct inode *inode) 3348 - { 3349 - if (inode->i_cdev) /* See trace_create_cpu_file() */ 3350 - return (long)inode->i_cdev - 1; 3351 - return RING_BUFFER_ALL_CPUS; 3352 - } 3353 - 3354 3911 static const struct seq_operations tracer_seq_ops = { 3355 3912 .start = s_start, 3356 3913 .next = s_next, ··· 3366 3945 free_cpumask_var(iter->started); 3367 3946 } 3368 3947 3369 - static struct trace_iterator * 3948 + struct trace_iterator * 3370 3949 __tracing_open(struct inode *inode, struct file *file, bool snapshot) 3371 3950 { 3372 3951 struct trace_array *tr = inode->i_private; ··· 3532 4111 event_file_get(file); 3533 4112 } 3534 4113 3535 - filp->private_data = inode->i_private; 3536 - 3537 4114 return 0; 3538 4115 } 3539 4116 ··· 3551 4132 return single_release(inode, filp); 3552 4133 } 3553 4134 3554 - static int tracing_release(struct inode *inode, struct file *file) 4135 + int tracing_release(struct inode *inode, struct file *file) 3555 4136 { 3556 4137 struct trace_array *tr = inode->i_private; 3557 4138 struct seq_file *m = file->private_data; ··· 4702 5283 return t->init(tr); 4703 5284 } 4704 5285 4705 - static void set_buffer_entries(struct array_buffer *buf, unsigned long val) 5286 + void trace_set_buffer_entries(struct array_buffer *buf, unsigned long val) 4706 5287 { 4707 5288 int cpu; 4708 5289 ··· 4713 5294 static void update_buffer_entries(struct array_buffer *buf, int cpu) 4714 5295 { 4715 5296 if (cpu == RING_BUFFER_ALL_CPUS) { 4716 - set_buffer_entries(buf, ring_buffer_size(buf->buffer, 0)); 5297 + trace_set_buffer_entries(buf, ring_buffer_size(buf->buffer, 0)); 4717 5298 } else { 4718 5299 per_cpu_ptr(buf->data, cpu)->entries = ring_buffer_size(buf->buffer, cpu); 4719 5300 } 4720 5301 } 4721 - 4722 - #ifdef CONFIG_TRACER_SNAPSHOT 4723 - /* resize @tr's buffer to the size of @size_tr's entries */ 4724 - static int resize_buffer_duplicate_size(struct array_buffer *trace_buf, 4725 - struct array_buffer *size_buf, int cpu_id) 4726 - { 4727 - int cpu, ret = 0; 4728 - 4729 - if (cpu_id == RING_BUFFER_ALL_CPUS) { 4730 - for_each_tracing_cpu(cpu) { 4731 - ret = ring_buffer_resize(trace_buf->buffer, 4732 - per_cpu_ptr(size_buf->data, cpu)->entries, cpu); 4733 - if (ret < 0) 4734 - break; 4735 - per_cpu_ptr(trace_buf->data, cpu)->entries = 4736 - per_cpu_ptr(size_buf->data, cpu)->entries; 4737 - } 4738 - } else { 4739 - ret = ring_buffer_resize(trace_buf->buffer, 4740 - per_cpu_ptr(size_buf->data, cpu_id)->entries, cpu_id); 4741 - if (ret == 0) 4742 - per_cpu_ptr(trace_buf->data, cpu_id)->entries = 4743 - per_cpu_ptr(size_buf->data, cpu_id)->entries; 4744 - } 4745 - 4746 - return ret; 4747 - } 4748 - #endif /* CONFIG_TRACER_SNAPSHOT */ 4749 5302 4750 5303 static int __tracing_resize_ring_buffer(struct trace_array *tr, 4751 5304 unsigned long size, int cpu) ··· 5141 5750 return ret; 5142 5751 } 5143 5752 5144 - static ssize_t 5145 - tracing_nsecs_read(unsigned long *ptr, char __user *ubuf, 5146 - size_t cnt, loff_t *ppos) 5753 + ssize_t tracing_nsecs_read(unsigned long *ptr, char __user *ubuf, 5754 + size_t cnt, loff_t *ppos) 5147 5755 { 5148 5756 char buf[64]; 5149 5757 int r; ··· 5154 5764 return simple_read_from_buffer(ubuf, cnt, ppos, buf, r); 5155 5765 } 5156 5766 5157 - static ssize_t 5158 - tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf, 5159 - size_t cnt, loff_t *ppos) 5767 + ssize_t tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf, 5768 + size_t cnt, loff_t *ppos) 5160 5769 { 5161 5770 unsigned long val; 5162 5771 int ret; ··· 5196 5807 5197 5808 return cnt; 5198 5809 } 5199 - 5200 - #ifdef CONFIG_TRACER_MAX_TRACE 5201 - 5202 - static ssize_t 5203 - tracing_max_lat_read(struct file *filp, char __user *ubuf, 5204 - size_t cnt, loff_t *ppos) 5205 - { 5206 - struct trace_array *tr = filp->private_data; 5207 - 5208 - return tracing_nsecs_read(&tr->max_latency, ubuf, cnt, ppos); 5209 - } 5210 - 5211 - static ssize_t 5212 - tracing_max_lat_write(struct file *filp, const char __user *ubuf, 5213 - size_t cnt, loff_t *ppos) 5214 - { 5215 - struct trace_array *tr = filp->private_data; 5216 - 5217 - return tracing_nsecs_write(&tr->max_latency, ubuf, cnt, ppos); 5218 - } 5219 - 5220 - #endif 5221 5810 5222 5811 static int open_pipe_on_cpu(struct trace_array *tr, int cpu) 5223 5812 { ··· 6579 7212 return ring_buffer_event_time_stamp(buffer, rbe); 6580 7213 } 6581 7214 6582 - struct ftrace_buffer_info { 6583 - struct trace_iterator iter; 6584 - void *spare; 6585 - unsigned int spare_cpu; 6586 - unsigned int spare_size; 6587 - unsigned int read; 6588 - }; 6589 - 6590 - #ifdef CONFIG_TRACER_SNAPSHOT 6591 - static int tracing_snapshot_open(struct inode *inode, struct file *file) 6592 - { 6593 - struct trace_array *tr = inode->i_private; 6594 - struct trace_iterator *iter; 6595 - struct seq_file *m; 6596 - int ret; 6597 - 6598 - ret = tracing_check_open_get_tr(tr); 6599 - if (ret) 6600 - return ret; 6601 - 6602 - if (file->f_mode & FMODE_READ) { 6603 - iter = __tracing_open(inode, file, true); 6604 - if (IS_ERR(iter)) 6605 - ret = PTR_ERR(iter); 6606 - } else { 6607 - /* Writes still need the seq_file to hold the private data */ 6608 - ret = -ENOMEM; 6609 - m = kzalloc_obj(*m); 6610 - if (!m) 6611 - goto out; 6612 - iter = kzalloc_obj(*iter); 6613 - if (!iter) { 6614 - kfree(m); 6615 - goto out; 6616 - } 6617 - ret = 0; 6618 - 6619 - iter->tr = tr; 6620 - iter->array_buffer = &tr->snapshot_buffer; 6621 - iter->cpu_file = tracing_get_cpu(inode); 6622 - m->private = iter; 6623 - file->private_data = m; 6624 - } 6625 - out: 6626 - if (ret < 0) 6627 - trace_array_put(tr); 6628 - 6629 - return ret; 6630 - } 6631 - 6632 - static void tracing_swap_cpu_buffer(void *tr) 6633 - { 6634 - update_max_tr_single((struct trace_array *)tr, current, smp_processor_id()); 6635 - } 6636 - 6637 - static ssize_t 6638 - tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt, 6639 - loff_t *ppos) 6640 - { 6641 - struct seq_file *m = filp->private_data; 6642 - struct trace_iterator *iter = m->private; 6643 - struct trace_array *tr = iter->tr; 6644 - unsigned long val; 6645 - int ret; 6646 - 6647 - ret = tracing_update_buffers(tr); 6648 - if (ret < 0) 6649 - return ret; 6650 - 6651 - ret = kstrtoul_from_user(ubuf, cnt, 10, &val); 6652 - if (ret) 6653 - return ret; 6654 - 6655 - guard(mutex)(&trace_types_lock); 6656 - 6657 - if (tracer_uses_snapshot(tr->current_trace)) 6658 - return -EBUSY; 6659 - 6660 - local_irq_disable(); 6661 - arch_spin_lock(&tr->max_lock); 6662 - if (tr->cond_snapshot) 6663 - ret = -EBUSY; 6664 - arch_spin_unlock(&tr->max_lock); 6665 - local_irq_enable(); 6666 - if (ret) 6667 - return ret; 6668 - 6669 - switch (val) { 6670 - case 0: 6671 - if (iter->cpu_file != RING_BUFFER_ALL_CPUS) 6672 - return -EINVAL; 6673 - if (tr->allocated_snapshot) 6674 - free_snapshot(tr); 6675 - break; 6676 - case 1: 6677 - /* Only allow per-cpu swap if the ring buffer supports it */ 6678 - #ifndef CONFIG_RING_BUFFER_ALLOW_SWAP 6679 - if (iter->cpu_file != RING_BUFFER_ALL_CPUS) 6680 - return -EINVAL; 6681 - #endif 6682 - if (tr->allocated_snapshot) 6683 - ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 6684 - &tr->array_buffer, iter->cpu_file); 6685 - 6686 - ret = tracing_arm_snapshot_locked(tr); 6687 - if (ret) 6688 - return ret; 6689 - 6690 - /* Now, we're going to swap */ 6691 - if (iter->cpu_file == RING_BUFFER_ALL_CPUS) { 6692 - local_irq_disable(); 6693 - update_max_tr(tr, current, smp_processor_id(), NULL); 6694 - local_irq_enable(); 6695 - } else { 6696 - smp_call_function_single(iter->cpu_file, tracing_swap_cpu_buffer, 6697 - (void *)tr, 1); 6698 - } 6699 - tracing_disarm_snapshot(tr); 6700 - break; 6701 - default: 6702 - if (tr->allocated_snapshot) { 6703 - if (iter->cpu_file == RING_BUFFER_ALL_CPUS) 6704 - tracing_reset_online_cpus(&tr->snapshot_buffer); 6705 - else 6706 - tracing_reset_cpu(&tr->snapshot_buffer, iter->cpu_file); 6707 - } 6708 - break; 6709 - } 6710 - 6711 - if (ret >= 0) { 6712 - *ppos += cnt; 6713 - ret = cnt; 6714 - } 6715 - 6716 - return ret; 6717 - } 6718 - 6719 - static int tracing_snapshot_release(struct inode *inode, struct file *file) 6720 - { 6721 - struct seq_file *m = file->private_data; 6722 - int ret; 6723 - 6724 - ret = tracing_release(inode, file); 6725 - 6726 - if (file->f_mode & FMODE_READ) 6727 - return ret; 6728 - 6729 - /* If write only, the seq_file is just a stub */ 6730 - if (m) 6731 - kfree(m->private); 6732 - kfree(m); 6733 - 6734 - return 0; 6735 - } 6736 - 6737 - static int tracing_buffers_open(struct inode *inode, struct file *filp); 6738 - static ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf, 6739 - size_t count, loff_t *ppos); 6740 - static int tracing_buffers_release(struct inode *inode, struct file *file); 6741 - static ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos, 6742 - struct pipe_inode_info *pipe, size_t len, unsigned int flags); 6743 - 6744 - static int snapshot_raw_open(struct inode *inode, struct file *filp) 6745 - { 6746 - struct ftrace_buffer_info *info; 6747 - int ret; 6748 - 6749 - /* The following checks for tracefs lockdown */ 6750 - ret = tracing_buffers_open(inode, filp); 6751 - if (ret < 0) 6752 - return ret; 6753 - 6754 - info = filp->private_data; 6755 - 6756 - if (tracer_uses_snapshot(info->iter.trace)) { 6757 - tracing_buffers_release(inode, filp); 6758 - return -EBUSY; 6759 - } 6760 - 6761 - info->iter.snapshot = true; 6762 - info->iter.array_buffer = &info->iter.tr->snapshot_buffer; 6763 - 6764 - return ret; 6765 - } 6766 - 6767 - #endif /* CONFIG_TRACER_SNAPSHOT */ 6768 - 6769 - 6770 7215 static const struct file_operations tracing_thresh_fops = { 6771 7216 .open = tracing_open_generic, 6772 7217 .read = tracing_thresh_read, 6773 7218 .write = tracing_thresh_write, 6774 7219 .llseek = generic_file_llseek, 6775 7220 }; 6776 - 6777 - #ifdef CONFIG_TRACER_MAX_TRACE 6778 - static const struct file_operations tracing_max_lat_fops = { 6779 - .open = tracing_open_generic_tr, 6780 - .read = tracing_max_lat_read, 6781 - .write = tracing_max_lat_write, 6782 - .llseek = generic_file_llseek, 6783 - .release = tracing_release_generic_tr, 6784 - }; 6785 - #endif 6786 7221 6787 7222 static const struct file_operations set_tracer_fops = { 6788 7223 .open = tracing_open_generic_tr, ··· 6671 7502 .llseek = seq_lseek, 6672 7503 .release = tracing_seq_release, 6673 7504 }; 6674 - 6675 - #ifdef CONFIG_TRACER_SNAPSHOT 6676 - static const struct file_operations snapshot_fops = { 6677 - .open = tracing_snapshot_open, 6678 - .read = seq_read, 6679 - .write = tracing_snapshot_write, 6680 - .llseek = tracing_lseek, 6681 - .release = tracing_snapshot_release, 6682 - }; 6683 - 6684 - static const struct file_operations snapshot_raw_fops = { 6685 - .open = snapshot_raw_open, 6686 - .read = tracing_buffers_read, 6687 - .release = tracing_buffers_release, 6688 - .splice_read = tracing_buffers_splice_read, 6689 - }; 6690 - 6691 - #endif /* CONFIG_TRACER_SNAPSHOT */ 6692 7505 6693 7506 /* 6694 7507 * trace_min_max_write - Write a u64 value to a trace_min_max_param struct ··· 7031 7880 .release = tracing_err_log_release, 7032 7881 }; 7033 7882 7034 - static int tracing_buffers_open(struct inode *inode, struct file *filp) 7883 + int tracing_buffers_open(struct inode *inode, struct file *filp) 7035 7884 { 7036 7885 struct trace_array *tr = inode->i_private; 7037 7886 struct ftrace_buffer_info *info; ··· 7079 7928 return trace_poll(iter, filp, poll_table); 7080 7929 } 7081 7930 7082 - static ssize_t 7083 - tracing_buffers_read(struct file *filp, char __user *ubuf, 7084 - size_t count, loff_t *ppos) 7931 + ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf, 7932 + size_t count, loff_t *ppos) 7085 7933 { 7086 7934 struct ftrace_buffer_info *info = filp->private_data; 7087 7935 struct trace_iterator *iter = &info->iter; ··· 7181 8031 return 0; 7182 8032 } 7183 8033 7184 - static int tracing_buffers_release(struct inode *inode, struct file *file) 8034 + int tracing_buffers_release(struct inode *inode, struct file *file) 7185 8035 { 7186 8036 struct ftrace_buffer_info *info = file->private_data; 7187 8037 struct trace_iterator *iter = &info->iter; ··· 7255 8105 spd->partial[i].private = 0; 7256 8106 } 7257 8107 7258 - static ssize_t 7259 - tracing_buffers_splice_read(struct file *file, loff_t *ppos, 7260 - struct pipe_inode_info *pipe, size_t len, 7261 - unsigned int flags) 8108 + ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos, 8109 + struct pipe_inode_info *pipe, size_t len, 8110 + unsigned int flags) 7262 8111 { 7263 8112 struct ftrace_buffer_info *info = file->private_data; 7264 8113 struct trace_iterator *iter = &info->iter; ··· 7410 8261 7411 8262 return 0; 7412 8263 } 7413 - 7414 - #ifdef CONFIG_TRACER_SNAPSHOT 7415 - static int get_snapshot_map(struct trace_array *tr) 7416 - { 7417 - int err = 0; 7418 - 7419 - /* 7420 - * Called with mmap_lock held. lockdep would be unhappy if we would now 7421 - * take trace_types_lock. Instead use the specific 7422 - * snapshot_trigger_lock. 7423 - */ 7424 - spin_lock(&tr->snapshot_trigger_lock); 7425 - 7426 - if (tr->snapshot || tr->mapped == UINT_MAX) 7427 - err = -EBUSY; 7428 - else 7429 - tr->mapped++; 7430 - 7431 - spin_unlock(&tr->snapshot_trigger_lock); 7432 - 7433 - /* Wait for update_max_tr() to observe iter->tr->mapped */ 7434 - if (tr->mapped == 1) 7435 - synchronize_rcu(); 7436 - 7437 - return err; 7438 - 7439 - } 7440 - static void put_snapshot_map(struct trace_array *tr) 7441 - { 7442 - spin_lock(&tr->snapshot_trigger_lock); 7443 - if (!WARN_ON(!tr->mapped)) 7444 - tr->mapped--; 7445 - spin_unlock(&tr->snapshot_trigger_lock); 7446 - } 7447 - #else 7448 - static inline int get_snapshot_map(struct trace_array *tr) { return 0; } 7449 - static inline void put_snapshot_map(struct trace_array *tr) { } 7450 - #endif 7451 8264 7452 8265 /* 7453 8266 * This is called when a VMA is duplicated (e.g., on fork()) to increment ··· 7590 8479 .llseek = generic_file_llseek, 7591 8480 }; 7592 8481 #endif /* CONFIG_DYNAMIC_FTRACE */ 7593 - 7594 - #if defined(CONFIG_TRACER_SNAPSHOT) && defined(CONFIG_DYNAMIC_FTRACE) 7595 - static void 7596 - ftrace_snapshot(unsigned long ip, unsigned long parent_ip, 7597 - struct trace_array *tr, struct ftrace_probe_ops *ops, 7598 - void *data) 7599 - { 7600 - tracing_snapshot_instance(tr); 7601 - } 7602 - 7603 - static void 7604 - ftrace_count_snapshot(unsigned long ip, unsigned long parent_ip, 7605 - struct trace_array *tr, struct ftrace_probe_ops *ops, 7606 - void *data) 7607 - { 7608 - struct ftrace_func_mapper *mapper = data; 7609 - long *count = NULL; 7610 - 7611 - if (mapper) 7612 - count = (long *)ftrace_func_mapper_find_ip(mapper, ip); 7613 - 7614 - if (count) { 7615 - 7616 - if (*count <= 0) 7617 - return; 7618 - 7619 - (*count)--; 7620 - } 7621 - 7622 - tracing_snapshot_instance(tr); 7623 - } 7624 - 7625 - static int 7626 - ftrace_snapshot_print(struct seq_file *m, unsigned long ip, 7627 - struct ftrace_probe_ops *ops, void *data) 7628 - { 7629 - struct ftrace_func_mapper *mapper = data; 7630 - long *count = NULL; 7631 - 7632 - seq_printf(m, "%ps:", (void *)ip); 7633 - 7634 - seq_puts(m, "snapshot"); 7635 - 7636 - if (mapper) 7637 - count = (long *)ftrace_func_mapper_find_ip(mapper, ip); 7638 - 7639 - if (count) 7640 - seq_printf(m, ":count=%ld\n", *count); 7641 - else 7642 - seq_puts(m, ":unlimited\n"); 7643 - 7644 - return 0; 7645 - } 7646 - 7647 - static int 7648 - ftrace_snapshot_init(struct ftrace_probe_ops *ops, struct trace_array *tr, 7649 - unsigned long ip, void *init_data, void **data) 7650 - { 7651 - struct ftrace_func_mapper *mapper = *data; 7652 - 7653 - if (!mapper) { 7654 - mapper = allocate_ftrace_func_mapper(); 7655 - if (!mapper) 7656 - return -ENOMEM; 7657 - *data = mapper; 7658 - } 7659 - 7660 - return ftrace_func_mapper_add_ip(mapper, ip, init_data); 7661 - } 7662 - 7663 - static void 7664 - ftrace_snapshot_free(struct ftrace_probe_ops *ops, struct trace_array *tr, 7665 - unsigned long ip, void *data) 7666 - { 7667 - struct ftrace_func_mapper *mapper = data; 7668 - 7669 - if (!ip) { 7670 - if (!mapper) 7671 - return; 7672 - free_ftrace_func_mapper(mapper, NULL); 7673 - return; 7674 - } 7675 - 7676 - ftrace_func_mapper_remove_ip(mapper, ip); 7677 - } 7678 - 7679 - static struct ftrace_probe_ops snapshot_probe_ops = { 7680 - .func = ftrace_snapshot, 7681 - .print = ftrace_snapshot_print, 7682 - }; 7683 - 7684 - static struct ftrace_probe_ops snapshot_count_probe_ops = { 7685 - .func = ftrace_count_snapshot, 7686 - .print = ftrace_snapshot_print, 7687 - .init = ftrace_snapshot_init, 7688 - .free = ftrace_snapshot_free, 7689 - }; 7690 - 7691 - static int 7692 - ftrace_trace_snapshot_callback(struct trace_array *tr, struct ftrace_hash *hash, 7693 - char *glob, char *cmd, char *param, int enable) 7694 - { 7695 - struct ftrace_probe_ops *ops; 7696 - void *count = (void *)-1; 7697 - char *number; 7698 - int ret; 7699 - 7700 - if (!tr) 7701 - return -ENODEV; 7702 - 7703 - /* hash funcs only work with set_ftrace_filter */ 7704 - if (!enable) 7705 - return -EINVAL; 7706 - 7707 - ops = param ? &snapshot_count_probe_ops : &snapshot_probe_ops; 7708 - 7709 - if (glob[0] == '!') { 7710 - ret = unregister_ftrace_function_probe_func(glob+1, tr, ops); 7711 - if (!ret) 7712 - tracing_disarm_snapshot(tr); 7713 - 7714 - return ret; 7715 - } 7716 - 7717 - if (!param) 7718 - goto out_reg; 7719 - 7720 - number = strsep(&param, ":"); 7721 - 7722 - if (!strlen(number)) 7723 - goto out_reg; 7724 - 7725 - /* 7726 - * We use the callback data field (which is a pointer) 7727 - * as our counter. 7728 - */ 7729 - ret = kstrtoul(number, 0, (unsigned long *)&count); 7730 - if (ret) 7731 - return ret; 7732 - 7733 - out_reg: 7734 - ret = tracing_arm_snapshot(tr); 7735 - if (ret < 0) 7736 - return ret; 7737 - 7738 - ret = register_ftrace_function_probe(glob, tr, ops, count); 7739 - if (ret < 0) 7740 - tracing_disarm_snapshot(tr); 7741 - 7742 - return ret < 0 ? ret : 0; 7743 - } 7744 - 7745 - static struct ftrace_func_command ftrace_snapshot_cmd = { 7746 - .name = "snapshot", 7747 - .func = ftrace_trace_snapshot_callback, 7748 - }; 7749 - 7750 - static __init int register_snapshot_cmd(void) 7751 - { 7752 - return register_ftrace_command(&ftrace_snapshot_cmd); 7753 - } 7754 - #else 7755 - static inline __init int register_snapshot_cmd(void) { return 0; } 7756 - #endif /* defined(CONFIG_TRACER_SNAPSHOT) && defined(CONFIG_DYNAMIC_FTRACE) */ 7757 8482 7758 8483 static struct dentry *tracing_get_dentry(struct trace_array *tr) 7759 8484 { ··· 8383 9436 memset(tscratch, 0, size); 8384 9437 } 8385 9438 8386 - static int 8387 - allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, unsigned long size) 9439 + int allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, int size) 8388 9440 { 8389 9441 enum ring_buffer_flags rb_flags; 8390 9442 struct trace_scratch *tscratch; ··· 8422 9476 } 8423 9477 8424 9478 /* Allocate the first page for all buffers */ 8425 - set_buffer_entries(&tr->array_buffer, 8426 - ring_buffer_size(tr->array_buffer.buffer, 0)); 9479 + trace_set_buffer_entries(&tr->array_buffer, 9480 + ring_buffer_size(tr->array_buffer.buffer, 0)); 8427 9481 8428 9482 return 0; 8429 9483 } ··· 8446 9500 if (ret) 8447 9501 return ret; 8448 9502 8449 - #ifdef CONFIG_TRACER_SNAPSHOT 8450 - /* Fix mapped buffer trace arrays do not have snapshot buffers */ 8451 - if (tr->range_addr_start) 8452 - return 0; 8453 - 8454 - ret = allocate_trace_buffer(tr, &tr->snapshot_buffer, 8455 - allocate_snapshot ? size : 1); 8456 - if (MEM_FAIL(ret, "Failed to allocate trace buffer\n")) { 9503 + ret = trace_allocate_snapshot(tr, size); 9504 + if (MEM_FAIL(ret, "Failed to allocate trace buffer\n")) 8457 9505 free_trace_buffer(&tr->array_buffer); 8458 - return -ENOMEM; 8459 - } 8460 - tr->allocated_snapshot = allocate_snapshot; 8461 9506 8462 - allocate_snapshot = false; 8463 - #endif 8464 - 8465 - return 0; 9507 + return ret; 8466 9508 } 8467 9509 8468 9510 static void free_trace_buffers(struct trace_array *tr) ··· 9572 10638 return done; 9573 10639 } 9574 10640 9575 - #ifdef CONFIG_TRACER_SNAPSHOT 9576 - __init static bool tr_needs_alloc_snapshot(const char *name) 9577 - { 9578 - char *test; 9579 - int len = strlen(name); 9580 - bool ret; 9581 - 9582 - if (!boot_snapshot_index) 9583 - return false; 9584 - 9585 - if (strncmp(name, boot_snapshot_info, len) == 0 && 9586 - boot_snapshot_info[len] == '\t') 9587 - return true; 9588 - 9589 - test = kmalloc(strlen(name) + 3, GFP_KERNEL); 9590 - if (!test) 9591 - return false; 9592 - 9593 - sprintf(test, "\t%s\t", name); 9594 - ret = strstr(boot_snapshot_info, test) == NULL; 9595 - kfree(test); 9596 - return ret; 9597 - } 9598 - 9599 - __init static void do_allocate_snapshot(const char *name) 9600 - { 9601 - if (!tr_needs_alloc_snapshot(name)) 9602 - return; 9603 - 9604 - /* 9605 - * When allocate_snapshot is set, the next call to 9606 - * allocate_trace_buffers() (called by trace_array_get_by_name()) 9607 - * will allocate the snapshot buffer. That will also clear 9608 - * this flag. 9609 - */ 9610 - allocate_snapshot = true; 9611 - } 9612 - #else 9613 - static inline void do_allocate_snapshot(const char *name) { } 9614 - #endif 9615 - 9616 10641 __init static int backup_instance_area(const char *backup, 9617 10642 unsigned long *addr, phys_addr_t *size) 9618 10643 { ··· 9721 10828 } 9722 10829 } else { 9723 10830 /* Only non mapped buffers have snapshot buffers */ 9724 - if (IS_ENABLED(CONFIG_TRACER_SNAPSHOT)) 9725 - do_allocate_snapshot(name); 10831 + do_allocate_snapshot(name); 9726 10832 } 9727 10833 9728 10834 tr = trace_array_create_systems(name, NULL, addr, size); ··· 9936 11044 return &global_trace; 9937 11045 } 9938 11046 #endif 9939 - 9940 - void __init ftrace_boot_snapshot(void) 9941 - { 9942 - #ifdef CONFIG_TRACER_SNAPSHOT 9943 - struct trace_array *tr; 9944 - 9945 - if (!snapshot_at_boot) 9946 - return; 9947 - 9948 - list_for_each_entry(tr, &ftrace_trace_arrays, list) { 9949 - if (!tr->allocated_snapshot) 9950 - continue; 9951 - 9952 - tracing_snapshot_instance(tr); 9953 - trace_array_puts(tr, "** Boot snapshot taken **\n"); 9954 - } 9955 - #endif 9956 - } 9957 11047 9958 11048 void __init early_trace_init(void) 9959 11049 {
+111 -16
kernel/trace/trace.h
··· 264 264 265 265 typedef bool (*cond_update_fn_t)(struct trace_array *tr, void *cond_data); 266 266 267 + #ifdef CONFIG_TRACER_SNAPSHOT 267 268 /** 268 269 * struct cond_snapshot - conditional snapshot data and callback 269 270 * ··· 307 306 void *cond_data; 308 307 cond_update_fn_t update; 309 308 }; 309 + #endif /* CONFIG_TRACER_SNAPSHOT */ 310 310 311 311 /* 312 312 * struct trace_func_repeats - used to keep track of the consecutive ··· 693 691 void tracing_reset_all_online_cpus_unlocked(void); 694 692 int tracing_open_generic(struct inode *inode, struct file *filp); 695 693 int tracing_open_generic_tr(struct inode *inode, struct file *filp); 694 + int tracing_release(struct inode *inode, struct file *file); 696 695 int tracing_release_generic_tr(struct inode *inode, struct file *file); 697 696 int tracing_open_file_tr(struct inode *inode, struct file *filp); 698 697 int tracing_release_file_tr(struct inode *inode, struct file *filp); ··· 703 700 void tracer_tracing_off(struct trace_array *tr); 704 701 void tracer_tracing_disable(struct trace_array *tr); 705 702 void tracer_tracing_enable(struct trace_array *tr); 703 + int allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, int size); 706 704 struct dentry *trace_create_file(const char *name, 707 705 umode_t mode, 708 706 struct dentry *parent, ··· 715 711 void *data, 716 712 long cpu, 717 713 const struct file_operations *fops); 718 - int tracing_get_cpu(struct inode *inode); 719 714 715 + struct trace_iterator *__tracing_open(struct inode *inode, struct file *file, 716 + bool snapshot); 717 + int tracing_buffers_open(struct inode *inode, struct file *filp); 718 + ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf, 719 + size_t count, loff_t *ppos); 720 + int tracing_buffers_release(struct inode *inode, struct file *file); 721 + ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos, 722 + struct pipe_inode_info *pipe, size_t len, unsigned int flags); 723 + 724 + ssize_t tracing_nsecs_read(unsigned long *ptr, char __user *ubuf, 725 + size_t cnt, loff_t *ppos); 726 + ssize_t tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf, 727 + size_t cnt, loff_t *ppos); 728 + 729 + void trace_set_buffer_entries(struct array_buffer *buf, unsigned long val); 730 + 731 + /* 732 + * Should be used after trace_array_get(), trace_types_lock 733 + * ensures that i_cdev was already initialized. 734 + */ 735 + static inline int tracing_get_cpu(struct inode *inode) 736 + { 737 + if (inode->i_cdev) /* See trace_create_cpu_file() */ 738 + return (long)inode->i_cdev - 1; 739 + return RING_BUFFER_ALL_CPUS; 740 + } 741 + void tracing_reset_cpu(struct array_buffer *buf, int cpu); 742 + 743 + struct ftrace_buffer_info { 744 + struct trace_iterator iter; 745 + void *spare; 746 + unsigned int spare_cpu; 747 + unsigned int spare_size; 748 + unsigned int read; 749 + }; 720 750 721 751 /** 722 752 * tracer_tracing_is_on_cpu - show real state of ring buffer enabled on for a cpu ··· 867 829 #if defined(CONFIG_TRACER_MAX_TRACE) && defined(CONFIG_FSNOTIFY) 868 830 # define LATENCY_FS_NOTIFY 869 831 #endif 832 + #endif /* CONFIG_TRACER_SNAPSHOT */ 870 833 871 834 #ifdef LATENCY_FS_NOTIFY 872 835 void latency_fsnotify(struct trace_array *tr); 873 836 #else 874 837 static inline void latency_fsnotify(struct trace_array *tr) { } 875 838 #endif 876 - #endif /* CONFIG_TRACER_SNAPSHOT */ 877 839 878 840 #ifdef CONFIG_STACKTRACE 879 841 void __trace_stack(struct trace_array *tr, unsigned int trace_ctx, int skip); ··· 889 851 { 890 852 return tracer->use_max_tr; 891 853 } 854 + void trace_create_maxlat_file(struct trace_array *tr, 855 + struct dentry *d_tracer); 892 856 #else 893 857 static inline bool tracer_uses_snapshot(struct tracer *tracer) 894 858 { 895 859 return false; 896 860 } 861 + static inline void trace_create_maxlat_file(struct trace_array *tr, 862 + struct dentry *d_tracer) { } 897 863 #endif 898 864 899 865 void trace_last_func_repeats(struct trace_array *tr, ··· 927 885 #define DYN_FTRACE_TEST_NAME2 trace_selftest_dynamic_test_func2 928 886 extern int DYN_FTRACE_TEST_NAME2(void); 929 887 888 + void __init trace_append_boot_param(char *buf, const char *str, 889 + char sep, int size); 930 890 extern void trace_set_ring_buffer_expanded(struct trace_array *tr); 931 891 extern bool tracing_selftest_disabled; 932 892 ··· 1869 1825 const char *system, 1870 1826 const char *event); 1871 1827 1872 - static inline void *event_file_data(struct file *filp) 1873 - { 1874 - return READ_ONCE(file_inode(filp)->i_private); 1875 - } 1876 - 1877 1828 extern struct mutex event_mutex; 1878 1829 extern struct list_head ftrace_events; 1879 1830 ··· 1889 1850 struct trace_event_file *file; 1890 1851 1891 1852 lockdep_assert_held(&event_mutex); 1892 - file = READ_ONCE(file_inode(filp)->i_private); 1853 + file = file_inode(filp)->i_private; 1893 1854 if (!file || file->flags & EVENT_FILE_FL_FREED) 1894 1855 return NULL; 1856 + return file; 1857 + } 1858 + 1859 + static inline void *event_file_data(struct file *filp) 1860 + { 1861 + struct trace_event_file *file; 1862 + 1863 + lockdep_assert_held(&event_mutex); 1864 + file = file_inode(filp)->i_private; 1865 + WARN_ON(!file || file->flags & EVENT_FILE_FL_FREED); 1895 1866 return file; 1896 1867 } 1897 1868 ··· 2207 2158 2208 2159 extern int trace_event_enable_disable(struct trace_event_file *file, 2209 2160 int enable, int soft_disable); 2210 - extern int tracing_alloc_snapshot(void); 2211 - extern void tracing_snapshot_cond(struct trace_array *tr, void *cond_data); 2212 - extern int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update); 2213 - 2214 - extern int tracing_snapshot_cond_disable(struct trace_array *tr); 2215 - extern void *tracing_cond_snapshot_data(struct trace_array *tr); 2216 2161 2217 2162 extern const char *__start___trace_bprintk_fmt[]; 2218 2163 extern const char *__stop___trace_bprintk_fmt[]; ··· 2294 2251 #endif 2295 2252 2296 2253 #ifdef CONFIG_TRACER_SNAPSHOT 2254 + extern const struct file_operations snapshot_fops; 2255 + extern const struct file_operations snapshot_raw_fops; 2256 + 2257 + /* Used when creating instances */ 2258 + int trace_allocate_snapshot(struct trace_array *tr, int size); 2259 + 2260 + int tracing_alloc_snapshot(void); 2261 + void tracing_snapshot_cond(struct trace_array *tr, void *cond_data); 2262 + int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update); 2263 + int tracing_snapshot_cond_disable(struct trace_array *tr); 2264 + void *tracing_cond_snapshot_data(struct trace_array *tr); 2297 2265 void tracing_snapshot_instance(struct trace_array *tr); 2298 2266 int tracing_alloc_snapshot_instance(struct trace_array *tr); 2267 + int tracing_arm_snapshot_locked(struct trace_array *tr); 2299 2268 int tracing_arm_snapshot(struct trace_array *tr); 2300 2269 void tracing_disarm_snapshot(struct trace_array *tr); 2301 - #else 2270 + void free_snapshot(struct trace_array *tr); 2271 + void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter); 2272 + int get_snapshot_map(struct trace_array *tr); 2273 + void put_snapshot_map(struct trace_array *tr); 2274 + int resize_buffer_duplicate_size(struct array_buffer *trace_buf, 2275 + struct array_buffer *size_buf, int cpu_id); 2276 + __init void do_allocate_snapshot(const char *name); 2277 + # ifdef CONFIG_DYNAMIC_FTRACE 2278 + __init int register_snapshot_cmd(void); 2279 + # else 2280 + static inline int register_snapshot_cmd(void) { return 0; } 2281 + # endif 2282 + #else /* !CONFIG_TRACER_SNAPSHOT */ 2283 + static inline int trace_allocate_snapshot(struct trace_array *tr, int size) { return 0; } 2302 2284 static inline void tracing_snapshot_instance(struct trace_array *tr) { } 2303 2285 static inline int tracing_alloc_snapshot_instance(struct trace_array *tr) 2304 2286 { 2305 2287 return 0; 2306 2288 } 2289 + static inline int tracing_arm_snapshot_locked(struct trace_array *tr) { return -EBUSY; } 2307 2290 static inline int tracing_arm_snapshot(struct trace_array *tr) { return 0; } 2308 2291 static inline void tracing_disarm_snapshot(struct trace_array *tr) { } 2309 - #endif 2292 + static inline void free_snapshot(struct trace_array *tr) {} 2293 + static inline void tracing_snapshot_cond(struct trace_array *tr, void *cond_data) 2294 + { 2295 + WARN_ONCE(1, "Snapshot feature not enabled, but internal conditional snapshot used"); 2296 + } 2297 + static inline void *tracing_cond_snapshot_data(struct trace_array *tr) 2298 + { 2299 + return NULL; 2300 + } 2301 + static inline int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update) 2302 + { 2303 + return -ENODEV; 2304 + } 2305 + static inline int tracing_snapshot_cond_disable(struct trace_array *tr) 2306 + { 2307 + return false; 2308 + } 2309 + static inline void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter) 2310 + { 2311 + /* Should never be called */ 2312 + WARN_ONCE(1, "Snapshot print function called without snapshot configured"); 2313 + } 2314 + static inline int get_snapshot_map(struct trace_array *tr) { return 0; } 2315 + static inline void put_snapshot_map(struct trace_array *tr) { } 2316 + static inline void do_allocate_snapshot(const char *name) { } 2317 + static inline int register_snapshot_cmd(void) { return 0; } 2318 + #endif /* CONFIG_TRACER_SNAPSHOT */ 2310 2319 2311 2320 #ifdef CONFIG_PREEMPT_TRACER 2312 2321 void tracer_preempt_on(unsigned long a0, unsigned long a1);
+17 -14
kernel/trace/trace_events.c
··· 1721 1721 1722 1722 len = get_call_len(call); 1723 1723 1724 - seq_printf(m, "%s:%s%*.s%s\n", call->class->system, 1724 + seq_printf(m, "%s:%s%*s%s\n", call->class->system, 1725 1725 trace_event_name(call), len, "", filter->filter_string); 1726 1726 1727 1727 return 0; ··· 1753 1753 len = get_call_len(call); 1754 1754 1755 1755 list_for_each_entry_rcu(data, &file->triggers, list) { 1756 - seq_printf(m, "%s:%s%*.s", call->class->system, 1756 + seq_printf(m, "%s:%s%*s", call->class->system, 1757 1757 trace_event_name(call), len, ""); 1758 1758 1759 1759 data->cmd_ops->print(m, data); ··· 2187 2187 static ssize_t 2188 2188 event_id_read(struct file *filp, char __user *ubuf, size_t cnt, loff_t *ppos) 2189 2189 { 2190 - int id = (long)event_file_data(filp); 2190 + /* id is directly in i_private and available for inode's lifetime. */ 2191 + int id = (long)file_inode(filp)->i_private; 2191 2192 char buf[32]; 2192 2193 int len; 2193 2194 2194 - if (unlikely(!id)) 2195 - return -ENODEV; 2195 + WARN_ON(!id); 2196 2196 2197 2197 len = sprintf(buf, "%d\n", id); 2198 2198 ··· 2250 2250 2251 2251 mutex_lock(&event_mutex); 2252 2252 file = event_file_file(filp); 2253 - if (file) { 2254 - if (file->flags & EVENT_FILE_FL_FREED) 2255 - err = -ENODEV; 2256 - else 2257 - err = apply_event_filter(file, buf); 2258 - } 2253 + if (file) 2254 + err = apply_event_filter(file, buf); 2259 2255 mutex_unlock(&event_mutex); 2260 2256 2261 2257 kfree(buf); ··· 3683 3687 } bootup_triggers[MAX_BOOT_TRIGGERS]; 3684 3688 3685 3689 static char bootup_trigger_buf[COMMAND_LINE_SIZE]; 3690 + static int boot_trigger_buf_len; 3686 3691 static int nr_boot_triggers; 3687 3692 3688 3693 static __init int setup_trace_triggers(char *str) 3689 3694 { 3690 3695 char *trigger; 3691 3696 char *buf; 3697 + int len = boot_trigger_buf_len; 3692 3698 int i; 3693 3699 3694 - strscpy(bootup_trigger_buf, str, COMMAND_LINE_SIZE); 3700 + if (len >= COMMAND_LINE_SIZE) 3701 + return 1; 3702 + 3703 + strscpy(bootup_trigger_buf + len, str, COMMAND_LINE_SIZE - len); 3695 3704 trace_set_ring_buffer_expanded(NULL); 3696 3705 disable_tracing_selftest("running event triggers"); 3697 3706 3698 - buf = bootup_trigger_buf; 3699 - for (i = 0; i < MAX_BOOT_TRIGGERS; i++) { 3707 + buf = bootup_trigger_buf + len; 3708 + boot_trigger_buf_len += strlen(buf) + 1; 3709 + 3710 + for (i = nr_boot_triggers; i < MAX_BOOT_TRIGGERS; i++) { 3700 3711 trigger = strsep(&buf, ","); 3701 3712 if (!trigger) 3702 3713 break;
+16 -13
kernel/trace/trace_events_hist.c
··· 1361 1361 field->flags & HIST_FIELD_FL_VAR_REF) { 1362 1362 if (field->system) { 1363 1363 static char full_name[MAX_FILTER_STR_VAL]; 1364 + static char *fmt; 1365 + int len; 1364 1366 1365 - strcat(full_name, field->system); 1366 - strcat(full_name, "."); 1367 - strcat(full_name, field->event_name); 1368 - strcat(full_name, "."); 1369 - strcat(full_name, field->name); 1367 + fmt = field->flags & HIST_FIELD_FL_VAR_REF ? "%s.%s.$%s" : "%s.%s.%s"; 1368 + 1369 + len = snprintf(full_name, sizeof(full_name), fmt, 1370 + field->system, field->event_name, 1371 + field->name); 1372 + if (len >= sizeof(full_name)) 1373 + return NULL; 1374 + 1370 1375 field_name = full_name; 1371 1376 } else 1372 1377 field_name = field->name; ··· 1745 1740 1746 1741 static void expr_field_str(struct hist_field *field, char *expr) 1747 1742 { 1748 - if (field->flags & HIST_FIELD_FL_VAR_REF) 1749 - strcat(expr, "$"); 1750 - else if (field->flags & HIST_FIELD_FL_CONST) { 1743 + if (field->flags & HIST_FIELD_FL_VAR_REF) { 1744 + if (!field->system) 1745 + strcat(expr, "$"); 1746 + } else if (field->flags & HIST_FIELD_FL_CONST) { 1751 1747 char str[HIST_CONST_DIGITS_MAX]; 1752 1748 1753 1749 snprintf(str, HIST_CONST_DIGITS_MAX, "%llu", field->constant); ··· 5842 5836 hist_file->file = file; 5843 5837 hist_file->last_act = get_hist_hit_count(event_file); 5844 5838 5845 - /* Clear private_data to avoid warning in single_open() */ 5846 - file->private_data = NULL; 5847 5839 ret = single_open(file, hist_show, hist_file); 5848 5840 if (ret) { 5849 5841 kfree(hist_file); ··· 6130 6126 if (ret) 6131 6127 return ret; 6132 6128 6133 - /* Clear private_data to avoid warning in single_open() */ 6134 - file->private_data = NULL; 6135 6129 ret = single_open(file, hist_debug_show, file); 6136 6130 if (ret) 6137 6131 tracing_release_file_tr(inode, file); ··· 6160 6158 else if (field_name) { 6161 6159 if (hist_field->flags & HIST_FIELD_FL_VAR_REF || 6162 6160 hist_field->flags & HIST_FIELD_FL_ALIAS) 6163 - seq_putc(m, '$'); 6161 + if (!hist_field->system) 6162 + seq_putc(m, '$'); 6164 6163 seq_printf(m, "%s", field_name); 6165 6164 } else if (hist_field->flags & HIST_FIELD_FL_TIMESTAMP) 6166 6165 seq_puts(m, "common_timestamp");
+2 -1
kernel/trace/trace_kprobe.c
··· 31 31 32 32 static int __init set_kprobe_boot_events(char *str) 33 33 { 34 - strscpy(kprobe_boot_events_buf, str, COMMAND_LINE_SIZE); 34 + trace_append_boot_param(kprobe_boot_events_buf, str, ';', 35 + COMMAND_LINE_SIZE); 35 36 disable_tracing_selftest("running kprobe events"); 36 37 37 38 return 1;
+11 -1
kernel/trace/trace_output.c
··· 723 723 { 724 724 const struct btf_param *param; 725 725 const struct btf_type *t; 726 + const struct btf_enum *enums; 726 727 const char *param_name; 727 728 char name[KSYM_NAME_LEN]; 728 729 unsigned long arg; 729 730 struct btf *btf; 730 731 s32 tid, nr = 0; 731 - int a, p, x; 732 + int a, p, x, i; 732 733 u16 encode; 733 734 734 735 trace_seq_printf(s, "("); ··· 783 782 break; 784 783 case BTF_KIND_ENUM: 785 784 trace_seq_printf(s, "%ld", arg); 785 + enums = btf_enum(t); 786 + for (i = 0; i < btf_vlen(t); i++) { 787 + if (arg == enums[i].val) { 788 + trace_seq_printf(s, " [%s]", 789 + btf_name_by_offset(btf, 790 + enums[i].name_off)); 791 + break; 792 + } 793 + } 786 794 break; 787 795 default: 788 796 /* This does not handle complex arguments */
+1
kernel/trace/trace_printk.c
··· 197 197 .notifier_call = module_trace_bprintk_format_notify, 198 198 }; 199 199 200 + __printf(2, 3) 200 201 int __trace_bprintk(unsigned long ip, const char *fmt, ...) 201 202 { 202 203 int ret;
+1066
kernel/trace/trace_snapshot.c
··· 1 + // SPDX-License-Identifier: GPL-2.0 2 + #include <linux/fsnotify.h> 3 + 4 + #include <asm/setup.h> /* COMMAND_LINE_SIZE */ 5 + 6 + #include "trace.h" 7 + 8 + /* Used if snapshot allocated at boot */ 9 + static bool allocate_snapshot; 10 + static bool snapshot_at_boot; 11 + 12 + static char boot_snapshot_info[COMMAND_LINE_SIZE] __initdata; 13 + static int boot_snapshot_index; 14 + 15 + static int __init boot_alloc_snapshot(char *str) 16 + { 17 + char *slot = boot_snapshot_info + boot_snapshot_index; 18 + int left = sizeof(boot_snapshot_info) - boot_snapshot_index; 19 + int ret; 20 + 21 + if (str[0] == '=') { 22 + str++; 23 + if (strlen(str) >= left) 24 + return -1; 25 + 26 + ret = snprintf(slot, left, "%s\t", str); 27 + boot_snapshot_index += ret; 28 + } else { 29 + allocate_snapshot = true; 30 + /* We also need the main ring buffer expanded */ 31 + trace_set_ring_buffer_expanded(NULL); 32 + } 33 + return 1; 34 + } 35 + __setup("alloc_snapshot", boot_alloc_snapshot); 36 + 37 + 38 + static int __init boot_snapshot(char *str) 39 + { 40 + snapshot_at_boot = true; 41 + boot_alloc_snapshot(str); 42 + return 1; 43 + } 44 + __setup("ftrace_boot_snapshot", boot_snapshot); 45 + static void tracing_snapshot_instance_cond(struct trace_array *tr, 46 + void *cond_data) 47 + { 48 + unsigned long flags; 49 + 50 + if (in_nmi()) { 51 + trace_array_puts(tr, "*** SNAPSHOT CALLED FROM NMI CONTEXT ***\n"); 52 + trace_array_puts(tr, "*** snapshot is being ignored ***\n"); 53 + return; 54 + } 55 + 56 + if (!tr->allocated_snapshot) { 57 + trace_array_puts(tr, "*** SNAPSHOT NOT ALLOCATED ***\n"); 58 + trace_array_puts(tr, "*** stopping trace here! ***\n"); 59 + tracer_tracing_off(tr); 60 + return; 61 + } 62 + 63 + if (tr->mapped) { 64 + trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n"); 65 + trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 66 + return; 67 + } 68 + 69 + /* Note, snapshot can not be used when the tracer uses it */ 70 + if (tracer_uses_snapshot(tr->current_trace)) { 71 + trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n"); 72 + trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n"); 73 + return; 74 + } 75 + 76 + local_irq_save(flags); 77 + update_max_tr(tr, current, smp_processor_id(), cond_data); 78 + local_irq_restore(flags); 79 + } 80 + 81 + void tracing_snapshot_instance(struct trace_array *tr) 82 + { 83 + tracing_snapshot_instance_cond(tr, NULL); 84 + } 85 + 86 + /** 87 + * tracing_snapshot_cond - conditionally take a snapshot of the current buffer. 88 + * @tr: The tracing instance to snapshot 89 + * @cond_data: The data to be tested conditionally, and possibly saved 90 + * 91 + * This is the same as tracing_snapshot() except that the snapshot is 92 + * conditional - the snapshot will only happen if the 93 + * cond_snapshot.update() implementation receiving the cond_data 94 + * returns true, which means that the trace array's cond_snapshot 95 + * update() operation used the cond_data to determine whether the 96 + * snapshot should be taken, and if it was, presumably saved it along 97 + * with the snapshot. 98 + */ 99 + void tracing_snapshot_cond(struct trace_array *tr, void *cond_data) 100 + { 101 + tracing_snapshot_instance_cond(tr, cond_data); 102 + } 103 + EXPORT_SYMBOL_GPL(tracing_snapshot_cond); 104 + 105 + /** 106 + * tracing_cond_snapshot_data - get the user data associated with a snapshot 107 + * @tr: The tracing instance 108 + * 109 + * When the user enables a conditional snapshot using 110 + * tracing_snapshot_cond_enable(), the user-defined cond_data is saved 111 + * with the snapshot. This accessor is used to retrieve it. 112 + * 113 + * Should not be called from cond_snapshot.update(), since it takes 114 + * the tr->max_lock lock, which the code calling 115 + * cond_snapshot.update() has already done. 116 + * 117 + * Returns the cond_data associated with the trace array's snapshot. 118 + */ 119 + void *tracing_cond_snapshot_data(struct trace_array *tr) 120 + { 121 + void *cond_data = NULL; 122 + 123 + local_irq_disable(); 124 + arch_spin_lock(&tr->max_lock); 125 + 126 + if (tr->cond_snapshot) 127 + cond_data = tr->cond_snapshot->cond_data; 128 + 129 + arch_spin_unlock(&tr->max_lock); 130 + local_irq_enable(); 131 + 132 + return cond_data; 133 + } 134 + EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data); 135 + 136 + /* resize @tr's buffer to the size of @size_tr's entries */ 137 + int resize_buffer_duplicate_size(struct array_buffer *trace_buf, 138 + struct array_buffer *size_buf, int cpu_id) 139 + { 140 + int cpu, ret = 0; 141 + 142 + if (cpu_id == RING_BUFFER_ALL_CPUS) { 143 + for_each_tracing_cpu(cpu) { 144 + ret = ring_buffer_resize(trace_buf->buffer, 145 + per_cpu_ptr(size_buf->data, cpu)->entries, cpu); 146 + if (ret < 0) 147 + break; 148 + per_cpu_ptr(trace_buf->data, cpu)->entries = 149 + per_cpu_ptr(size_buf->data, cpu)->entries; 150 + } 151 + } else { 152 + ret = ring_buffer_resize(trace_buf->buffer, 153 + per_cpu_ptr(size_buf->data, cpu_id)->entries, cpu_id); 154 + if (ret == 0) 155 + per_cpu_ptr(trace_buf->data, cpu_id)->entries = 156 + per_cpu_ptr(size_buf->data, cpu_id)->entries; 157 + } 158 + 159 + return ret; 160 + } 161 + 162 + int tracing_alloc_snapshot_instance(struct trace_array *tr) 163 + { 164 + int order; 165 + int ret; 166 + 167 + if (!tr->allocated_snapshot) { 168 + 169 + /* Make the snapshot buffer have the same order as main buffer */ 170 + order = ring_buffer_subbuf_order_get(tr->array_buffer.buffer); 171 + ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order); 172 + if (ret < 0) 173 + return ret; 174 + 175 + /* allocate spare buffer */ 176 + ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 177 + &tr->array_buffer, RING_BUFFER_ALL_CPUS); 178 + if (ret < 0) 179 + return ret; 180 + 181 + tr->allocated_snapshot = true; 182 + } 183 + 184 + return 0; 185 + } 186 + 187 + void free_snapshot(struct trace_array *tr) 188 + { 189 + /* 190 + * We don't free the ring buffer. instead, resize it because 191 + * The max_tr ring buffer has some state (e.g. ring->clock) and 192 + * we want preserve it. 193 + */ 194 + ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, 0); 195 + ring_buffer_resize(tr->snapshot_buffer.buffer, 1, RING_BUFFER_ALL_CPUS); 196 + trace_set_buffer_entries(&tr->snapshot_buffer, 1); 197 + tracing_reset_online_cpus(&tr->snapshot_buffer); 198 + tr->allocated_snapshot = false; 199 + } 200 + 201 + int tracing_arm_snapshot_locked(struct trace_array *tr) 202 + { 203 + int ret; 204 + 205 + lockdep_assert_held(&trace_types_lock); 206 + 207 + spin_lock(&tr->snapshot_trigger_lock); 208 + if (tr->snapshot == UINT_MAX || tr->mapped) { 209 + spin_unlock(&tr->snapshot_trigger_lock); 210 + return -EBUSY; 211 + } 212 + 213 + tr->snapshot++; 214 + spin_unlock(&tr->snapshot_trigger_lock); 215 + 216 + ret = tracing_alloc_snapshot_instance(tr); 217 + if (ret) { 218 + spin_lock(&tr->snapshot_trigger_lock); 219 + tr->snapshot--; 220 + spin_unlock(&tr->snapshot_trigger_lock); 221 + } 222 + 223 + return ret; 224 + } 225 + 226 + int tracing_arm_snapshot(struct trace_array *tr) 227 + { 228 + guard(mutex)(&trace_types_lock); 229 + return tracing_arm_snapshot_locked(tr); 230 + } 231 + 232 + void tracing_disarm_snapshot(struct trace_array *tr) 233 + { 234 + spin_lock(&tr->snapshot_trigger_lock); 235 + if (!WARN_ON(!tr->snapshot)) 236 + tr->snapshot--; 237 + spin_unlock(&tr->snapshot_trigger_lock); 238 + } 239 + 240 + /** 241 + * tracing_snapshot_alloc - allocate and take a snapshot of the current buffer. 242 + * 243 + * This is similar to tracing_snapshot(), but it will allocate the 244 + * snapshot buffer if it isn't already allocated. Use this only 245 + * where it is safe to sleep, as the allocation may sleep. 246 + * 247 + * This causes a swap between the snapshot buffer and the current live 248 + * tracing buffer. You can use this to take snapshots of the live 249 + * trace when some condition is triggered, but continue to trace. 250 + */ 251 + void tracing_snapshot_alloc(void) 252 + { 253 + int ret; 254 + 255 + ret = tracing_alloc_snapshot(); 256 + if (ret < 0) 257 + return; 258 + 259 + tracing_snapshot(); 260 + } 261 + EXPORT_SYMBOL_GPL(tracing_snapshot_alloc); 262 + 263 + /** 264 + * tracing_snapshot_cond_enable - enable conditional snapshot for an instance 265 + * @tr: The tracing instance 266 + * @cond_data: User data to associate with the snapshot 267 + * @update: Implementation of the cond_snapshot update function 268 + * 269 + * Check whether the conditional snapshot for the given instance has 270 + * already been enabled, or if the current tracer is already using a 271 + * snapshot; if so, return -EBUSY, else create a cond_snapshot and 272 + * save the cond_data and update function inside. 273 + * 274 + * Returns 0 if successful, error otherwise. 275 + */ 276 + int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, 277 + cond_update_fn_t update) 278 + { 279 + struct cond_snapshot *cond_snapshot __free(kfree) = 280 + kzalloc_obj(*cond_snapshot); 281 + int ret; 282 + 283 + if (!cond_snapshot) 284 + return -ENOMEM; 285 + 286 + cond_snapshot->cond_data = cond_data; 287 + cond_snapshot->update = update; 288 + 289 + guard(mutex)(&trace_types_lock); 290 + 291 + if (tracer_uses_snapshot(tr->current_trace)) 292 + return -EBUSY; 293 + 294 + /* 295 + * The cond_snapshot can only change to NULL without the 296 + * trace_types_lock. We don't care if we race with it going 297 + * to NULL, but we want to make sure that it's not set to 298 + * something other than NULL when we get here, which we can 299 + * do safely with only holding the trace_types_lock and not 300 + * having to take the max_lock. 301 + */ 302 + if (tr->cond_snapshot) 303 + return -EBUSY; 304 + 305 + ret = tracing_arm_snapshot_locked(tr); 306 + if (ret) 307 + return ret; 308 + 309 + local_irq_disable(); 310 + arch_spin_lock(&tr->max_lock); 311 + tr->cond_snapshot = no_free_ptr(cond_snapshot); 312 + arch_spin_unlock(&tr->max_lock); 313 + local_irq_enable(); 314 + 315 + return 0; 316 + } 317 + EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable); 318 + 319 + /** 320 + * tracing_snapshot_cond_disable - disable conditional snapshot for an instance 321 + * @tr: The tracing instance 322 + * 323 + * Check whether the conditional snapshot for the given instance is 324 + * enabled; if so, free the cond_snapshot associated with it, 325 + * otherwise return -EINVAL. 326 + * 327 + * Returns 0 if successful, error otherwise. 328 + */ 329 + int tracing_snapshot_cond_disable(struct trace_array *tr) 330 + { 331 + int ret = 0; 332 + 333 + local_irq_disable(); 334 + arch_spin_lock(&tr->max_lock); 335 + 336 + if (!tr->cond_snapshot) 337 + ret = -EINVAL; 338 + else { 339 + kfree(tr->cond_snapshot); 340 + tr->cond_snapshot = NULL; 341 + } 342 + 343 + arch_spin_unlock(&tr->max_lock); 344 + local_irq_enable(); 345 + 346 + tracing_disarm_snapshot(tr); 347 + 348 + return ret; 349 + } 350 + EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable); 351 + 352 + #ifdef CONFIG_TRACER_MAX_TRACE 353 + #ifdef LATENCY_FS_NOTIFY 354 + static struct workqueue_struct *fsnotify_wq; 355 + 356 + static void latency_fsnotify_workfn(struct work_struct *work) 357 + { 358 + struct trace_array *tr = container_of(work, struct trace_array, 359 + fsnotify_work); 360 + fsnotify_inode(tr->d_max_latency->d_inode, FS_MODIFY); 361 + } 362 + 363 + static void latency_fsnotify_workfn_irq(struct irq_work *iwork) 364 + { 365 + struct trace_array *tr = container_of(iwork, struct trace_array, 366 + fsnotify_irqwork); 367 + queue_work(fsnotify_wq, &tr->fsnotify_work); 368 + } 369 + 370 + __init static int latency_fsnotify_init(void) 371 + { 372 + fsnotify_wq = alloc_workqueue("tr_max_lat_wq", 373 + WQ_UNBOUND | WQ_HIGHPRI, 0); 374 + if (!fsnotify_wq) { 375 + pr_err("Unable to allocate tr_max_lat_wq\n"); 376 + return -ENOMEM; 377 + } 378 + return 0; 379 + } 380 + 381 + late_initcall_sync(latency_fsnotify_init); 382 + 383 + void latency_fsnotify(struct trace_array *tr) 384 + { 385 + if (!fsnotify_wq) 386 + return; 387 + /* 388 + * We cannot call queue_work(&tr->fsnotify_work) from here because it's 389 + * possible that we are called from __schedule() or do_idle(), which 390 + * could cause a deadlock. 391 + */ 392 + irq_work_queue(&tr->fsnotify_irqwork); 393 + } 394 + #endif /* LATENCY_FS_NOTIFY */ 395 + 396 + static const struct file_operations tracing_max_lat_fops; 397 + 398 + void trace_create_maxlat_file(struct trace_array *tr, 399 + struct dentry *d_tracer) 400 + { 401 + #ifdef LATENCY_FS_NOTIFY 402 + INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); 403 + init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); 404 + #endif 405 + tr->d_max_latency = trace_create_file("tracing_max_latency", 406 + TRACE_MODE_WRITE, 407 + d_tracer, tr, 408 + &tracing_max_lat_fops); 409 + } 410 + 411 + /* 412 + * Copy the new maximum trace into the separate maximum-trace 413 + * structure. (this way the maximum trace is permanently saved, 414 + * for later retrieval via /sys/kernel/tracing/tracing_max_latency) 415 + */ 416 + static void 417 + __update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu) 418 + { 419 + struct array_buffer *trace_buf = &tr->array_buffer; 420 + struct trace_array_cpu *data = per_cpu_ptr(trace_buf->data, cpu); 421 + struct array_buffer *max_buf = &tr->snapshot_buffer; 422 + struct trace_array_cpu *max_data = per_cpu_ptr(max_buf->data, cpu); 423 + 424 + max_buf->cpu = cpu; 425 + max_buf->time_start = data->preempt_timestamp; 426 + 427 + max_data->saved_latency = tr->max_latency; 428 + max_data->critical_start = data->critical_start; 429 + max_data->critical_end = data->critical_end; 430 + 431 + strscpy(max_data->comm, tsk->comm); 432 + max_data->pid = tsk->pid; 433 + /* 434 + * If tsk == current, then use current_uid(), as that does not use 435 + * RCU. The irq tracer can be called out of RCU scope. 436 + */ 437 + if (tsk == current) 438 + max_data->uid = current_uid(); 439 + else 440 + max_data->uid = task_uid(tsk); 441 + 442 + max_data->nice = tsk->static_prio - 20 - MAX_RT_PRIO; 443 + max_data->policy = tsk->policy; 444 + max_data->rt_priority = tsk->rt_priority; 445 + 446 + /* record this tasks comm */ 447 + tracing_record_cmdline(tsk); 448 + latency_fsnotify(tr); 449 + } 450 + #else 451 + static inline void __update_max_tr(struct trace_array *tr, 452 + struct task_struct *tsk, int cpu) { } 453 + #endif /* CONFIG_TRACER_MAX_TRACE */ 454 + 455 + /** 456 + * update_max_tr - snapshot all trace buffers from global_trace to max_tr 457 + * @tr: tracer 458 + * @tsk: the task with the latency 459 + * @cpu: The cpu that initiated the trace. 460 + * @cond_data: User data associated with a conditional snapshot 461 + * 462 + * Flip the buffers between the @tr and the max_tr and record information 463 + * about which task was the cause of this latency. 464 + */ 465 + void 466 + update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu, 467 + void *cond_data) 468 + { 469 + if (tr->stop_count) 470 + return; 471 + 472 + WARN_ON_ONCE(!irqs_disabled()); 473 + 474 + if (!tr->allocated_snapshot) { 475 + /* Only the nop tracer should hit this when disabling */ 476 + WARN_ON_ONCE(tr->current_trace != &nop_trace); 477 + return; 478 + } 479 + 480 + arch_spin_lock(&tr->max_lock); 481 + 482 + /* Inherit the recordable setting from array_buffer */ 483 + if (ring_buffer_record_is_set_on(tr->array_buffer.buffer)) 484 + ring_buffer_record_on(tr->snapshot_buffer.buffer); 485 + else 486 + ring_buffer_record_off(tr->snapshot_buffer.buffer); 487 + 488 + if (tr->cond_snapshot && !tr->cond_snapshot->update(tr, cond_data)) { 489 + arch_spin_unlock(&tr->max_lock); 490 + return; 491 + } 492 + 493 + swap(tr->array_buffer.buffer, tr->snapshot_buffer.buffer); 494 + 495 + __update_max_tr(tr, tsk, cpu); 496 + 497 + arch_spin_unlock(&tr->max_lock); 498 + 499 + /* Any waiters on the old snapshot buffer need to wake up */ 500 + ring_buffer_wake_waiters(tr->array_buffer.buffer, RING_BUFFER_ALL_CPUS); 501 + } 502 + 503 + /** 504 + * update_max_tr_single - only copy one trace over, and reset the rest 505 + * @tr: tracer 506 + * @tsk: task with the latency 507 + * @cpu: the cpu of the buffer to copy. 508 + * 509 + * Flip the trace of a single CPU buffer between the @tr and the max_tr. 510 + */ 511 + void 512 + update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu) 513 + { 514 + int ret; 515 + 516 + if (tr->stop_count) 517 + return; 518 + 519 + WARN_ON_ONCE(!irqs_disabled()); 520 + if (!tr->allocated_snapshot) { 521 + /* Only the nop tracer should hit this when disabling */ 522 + WARN_ON_ONCE(tr->current_trace != &nop_trace); 523 + return; 524 + } 525 + 526 + arch_spin_lock(&tr->max_lock); 527 + 528 + ret = ring_buffer_swap_cpu(tr->snapshot_buffer.buffer, tr->array_buffer.buffer, cpu); 529 + 530 + if (ret == -EBUSY) { 531 + /* 532 + * We failed to swap the buffer due to a commit taking 533 + * place on this CPU. We fail to record, but we reset 534 + * the max trace buffer (no one writes directly to it) 535 + * and flag that it failed. 536 + * Another reason is resize is in progress. 537 + */ 538 + trace_array_printk_buf(tr->snapshot_buffer.buffer, _THIS_IP_, 539 + "Failed to swap buffers due to commit or resize in progress\n"); 540 + } 541 + 542 + WARN_ON_ONCE(ret && ret != -EAGAIN && ret != -EBUSY); 543 + 544 + __update_max_tr(tr, tsk, cpu); 545 + arch_spin_unlock(&tr->max_lock); 546 + } 547 + 548 + static void show_snapshot_main_help(struct seq_file *m) 549 + { 550 + seq_puts(m, "# echo 0 > snapshot : Clears and frees snapshot buffer\n" 551 + "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n" 552 + "# Takes a snapshot of the main buffer.\n" 553 + "# echo 2 > snapshot : Clears snapshot buffer (but does not allocate or free)\n" 554 + "# (Doesn't have to be '2' works with any number that\n" 555 + "# is not a '0' or '1')\n"); 556 + } 557 + 558 + static void show_snapshot_percpu_help(struct seq_file *m) 559 + { 560 + seq_puts(m, "# echo 0 > snapshot : Invalid for per_cpu snapshot file.\n"); 561 + #ifdef CONFIG_RING_BUFFER_ALLOW_SWAP 562 + seq_puts(m, "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n" 563 + "# Takes a snapshot of the main buffer for this cpu.\n"); 564 + #else 565 + seq_puts(m, "# echo 1 > snapshot : Not supported with this kernel.\n" 566 + "# Must use main snapshot file to allocate.\n"); 567 + #endif 568 + seq_puts(m, "# echo 2 > snapshot : Clears this cpu's snapshot buffer (but does not allocate)\n" 569 + "# (Doesn't have to be '2' works with any number that\n" 570 + "# is not a '0' or '1')\n"); 571 + } 572 + 573 + void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter) 574 + { 575 + if (iter->tr->allocated_snapshot) 576 + seq_puts(m, "#\n# * Snapshot is allocated *\n#\n"); 577 + else 578 + seq_puts(m, "#\n# * Snapshot is freed *\n#\n"); 579 + 580 + seq_puts(m, "# Snapshot commands:\n"); 581 + if (iter->cpu_file == RING_BUFFER_ALL_CPUS) 582 + show_snapshot_main_help(m); 583 + else 584 + show_snapshot_percpu_help(m); 585 + } 586 + 587 + static int tracing_snapshot_open(struct inode *inode, struct file *file) 588 + { 589 + struct trace_array *tr = inode->i_private; 590 + struct trace_iterator *iter; 591 + struct seq_file *m; 592 + int ret; 593 + 594 + ret = tracing_check_open_get_tr(tr); 595 + if (ret) 596 + return ret; 597 + 598 + if (file->f_mode & FMODE_READ) { 599 + iter = __tracing_open(inode, file, true); 600 + if (IS_ERR(iter)) 601 + ret = PTR_ERR(iter); 602 + } else { 603 + /* Writes still need the seq_file to hold the private data */ 604 + ret = -ENOMEM; 605 + m = kzalloc_obj(*m); 606 + if (!m) 607 + goto out; 608 + iter = kzalloc_obj(*iter); 609 + if (!iter) { 610 + kfree(m); 611 + goto out; 612 + } 613 + ret = 0; 614 + 615 + iter->tr = tr; 616 + iter->array_buffer = &tr->snapshot_buffer; 617 + iter->cpu_file = tracing_get_cpu(inode); 618 + m->private = iter; 619 + file->private_data = m; 620 + } 621 + out: 622 + if (ret < 0) 623 + trace_array_put(tr); 624 + 625 + return ret; 626 + } 627 + 628 + static void tracing_swap_cpu_buffer(void *tr) 629 + { 630 + update_max_tr_single((struct trace_array *)tr, current, smp_processor_id()); 631 + } 632 + 633 + static ssize_t 634 + tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt, 635 + loff_t *ppos) 636 + { 637 + struct seq_file *m = filp->private_data; 638 + struct trace_iterator *iter = m->private; 639 + struct trace_array *tr = iter->tr; 640 + unsigned long val; 641 + int ret; 642 + 643 + ret = tracing_update_buffers(tr); 644 + if (ret < 0) 645 + return ret; 646 + 647 + ret = kstrtoul_from_user(ubuf, cnt, 10, &val); 648 + if (ret) 649 + return ret; 650 + 651 + guard(mutex)(&trace_types_lock); 652 + 653 + if (tracer_uses_snapshot(tr->current_trace)) 654 + return -EBUSY; 655 + 656 + local_irq_disable(); 657 + arch_spin_lock(&tr->max_lock); 658 + if (tr->cond_snapshot) 659 + ret = -EBUSY; 660 + arch_spin_unlock(&tr->max_lock); 661 + local_irq_enable(); 662 + if (ret) 663 + return ret; 664 + 665 + switch (val) { 666 + case 0: 667 + if (iter->cpu_file != RING_BUFFER_ALL_CPUS) 668 + return -EINVAL; 669 + if (tr->allocated_snapshot) 670 + free_snapshot(tr); 671 + break; 672 + case 1: 673 + /* Only allow per-cpu swap if the ring buffer supports it */ 674 + #ifndef CONFIG_RING_BUFFER_ALLOW_SWAP 675 + if (iter->cpu_file != RING_BUFFER_ALL_CPUS) 676 + return -EINVAL; 677 + #endif 678 + if (tr->allocated_snapshot) 679 + ret = resize_buffer_duplicate_size(&tr->snapshot_buffer, 680 + &tr->array_buffer, iter->cpu_file); 681 + 682 + ret = tracing_arm_snapshot_locked(tr); 683 + if (ret) 684 + return ret; 685 + 686 + /* Now, we're going to swap */ 687 + if (iter->cpu_file == RING_BUFFER_ALL_CPUS) { 688 + local_irq_disable(); 689 + update_max_tr(tr, current, smp_processor_id(), NULL); 690 + local_irq_enable(); 691 + } else { 692 + smp_call_function_single(iter->cpu_file, tracing_swap_cpu_buffer, 693 + (void *)tr, 1); 694 + } 695 + tracing_disarm_snapshot(tr); 696 + break; 697 + default: 698 + if (tr->allocated_snapshot) { 699 + if (iter->cpu_file == RING_BUFFER_ALL_CPUS) 700 + tracing_reset_online_cpus(&tr->snapshot_buffer); 701 + else 702 + tracing_reset_cpu(&tr->snapshot_buffer, iter->cpu_file); 703 + } 704 + break; 705 + } 706 + 707 + if (ret >= 0) { 708 + *ppos += cnt; 709 + ret = cnt; 710 + } 711 + 712 + return ret; 713 + } 714 + 715 + static int tracing_snapshot_release(struct inode *inode, struct file *file) 716 + { 717 + struct seq_file *m = file->private_data; 718 + int ret; 719 + 720 + ret = tracing_release(inode, file); 721 + 722 + if (file->f_mode & FMODE_READ) 723 + return ret; 724 + 725 + /* If write only, the seq_file is just a stub */ 726 + if (m) 727 + kfree(m->private); 728 + kfree(m); 729 + 730 + return 0; 731 + } 732 + 733 + static int snapshot_raw_open(struct inode *inode, struct file *filp) 734 + { 735 + struct ftrace_buffer_info *info; 736 + int ret; 737 + 738 + /* The following checks for tracefs lockdown */ 739 + ret = tracing_buffers_open(inode, filp); 740 + if (ret < 0) 741 + return ret; 742 + 743 + info = filp->private_data; 744 + 745 + if (tracer_uses_snapshot(info->iter.trace)) { 746 + tracing_buffers_release(inode, filp); 747 + return -EBUSY; 748 + } 749 + 750 + info->iter.snapshot = true; 751 + info->iter.array_buffer = &info->iter.tr->snapshot_buffer; 752 + 753 + return ret; 754 + } 755 + 756 + const struct file_operations snapshot_fops = { 757 + .open = tracing_snapshot_open, 758 + .read = seq_read, 759 + .write = tracing_snapshot_write, 760 + .llseek = tracing_lseek, 761 + .release = tracing_snapshot_release, 762 + }; 763 + 764 + const struct file_operations snapshot_raw_fops = { 765 + .open = snapshot_raw_open, 766 + .read = tracing_buffers_read, 767 + .release = tracing_buffers_release, 768 + .splice_read = tracing_buffers_splice_read, 769 + }; 770 + 771 + #ifdef CONFIG_TRACER_MAX_TRACE 772 + static ssize_t 773 + tracing_max_lat_read(struct file *filp, char __user *ubuf, 774 + size_t cnt, loff_t *ppos) 775 + { 776 + struct trace_array *tr = filp->private_data; 777 + 778 + return tracing_nsecs_read(&tr->max_latency, ubuf, cnt, ppos); 779 + } 780 + 781 + static ssize_t 782 + tracing_max_lat_write(struct file *filp, const char __user *ubuf, 783 + size_t cnt, loff_t *ppos) 784 + { 785 + struct trace_array *tr = filp->private_data; 786 + 787 + return tracing_nsecs_write(&tr->max_latency, ubuf, cnt, ppos); 788 + } 789 + 790 + static const struct file_operations tracing_max_lat_fops = { 791 + .open = tracing_open_generic_tr, 792 + .read = tracing_max_lat_read, 793 + .write = tracing_max_lat_write, 794 + .llseek = generic_file_llseek, 795 + .release = tracing_release_generic_tr, 796 + }; 797 + #endif /* CONFIG_TRACER_MAX_TRACE */ 798 + 799 + int get_snapshot_map(struct trace_array *tr) 800 + { 801 + int err = 0; 802 + 803 + /* 804 + * Called with mmap_lock held. lockdep would be unhappy if we would now 805 + * take trace_types_lock. Instead use the specific 806 + * snapshot_trigger_lock. 807 + */ 808 + spin_lock(&tr->snapshot_trigger_lock); 809 + 810 + if (tr->snapshot || tr->mapped == UINT_MAX) 811 + err = -EBUSY; 812 + else 813 + tr->mapped++; 814 + 815 + spin_unlock(&tr->snapshot_trigger_lock); 816 + 817 + /* Wait for update_max_tr() to observe iter->tr->mapped */ 818 + if (tr->mapped == 1) 819 + synchronize_rcu(); 820 + 821 + return err; 822 + 823 + } 824 + 825 + void put_snapshot_map(struct trace_array *tr) 826 + { 827 + spin_lock(&tr->snapshot_trigger_lock); 828 + if (!WARN_ON(!tr->mapped)) 829 + tr->mapped--; 830 + spin_unlock(&tr->snapshot_trigger_lock); 831 + } 832 + 833 + #ifdef CONFIG_DYNAMIC_FTRACE 834 + static void 835 + ftrace_snapshot(unsigned long ip, unsigned long parent_ip, 836 + struct trace_array *tr, struct ftrace_probe_ops *ops, 837 + void *data) 838 + { 839 + tracing_snapshot_instance(tr); 840 + } 841 + 842 + static void 843 + ftrace_count_snapshot(unsigned long ip, unsigned long parent_ip, 844 + struct trace_array *tr, struct ftrace_probe_ops *ops, 845 + void *data) 846 + { 847 + struct ftrace_func_mapper *mapper = data; 848 + long *count = NULL; 849 + 850 + if (mapper) 851 + count = (long *)ftrace_func_mapper_find_ip(mapper, ip); 852 + 853 + if (count) { 854 + 855 + if (*count <= 0) 856 + return; 857 + 858 + (*count)--; 859 + } 860 + 861 + tracing_snapshot_instance(tr); 862 + } 863 + 864 + static int 865 + ftrace_snapshot_print(struct seq_file *m, unsigned long ip, 866 + struct ftrace_probe_ops *ops, void *data) 867 + { 868 + struct ftrace_func_mapper *mapper = data; 869 + long *count = NULL; 870 + 871 + seq_printf(m, "%ps:", (void *)ip); 872 + 873 + seq_puts(m, "snapshot"); 874 + 875 + if (mapper) 876 + count = (long *)ftrace_func_mapper_find_ip(mapper, ip); 877 + 878 + if (count) 879 + seq_printf(m, ":count=%ld\n", *count); 880 + else 881 + seq_puts(m, ":unlimited\n"); 882 + 883 + return 0; 884 + } 885 + 886 + static int 887 + ftrace_snapshot_init(struct ftrace_probe_ops *ops, struct trace_array *tr, 888 + unsigned long ip, void *init_data, void **data) 889 + { 890 + struct ftrace_func_mapper *mapper = *data; 891 + 892 + if (!mapper) { 893 + mapper = allocate_ftrace_func_mapper(); 894 + if (!mapper) 895 + return -ENOMEM; 896 + *data = mapper; 897 + } 898 + 899 + return ftrace_func_mapper_add_ip(mapper, ip, init_data); 900 + } 901 + 902 + static void 903 + ftrace_snapshot_free(struct ftrace_probe_ops *ops, struct trace_array *tr, 904 + unsigned long ip, void *data) 905 + { 906 + struct ftrace_func_mapper *mapper = data; 907 + 908 + if (!ip) { 909 + if (!mapper) 910 + return; 911 + free_ftrace_func_mapper(mapper, NULL); 912 + return; 913 + } 914 + 915 + ftrace_func_mapper_remove_ip(mapper, ip); 916 + } 917 + 918 + static struct ftrace_probe_ops snapshot_probe_ops = { 919 + .func = ftrace_snapshot, 920 + .print = ftrace_snapshot_print, 921 + }; 922 + 923 + static struct ftrace_probe_ops snapshot_count_probe_ops = { 924 + .func = ftrace_count_snapshot, 925 + .print = ftrace_snapshot_print, 926 + .init = ftrace_snapshot_init, 927 + .free = ftrace_snapshot_free, 928 + }; 929 + 930 + static int 931 + ftrace_trace_snapshot_callback(struct trace_array *tr, struct ftrace_hash *hash, 932 + char *glob, char *cmd, char *param, int enable) 933 + { 934 + struct ftrace_probe_ops *ops; 935 + void *count = (void *)-1; 936 + char *number; 937 + int ret; 938 + 939 + if (!tr) 940 + return -ENODEV; 941 + 942 + /* hash funcs only work with set_ftrace_filter */ 943 + if (!enable) 944 + return -EINVAL; 945 + 946 + ops = param ? &snapshot_count_probe_ops : &snapshot_probe_ops; 947 + 948 + if (glob[0] == '!') { 949 + ret = unregister_ftrace_function_probe_func(glob+1, tr, ops); 950 + if (!ret) 951 + tracing_disarm_snapshot(tr); 952 + 953 + return ret; 954 + } 955 + 956 + if (!param) 957 + goto out_reg; 958 + 959 + number = strsep(&param, ":"); 960 + 961 + if (!strlen(number)) 962 + goto out_reg; 963 + 964 + /* 965 + * We use the callback data field (which is a pointer) 966 + * as our counter. 967 + */ 968 + ret = kstrtoul(number, 0, (unsigned long *)&count); 969 + if (ret) 970 + return ret; 971 + 972 + out_reg: 973 + ret = tracing_arm_snapshot(tr); 974 + if (ret < 0) 975 + return ret; 976 + 977 + ret = register_ftrace_function_probe(glob, tr, ops, count); 978 + if (ret < 0) 979 + tracing_disarm_snapshot(tr); 980 + 981 + return ret < 0 ? ret : 0; 982 + } 983 + 984 + static struct ftrace_func_command ftrace_snapshot_cmd = { 985 + .name = "snapshot", 986 + .func = ftrace_trace_snapshot_callback, 987 + }; 988 + 989 + __init int register_snapshot_cmd(void) 990 + { 991 + return register_ftrace_command(&ftrace_snapshot_cmd); 992 + } 993 + #endif /* CONFIG_DYNAMIC_FTRACE */ 994 + 995 + int trace_allocate_snapshot(struct trace_array *tr, int size) 996 + { 997 + int ret; 998 + 999 + /* Fix mapped buffer trace arrays do not have snapshot buffers */ 1000 + if (tr->range_addr_start) 1001 + return 0; 1002 + 1003 + /* allocate_snapshot can only be true during system boot */ 1004 + ret = allocate_trace_buffer(tr, &tr->snapshot_buffer, 1005 + allocate_snapshot ? size : 1); 1006 + if (ret < 0) 1007 + return -ENOMEM; 1008 + 1009 + tr->allocated_snapshot = allocate_snapshot; 1010 + 1011 + allocate_snapshot = false; 1012 + return 0; 1013 + } 1014 + 1015 + __init static bool tr_needs_alloc_snapshot(const char *name) 1016 + { 1017 + char *test; 1018 + int len = strlen(name); 1019 + bool ret; 1020 + 1021 + if (!boot_snapshot_index) 1022 + return false; 1023 + 1024 + if (strncmp(name, boot_snapshot_info, len) == 0 && 1025 + boot_snapshot_info[len] == '\t') 1026 + return true; 1027 + 1028 + test = kmalloc(strlen(name) + 3, GFP_KERNEL); 1029 + if (!test) 1030 + return false; 1031 + 1032 + sprintf(test, "\t%s\t", name); 1033 + ret = strstr(boot_snapshot_info, test) == NULL; 1034 + kfree(test); 1035 + return ret; 1036 + } 1037 + 1038 + __init void do_allocate_snapshot(const char *name) 1039 + { 1040 + if (!tr_needs_alloc_snapshot(name)) 1041 + return; 1042 + 1043 + /* 1044 + * When allocate_snapshot is set, the next call to 1045 + * allocate_trace_buffers() (called by trace_array_get_by_name()) 1046 + * will allocate the snapshot buffer. That will also clear 1047 + * this flag. 1048 + */ 1049 + allocate_snapshot = true; 1050 + } 1051 + 1052 + void __init ftrace_boot_snapshot(void) 1053 + { 1054 + struct trace_array *tr; 1055 + 1056 + if (!snapshot_at_boot) 1057 + return; 1058 + 1059 + list_for_each_entry(tr, &ftrace_trace_arrays, list) { 1060 + if (!tr->allocated_snapshot) 1061 + continue; 1062 + 1063 + tracing_snapshot_instance(tr); 1064 + trace_array_puts(tr, "** Boot snapshot taken **\n"); 1065 + } 1066 + }
+2
kernel/tracepoint.c
··· 300 300 lockdep_is_held(&tracepoints_mutex)); 301 301 old = func_add(&tp_funcs, func, prio); 302 302 if (IS_ERR(old)) { 303 + if (tp->ext && tp->ext->unregfunc && !static_key_enabled(&tp->key)) 304 + tp->ext->unregfunc(); 303 305 WARN_ON_ONCE(warn && PTR_ERR(old) != -ENOMEM); 304 306 return PTR_ERR(old); 305 307 }
+1 -1
mm/damon/core.c
··· 2510 2510 break; 2511 2511 sidx++; 2512 2512 } 2513 - trace_damos_stat_after_apply_interval(cidx, sidx, &s->stat); 2513 + trace_call__damos_stat_after_apply_interval(cidx, sidx, &s->stat); 2514 2514 } 2515 2515 2516 2516 static void kdamond_apply_schemes(struct damon_ctx *c)
+34
tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-fully-qualified-var-ref.tc
··· 1 + #!/bin/sh 2 + # SPDX-License-Identifier: GPL-2.0 3 + # description: event trigger - test fully-qualified variable reference support 4 + # requires: set_event synthetic_events events/sched/sched_process_fork/hist ping:program 5 + 6 + fail() { #msg 7 + echo $1 8 + exit_fail 9 + } 10 + 11 + echo "Test fully-qualified variable reference support" 12 + 13 + echo 'wakeup_latency u64 lat; pid_t pid; int prio; char comm[16]' > synthetic_events 14 + echo 'hist:keys=comm:ts0=common_timestamp.usecs if comm=="ping"' > events/sched/sched_waking/trigger 15 + echo 'hist:keys=comm:ts0=common_timestamp.usecs if comm=="ping"' > events/sched/sched_wakeup/trigger 16 + echo 'hist:keys=next_comm:wakeup_lat=common_timestamp.usecs-sched.sched_wakeup.$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,sched.sched_waking.prio,next_comm) if next_comm=="ping"' > events/sched/sched_switch/trigger 17 + echo 'hist:keys=pid,prio,comm:vals=lat:sort=pid,prio' > events/synthetic/wakeup_latency/trigger 18 + 19 + ping $LOCALHOST -c 3 20 + if ! grep -q "ping" events/synthetic/wakeup_latency/hist; then 21 + fail "Failed to create inter-event histogram" 22 + fi 23 + 24 + if ! grep -q "synthetic_prio=prio" events/sched/sched_waking/hist; then 25 + fail "Failed to create histogram with fully-qualified variable reference" 26 + fi 27 + 28 + echo '!hist:keys=next_comm:wakeup_lat=common_timestamp.usecs-sched.sched_wakeup.$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,sched.sched_waking.prio,next_comm) if next_comm=="ping"' >> events/sched/sched_switch/trigger 29 + 30 + if grep -q "synthetic_prio=prio" events/sched/sched_waking/hist; then 31 + fail "Failed to remove histogram with fully-qualified variable reference" 32 + fi 33 + 34 + exit 0