Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'trace-v6.16' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

Pull tracing updates from Steven Rostedt:

- Have module addresses get updated in the persistent ring buffer

The addresses of the modules from the previous boot are saved in the
persistent ring buffer. If the same modules are loaded and an address
is in the old buffer points to an address that was both saved in the
persistent ring buffer and is loaded in memory, shift the address to
point to the address that is loaded in memory in the trace event.

- Print function names for irqs off and preempt off callsites

When ignoring the print fmt of a trace event and just printing the
fields directly, have the fields for preempt off and irqs off events
still show the function name (via kallsyms) instead of just showing
the raw address.

- Clean ups of the histogram code

The histogram functions saved over 800 bytes on the stack to process
events as they come in. Instead, create per-cpu buffers that can hold
this information and have a separate location for each context level
(thread, softirq, IRQ and NMI).

Also add some more comments to the code.

- Add "common_comm" field for histograms

Add "common_comm" that uses the current->comm as a field in an event
histogram and acts like any of the other fields of the event.

- Show "subops" in the enabled_functions file

When the function graph infrastructure is used, a subsystem has a
"subops" that it attaches its callback function to. Instead of the
enabled_functions just showing a function calling the function that
calls the subops functions, also show the subops functions that will
get called for that function too.

- Add "copy_trace_marker" option to instances

There are cases where an instance is created for tooling to write
into, but the old tooling has the top level instance hardcoded into
the application. New tools want to consume the data from an instance
and not the top level buffer. By adding a copy_trace_marker option,
whenever the top instance trace_marker is written into, a copy of it
is also written into the instance with this option set. This allows
new tools to read what old tools are writing into the top buffer.

If this option is cleared by the top instance, then what is written
into the trace_marker is not written into the top instance. This is a
way to redirect the trace_marker writes into another instance.

- Have tracepoints created by DECLARE_TRACE() use trace_<name>_tp()

If a tracepoint is created by DECLARE_TRACE() instead of
TRACE_EVENT(), then it will not be exposed via tracefs. Currently
there's no way to differentiate in the kernel the tracepoint
functions between those that are exposed via tracefs or not. A
calling convention has been made manually to append a "_tp" prefix
for events created by DECLARE_TRACE(). Instead of doing this
manually, force it so that all DECLARE_TRACE() events have this
notation.

- Use __string() for task->comm in some sched events

Instead of hardcoding the comm to be TASK_COMM_LEN in some of the
scheduler events use __string() which makes it dynamic. Note, if
these events are parsed by user space it they may break, and the
event may have to be converted back to the hardcoded size.

- Have function graph "depth" be unsigned to the user

Internally to the kernel, the "depth" field of the function graph
event is signed due to -1 being used for end of boundary. What
actually gets recorded in the event itself is zero or positive.
Reflect this to user space by showing "depth" as unsigned int and be
consistent across all events.

- Allow an arbitrary long CPU string to osnoise_cpus_write()

The filtering of which CPUs to write to can exceed 256 bytes. If a
machine has 256 CPUs, and the filter is to filter every other CPU,
the write would take a string larger than 256 bytes. Instead of using
a fixed size buffer on the stack that is 256 bytes, allocate it to
handle what is passed in.

- Stop having ftrace check the per-cpu data "disabled" flag

The "disabled" flag in the data structure passed to most ftrace
functions is checked to know if tracing has been disabled or not.
This flag was added back in 2008 before the ring buffer had its own
way to disable tracing. The "disable" flag is now not always set when
needed, and the ring buffer flag should be used in all locations
where the disabled is needed. Since the "disable" flag is redundant
and incorrect, stop using it. Fix up some locations that use the
"disable" flag to use the ring buffer info.

- Use a new tracer_tracing_disable/enable() instead of data->disable
flag

There's a few cases that set the data->disable flag to stop tracing,
but this flag is not consistently used. It is also an on/off switch
where if a function set it and calls another function that sets it,
the called function may incorrectly enable it.

Use a new trace_tracing_disable() and tracer_tracing_enable() that
uses a counter and can be nested. These use the ring buffer flags
which are always checked making the disabling more consistent.

- Save the trace clock in the persistent ring buffer

Save what clock was used for tracing in the persistent ring buffer
and set it back to that clock after a reboot.

- Remove unused reference to a per CPU data pointer in mmiotrace
functions

- Remove unused buffer_page field from trace_array_cpu structure

- Remove more strncpy() instances

- Other minor clean ups and fixes

* tag 'trace-v6.16' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (36 commits)
tracing: Fix compilation warning on arm32
tracing: Record trace_clock and recover when reboot
tracing/sched: Use __string() instead of fixed lengths for task->comm
tracepoint: Have tracepoints created with DECLARE_TRACE() have _tp suffix
tracing: Cleanup upper_empty() in pid_list
tracing: Allow the top level trace_marker to write into another instances
tracing: Add a helper function to handle the dereference arg in verifier
tracing: Remove unnecessary "goto out" that simply returns ret is trigger code
tracing: Fix error handling in event_trigger_parse()
tracing: Rename event_trigger_alloc() to trigger_data_alloc()
tracing: Replace deprecated strncpy() with strscpy() for stack_trace_filter_buf
tracing: Remove unused buffer_page field from trace_array_cpu structure
tracing: Use atomic_inc_return() for updating "disabled" counter in irqsoff tracer
tracing: Convert the per CPU "disabled" counter to local from atomic
tracing: branch: Use trace_tracing_is_on_cpu() instead of "disabled" field
ring-buffer: Add ring_buffer_record_is_on_cpu()
tracing: Do not use per CPU array_buffer.data->disabled for cpumask
ftrace: Do not disabled function graph based on "disabled" field
tracing: kdb: Use tracer_tracing_on/off() instead of setting per CPU disabled
tracing: Use tracer_tracing_disable() instead of "disabled" field for ftrace_dump_one()
...

+720 -356
+13
Documentation/trace/ftrace.rst
··· 1205 1205 default instance. The only way the top level instance has this flag 1206 1206 cleared, is by it being set in another instance. 1207 1207 1208 + copy_trace_marker 1209 + If there are applications that hard code writing into the top level 1210 + trace_marker file (/sys/kernel/tracing/trace_marker or trace_marker_raw), 1211 + and the tooling would like it to go into an instance, this option can 1212 + be used. Create an instance and set this option, and then all writes 1213 + into the top level trace_marker file will also be redirected into this 1214 + instance. 1215 + 1216 + Note, by default this option is set for the top level instance. If it 1217 + is disabled, then writes to the trace_marker or trace_marker_raw files 1218 + will not be written into the top level file. If no instance has this 1219 + option set, then a write will error with the errno of ENODEV. 1220 + 1208 1221 annotate 1209 1222 It is sometimes confusing when the CPU buffers are full 1210 1223 and one CPU buffer had a lot of events recently, thus
+11 -6
Documentation/trace/tracepoints.rst
··· 71 71 void somefct(void) 72 72 { 73 73 ... 74 - trace_subsys_eventname(arg, task); 74 + trace_subsys_eventname_tp(arg, task); 75 75 ... 76 76 } 77 77 ··· 129 129 for (i = 0; i < count; i++) 130 130 tot += calculate_nuggets(); 131 131 132 - trace_foo_bar(tot); 132 + trace_foo_bar_tp(tot); 133 133 } 134 134 135 - All trace_<tracepoint>() calls have a matching trace_<tracepoint>_enabled() 135 + All trace_<tracepoint>_tp() calls have a matching trace_<tracepoint>_enabled() 136 136 function defined that returns true if the tracepoint is enabled and 137 - false otherwise. The trace_<tracepoint>() should always be within the 137 + false otherwise. The trace_<tracepoint>_tp() should always be within the 138 138 block of the if (trace_<tracepoint>_enabled()) to prevent races between 139 139 the tracepoint being enabled and the check being seen. 140 140 ··· 143 143 with jump labels and avoid conditional branches. 144 144 145 145 .. note:: The convenience macro TRACE_EVENT provides an alternative way to 146 - define tracepoints. Check http://lwn.net/Articles/379903, 146 + define tracepoints. Note, DECLARE_TRACE(foo) creates a function 147 + "trace_foo_tp()" whereas TRACE_EVENT(foo) creates a function 148 + "trace_foo()", and also exposes the tracepoint as a trace event in 149 + /sys/kernel/tracing/events directory. Check http://lwn.net/Articles/379903, 147 150 http://lwn.net/Articles/381064 and http://lwn.net/Articles/383362 148 151 for a series of articles with more details. 149 152 ··· 162 159 163 160 void do_trace_foo_bar_wrapper(args) 164 161 { 165 - trace_foo_bar(args); 162 + trace_foo_bar_tp(args); // for tracepoints created via DECLARE_TRACE 163 + // or 164 + trace_foo_bar(args); // for tracepoints created via TRACE_EVENT 166 165 } 167 166 168 167 In the header file::
+2
include/linux/ftrace.h
··· 328 328 * DIRECT - Used by the direct ftrace_ops helper for direct functions 329 329 * (internal ftrace only, should not be used by others) 330 330 * SUBOP - Is controlled by another op in field managed. 331 + * GRAPH - Is a component of the fgraph_ops structure 331 332 */ 332 333 enum { 333 334 FTRACE_OPS_FL_ENABLED = BIT(0), ··· 350 349 FTRACE_OPS_FL_PERMANENT = BIT(16), 351 350 FTRACE_OPS_FL_DIRECT = BIT(17), 352 351 FTRACE_OPS_FL_SUBOP = BIT(18), 352 + FTRACE_OPS_FL_GRAPH = BIT(19), 353 353 }; 354 354 355 355 #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
+1
include/linux/ring_buffer.h
··· 192 192 void ring_buffer_record_on(struct trace_buffer *buffer); 193 193 bool ring_buffer_record_is_on(struct trace_buffer *buffer); 194 194 bool ring_buffer_record_is_set_on(struct trace_buffer *buffer); 195 + bool ring_buffer_record_is_on_cpu(struct trace_buffer *buffer, int cpu); 195 196 void ring_buffer_record_disable_cpu(struct trace_buffer *buffer, int cpu); 196 197 void ring_buffer_record_enable_cpu(struct trace_buffer *buffer, int cpu); 197 198
+26 -12
include/linux/tracepoint.h
··· 464 464 #endif 465 465 466 466 #define DECLARE_TRACE(name, proto, args) \ 467 - __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 467 + __DECLARE_TRACE(name##_tp, PARAMS(proto), PARAMS(args), \ 468 468 cpu_online(raw_smp_processor_id()), \ 469 469 PARAMS(void *__data, proto)) 470 470 471 471 #define DECLARE_TRACE_CONDITION(name, proto, args, cond) \ 472 - __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 472 + __DECLARE_TRACE(name##_tp, PARAMS(proto), PARAMS(args), \ 473 473 cpu_online(raw_smp_processor_id()) && (PARAMS(cond)), \ 474 474 PARAMS(void *__data, proto)) 475 475 476 476 #define DECLARE_TRACE_SYSCALL(name, proto, args) \ 477 + __DECLARE_TRACE_SYSCALL(name##_tp, PARAMS(proto), PARAMS(args), \ 478 + PARAMS(void *__data, proto)) 479 + 480 + #define DECLARE_TRACE_EVENT(name, proto, args) \ 481 + __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 482 + cpu_online(raw_smp_processor_id()), \ 483 + PARAMS(void *__data, proto)) 484 + 485 + #define DECLARE_TRACE_EVENT_CONDITION(name, proto, args, cond) \ 486 + __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 487 + cpu_online(raw_smp_processor_id()) && (PARAMS(cond)), \ 488 + PARAMS(void *__data, proto)) 489 + 490 + #define DECLARE_TRACE_EVENT_SYSCALL(name, proto, args) \ 477 491 __DECLARE_TRACE_SYSCALL(name, PARAMS(proto), PARAMS(args), \ 478 492 PARAMS(void *__data, proto)) 479 493 ··· 605 591 606 592 #define DECLARE_EVENT_CLASS(name, proto, args, tstruct, assign, print) 607 593 #define DEFINE_EVENT(template, name, proto, args) \ 608 - DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 594 + DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args)) 609 595 #define DEFINE_EVENT_FN(template, name, proto, args, reg, unreg)\ 610 - DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 596 + DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args)) 611 597 #define DEFINE_EVENT_PRINT(template, name, proto, args, print) \ 612 - DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 598 + DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args)) 613 599 #define DEFINE_EVENT_CONDITION(template, name, proto, \ 614 600 args, cond) \ 615 - DECLARE_TRACE_CONDITION(name, PARAMS(proto), \ 601 + DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \ 616 602 PARAMS(args), PARAMS(cond)) 617 603 618 604 #define TRACE_EVENT(name, proto, args, struct, assign, print) \ 619 - DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 605 + DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args)) 620 606 #define TRACE_EVENT_FN(name, proto, args, struct, \ 621 607 assign, print, reg, unreg) \ 622 - DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 623 - #define TRACE_EVENT_FN_COND(name, proto, args, cond, struct, \ 608 + DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args)) 609 + #define TRACE_EVENT_FN_COND(name, proto, args, cond, struct, \ 624 610 assign, print, reg, unreg) \ 625 - DECLARE_TRACE_CONDITION(name, PARAMS(proto), \ 611 + DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \ 626 612 PARAMS(args), PARAMS(cond)) 627 613 #define TRACE_EVENT_CONDITION(name, proto, args, cond, \ 628 614 struct, assign, print) \ 629 - DECLARE_TRACE_CONDITION(name, PARAMS(proto), \ 615 + DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \ 630 616 PARAMS(args), PARAMS(cond)) 631 617 #define TRACE_EVENT_SYSCALL(name, proto, args, struct, assign, \ 632 618 print, reg, unreg) \ 633 - DECLARE_TRACE_SYSCALL(name, PARAMS(proto), PARAMS(args)) 619 + DECLARE_TRACE_EVENT_SYSCALL(name, PARAMS(proto), PARAMS(args)) 634 620 635 621 #define TRACE_EVENT_FLAGS(event, flag) 636 622
+4 -4
include/trace/bpf_probe.h
··· 119 119 120 120 #undef DECLARE_TRACE 121 121 #define DECLARE_TRACE(call, proto, args) \ 122 - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ 123 - __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), 0) 122 + __BPF_DECLARE_TRACE(call##_tp, PARAMS(proto), PARAMS(args)) \ 123 + __DEFINE_EVENT(call##_tp, call##_tp, PARAMS(proto), PARAMS(args), 0) 124 124 125 125 #undef DECLARE_TRACE_WRITABLE 126 126 #define DECLARE_TRACE_WRITABLE(call, proto, args, size) \ 127 127 __CHECK_WRITABLE_BUF_SIZE(call, PARAMS(proto), PARAMS(args), size) \ 128 - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ 129 - __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), size) 128 + __BPF_DECLARE_TRACE(call##_tp, PARAMS(proto), PARAMS(args)) \ 129 + __DEFINE_EVENT(call##_tp, call##_tp, PARAMS(proto), PARAMS(args), size) 130 130 131 131 #include TRACE_INCLUDE(TRACE_INCLUDE_FILE) 132 132
+16 -1
include/trace/define_trace.h
··· 74 74 75 75 #undef DECLARE_TRACE 76 76 #define DECLARE_TRACE(name, proto, args) \ 77 - DEFINE_TRACE(name, PARAMS(proto), PARAMS(args)) 77 + DEFINE_TRACE(name##_tp, PARAMS(proto), PARAMS(args)) 78 78 79 79 #undef DECLARE_TRACE_CONDITION 80 80 #define DECLARE_TRACE_CONDITION(name, proto, args, cond) \ 81 + DEFINE_TRACE(name##_tp, PARAMS(proto), PARAMS(args)) 82 + 83 + #undef DECLARE_TRACE_EVENT 84 + #define DECLARE_TRACE_EVENT(name, proto, args) \ 85 + DEFINE_TRACE(name, PARAMS(proto), PARAMS(args)) 86 + 87 + #undef DECLARE_TRACE_EVENT_CONDITION 88 + #define DECLARE_TRACE_EVENT_CONDITION(name, proto, args, cond) \ 81 89 DEFINE_TRACE(name, PARAMS(proto), PARAMS(args)) 82 90 83 91 /* If requested, create helpers for calling these tracepoints from Rust. */ ··· 123 115 #undef DECLARE_TRACE_CONDITION 124 116 #define DECLARE_TRACE_CONDITION(name, proto, args, cond) 125 117 118 + #undef DECLARE_TRACE_EVENT 119 + #define DECLARE_TRACE_EVENT(name, proto, args) 120 + #undef DECLARE_TRACE_EVENT_CONDITION 121 + #define DECLARE_TRACE_EVENT_CONDITION(name, proto, args, cond) 122 + 126 123 #ifdef TRACEPOINTS_ENABLED 127 124 #include <trace/trace_events.h> 128 125 #include <trace/perf.h> ··· 149 136 #undef TRACE_HEADER_MULTI_READ 150 137 #undef DECLARE_TRACE 151 138 #undef DECLARE_TRACE_CONDITION 139 + #undef DECLARE_TRACE_EVENT 140 + #undef DECLARE_TRACE_EVENT_CONDITION 152 141 153 142 /* Only undef what we defined in this file */ 154 143 #ifdef UNDEF_TRACE_INCLUDE_FILE
+62 -62
include/trace/events/sched.h
··· 20 20 TP_ARGS(t), 21 21 22 22 TP_STRUCT__entry( 23 - __array( char, comm, TASK_COMM_LEN ) 24 - __field( pid_t, pid ) 23 + __string( comm, t->comm ) 24 + __field( pid_t, pid ) 25 25 ), 26 26 27 27 TP_fast_assign( 28 - memcpy(__entry->comm, t->comm, TASK_COMM_LEN); 28 + __assign_str(comm); 29 29 __entry->pid = t->pid; 30 30 ), 31 31 32 - TP_printk("comm=%s pid=%d", __entry->comm, __entry->pid) 32 + TP_printk("comm=%s pid=%d", __get_str(comm), __entry->pid) 33 33 ); 34 34 35 35 /* ··· 276 276 TP_ARGS(p, dest_cpu), 277 277 278 278 TP_STRUCT__entry( 279 - __array( char, comm, TASK_COMM_LEN ) 280 - __field( pid_t, pid ) 281 - __field( int, prio ) 282 - __field( int, orig_cpu ) 283 - __field( int, dest_cpu ) 279 + __string( comm, p->comm ) 280 + __field( pid_t, pid ) 281 + __field( int, prio ) 282 + __field( int, orig_cpu ) 283 + __field( int, dest_cpu ) 284 284 ), 285 285 286 286 TP_fast_assign( 287 - memcpy(__entry->comm, p->comm, TASK_COMM_LEN); 287 + __assign_str(comm); 288 288 __entry->pid = p->pid; 289 289 __entry->prio = p->prio; /* XXX SCHED_DEADLINE */ 290 290 __entry->orig_cpu = task_cpu(p); ··· 292 292 ), 293 293 294 294 TP_printk("comm=%s pid=%d prio=%d orig_cpu=%d dest_cpu=%d", 295 - __entry->comm, __entry->pid, __entry->prio, 295 + __get_str(comm), __entry->pid, __entry->prio, 296 296 __entry->orig_cpu, __entry->dest_cpu) 297 297 ); 298 298 ··· 303 303 TP_ARGS(p), 304 304 305 305 TP_STRUCT__entry( 306 - __array( char, comm, TASK_COMM_LEN ) 307 - __field( pid_t, pid ) 308 - __field( int, prio ) 306 + __string( comm, p->comm ) 307 + __field( pid_t, pid ) 308 + __field( int, prio ) 309 309 ), 310 310 311 311 TP_fast_assign( 312 - memcpy(__entry->comm, p->comm, TASK_COMM_LEN); 312 + __assign_str(comm); 313 313 __entry->pid = p->pid; 314 314 __entry->prio = p->prio; /* XXX SCHED_DEADLINE */ 315 315 ), 316 316 317 317 TP_printk("comm=%s pid=%d prio=%d", 318 - __entry->comm, __entry->pid, __entry->prio) 318 + __get_str(comm), __entry->pid, __entry->prio) 319 319 ); 320 320 321 321 /* ··· 375 375 TP_ARGS(pid), 376 376 377 377 TP_STRUCT__entry( 378 - __array( char, comm, TASK_COMM_LEN ) 378 + __string( comm, current->comm ) 379 379 __field( pid_t, pid ) 380 380 __field( int, prio ) 381 381 ), 382 382 383 383 TP_fast_assign( 384 - memcpy(__entry->comm, current->comm, TASK_COMM_LEN); 384 + __assign_str(comm); 385 385 __entry->pid = pid_nr(pid); 386 386 __entry->prio = current->prio; /* XXX SCHED_DEADLINE */ 387 387 ), 388 388 389 389 TP_printk("comm=%s pid=%d prio=%d", 390 - __entry->comm, __entry->pid, __entry->prio) 390 + __get_str(comm), __entry->pid, __entry->prio) 391 391 ); 392 392 393 393 /* ··· 400 400 TP_ARGS(parent, child), 401 401 402 402 TP_STRUCT__entry( 403 - __array( char, parent_comm, TASK_COMM_LEN ) 404 - __field( pid_t, parent_pid ) 405 - __array( char, child_comm, TASK_COMM_LEN ) 406 - __field( pid_t, child_pid ) 403 + __string( parent_comm, parent->comm ) 404 + __field( pid_t, parent_pid ) 405 + __string( child_comm, child->comm ) 406 + __field( pid_t, child_pid ) 407 407 ), 408 408 409 409 TP_fast_assign( 410 - memcpy(__entry->parent_comm, parent->comm, TASK_COMM_LEN); 410 + __assign_str(parent_comm); 411 411 __entry->parent_pid = parent->pid; 412 - memcpy(__entry->child_comm, child->comm, TASK_COMM_LEN); 412 + __assign_str(child_comm); 413 413 __entry->child_pid = child->pid; 414 414 ), 415 415 416 416 TP_printk("comm=%s pid=%d child_comm=%s child_pid=%d", 417 - __entry->parent_comm, __entry->parent_pid, 418 - __entry->child_comm, __entry->child_pid) 417 + __get_str(parent_comm), __entry->parent_pid, 418 + __get_str(child_comm), __entry->child_pid) 419 419 ); 420 420 421 421 /* ··· 499 499 TP_ARGS(__perf_task(tsk), __perf_count(delay)), 500 500 501 501 TP_STRUCT__entry( 502 - __array( char, comm, TASK_COMM_LEN ) 503 - __field( pid_t, pid ) 504 - __field( u64, delay ) 502 + __string( comm, tsk->comm ) 503 + __field( pid_t, pid ) 504 + __field( u64, delay ) 505 505 ), 506 506 507 507 TP_fast_assign( 508 - memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); 508 + __assign_str(comm); 509 509 __entry->pid = tsk->pid; 510 510 __entry->delay = delay; 511 511 ), 512 512 513 513 TP_printk("comm=%s pid=%d delay=%Lu [ns]", 514 - __entry->comm, __entry->pid, 514 + __get_str(comm), __entry->pid, 515 515 (unsigned long long)__entry->delay) 516 516 ); 517 517 ··· 557 557 TP_ARGS(tsk, __perf_count(runtime)), 558 558 559 559 TP_STRUCT__entry( 560 - __array( char, comm, TASK_COMM_LEN ) 561 - __field( pid_t, pid ) 562 - __field( u64, runtime ) 560 + __string( comm, tsk->comm ) 561 + __field( pid_t, pid ) 562 + __field( u64, runtime ) 563 563 ), 564 564 565 565 TP_fast_assign( 566 - memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); 566 + __assign_str(comm); 567 567 __entry->pid = tsk->pid; 568 568 __entry->runtime = runtime; 569 569 ), 570 570 571 571 TP_printk("comm=%s pid=%d runtime=%Lu [ns]", 572 - __entry->comm, __entry->pid, 572 + __get_str(comm), __entry->pid, 573 573 (unsigned long long)__entry->runtime) 574 574 ); 575 575 ··· 588 588 TP_ARGS(tsk, pi_task), 589 589 590 590 TP_STRUCT__entry( 591 - __array( char, comm, TASK_COMM_LEN ) 592 - __field( pid_t, pid ) 593 - __field( int, oldprio ) 594 - __field( int, newprio ) 591 + __string( comm, tsk->comm ) 592 + __field( pid_t, pid ) 593 + __field( int, oldprio ) 594 + __field( int, newprio ) 595 595 ), 596 596 597 597 TP_fast_assign( 598 - memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); 598 + __assign_str(comm); 599 599 __entry->pid = tsk->pid; 600 600 __entry->oldprio = tsk->prio; 601 601 __entry->newprio = pi_task ? ··· 605 605 ), 606 606 607 607 TP_printk("comm=%s pid=%d oldprio=%d newprio=%d", 608 - __entry->comm, __entry->pid, 608 + __get_str(comm), __entry->pid, 609 609 __entry->oldprio, __entry->newprio) 610 610 ); 611 611 ··· 615 615 TP_ARGS(tsk), 616 616 617 617 TP_STRUCT__entry( 618 - __array( char, comm, TASK_COMM_LEN ) 619 - __field( pid_t, pid ) 618 + __string( comm, tsk->comm ) 619 + __field( pid_t, pid ) 620 620 ), 621 621 622 622 TP_fast_assign( 623 - memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); 623 + __assign_str(comm); 624 624 __entry->pid = tsk->pid; 625 625 ), 626 626 627 - TP_printk("comm=%s pid=%d", __entry->comm, __entry->pid) 627 + TP_printk("comm=%s pid=%d", __get_str(comm), __entry->pid) 628 628 ); 629 629 #endif /* CONFIG_DETECT_HUNG_TASK */ 630 630 ··· 799 799 * 800 800 * Postfixed with _tp to make them easily identifiable in the code. 801 801 */ 802 - DECLARE_TRACE(pelt_cfs_tp, 802 + DECLARE_TRACE(pelt_cfs, 803 803 TP_PROTO(struct cfs_rq *cfs_rq), 804 804 TP_ARGS(cfs_rq)); 805 805 806 - DECLARE_TRACE(pelt_rt_tp, 806 + DECLARE_TRACE(pelt_rt, 807 807 TP_PROTO(struct rq *rq), 808 808 TP_ARGS(rq)); 809 809 810 - DECLARE_TRACE(pelt_dl_tp, 810 + DECLARE_TRACE(pelt_dl, 811 811 TP_PROTO(struct rq *rq), 812 812 TP_ARGS(rq)); 813 813 814 - DECLARE_TRACE(pelt_hw_tp, 814 + DECLARE_TRACE(pelt_hw, 815 815 TP_PROTO(struct rq *rq), 816 816 TP_ARGS(rq)); 817 817 818 - DECLARE_TRACE(pelt_irq_tp, 818 + DECLARE_TRACE(pelt_irq, 819 819 TP_PROTO(struct rq *rq), 820 820 TP_ARGS(rq)); 821 821 822 - DECLARE_TRACE(pelt_se_tp, 822 + DECLARE_TRACE(pelt_se, 823 823 TP_PROTO(struct sched_entity *se), 824 824 TP_ARGS(se)); 825 825 826 - DECLARE_TRACE(sched_cpu_capacity_tp, 826 + DECLARE_TRACE(sched_cpu_capacity, 827 827 TP_PROTO(struct rq *rq), 828 828 TP_ARGS(rq)); 829 829 830 - DECLARE_TRACE(sched_overutilized_tp, 830 + DECLARE_TRACE(sched_overutilized, 831 831 TP_PROTO(struct root_domain *rd, bool overutilized), 832 832 TP_ARGS(rd, overutilized)); 833 833 834 - DECLARE_TRACE(sched_util_est_cfs_tp, 834 + DECLARE_TRACE(sched_util_est_cfs, 835 835 TP_PROTO(struct cfs_rq *cfs_rq), 836 836 TP_ARGS(cfs_rq)); 837 837 838 - DECLARE_TRACE(sched_util_est_se_tp, 838 + DECLARE_TRACE(sched_util_est_se, 839 839 TP_PROTO(struct sched_entity *se), 840 840 TP_ARGS(se)); 841 841 842 - DECLARE_TRACE(sched_update_nr_running_tp, 842 + DECLARE_TRACE(sched_update_nr_running, 843 843 TP_PROTO(struct rq *rq, int change), 844 844 TP_ARGS(rq, change)); 845 845 846 - DECLARE_TRACE(sched_compute_energy_tp, 846 + DECLARE_TRACE(sched_compute_energy, 847 847 TP_PROTO(struct task_struct *p, int dst_cpu, unsigned long energy, 848 848 unsigned long max_util, unsigned long busy_time), 849 849 TP_ARGS(p, dst_cpu, energy, max_util, busy_time)); 850 850 851 - DECLARE_TRACE(sched_entry_tp, 851 + DECLARE_TRACE(sched_entry, 852 852 TP_PROTO(bool preempt, unsigned long ip), 853 853 TP_ARGS(preempt, ip)); 854 854 855 - DECLARE_TRACE(sched_exit_tp, 855 + DECLARE_TRACE(sched_exit, 856 856 TP_PROTO(bool is_switch, unsigned long ip), 857 857 TP_ARGS(is_switch, ip)); 858 858 859 - DECLARE_TRACE_CONDITION(sched_set_state_tp, 859 + DECLARE_TRACE_CONDITION(sched_set_state, 860 860 TP_PROTO(struct task_struct *tsk, int state), 861 861 TP_ARGS(tsk, state), 862 862 TP_CONDITION(!!(tsk->__state) != !!state));
+1 -1
include/trace/events/tcp.h
··· 356 356 __entry->skb_addr, __entry->skb_len, __entry->msg_left, 357 357 __entry->size_goal)); 358 358 359 - DECLARE_TRACE(tcp_cwnd_reduction_tp, 359 + DECLARE_TRACE(tcp_cwnd_reduction, 360 360 TP_PROTO(const struct sock *sk, int newly_acked_sacked, 361 361 int newly_lost, int flag), 362 362 TP_ARGS(sk, newly_acked_sacked, newly_lost, flag)
+2
kernel/trace/fgraph.c
··· 1382 1382 /* Always save the function, and reset at unregistering */ 1383 1383 gops->saved_func = gops->entryfunc; 1384 1384 1385 + gops->ops.flags |= FTRACE_OPS_FL_GRAPH; 1386 + 1385 1387 ret = ftrace_startup_subops(&graph_ops, &gops->ops, command); 1386 1388 if (!ret) 1387 1389 fgraph_array[i] = gops;
+44 -1
kernel/trace/ftrace.c
··· 4373 4373 } 4374 4374 #endif 4375 4375 4376 + static void print_subops(struct seq_file *m, struct ftrace_ops *ops, struct dyn_ftrace *rec) 4377 + { 4378 + struct ftrace_ops *subops; 4379 + bool first = true; 4380 + 4381 + list_for_each_entry(subops, &ops->subop_list, list) { 4382 + if (!((subops->flags & FTRACE_OPS_FL_ENABLED) && 4383 + hash_contains_ip(rec->ip, subops->func_hash))) 4384 + continue; 4385 + if (first) { 4386 + seq_printf(m, "\tsubops:"); 4387 + first = false; 4388 + } 4389 + #ifdef CONFIG_FUNCTION_GRAPH_TRACER 4390 + if (subops->flags & FTRACE_OPS_FL_GRAPH) { 4391 + struct fgraph_ops *gops; 4392 + 4393 + gops = container_of(subops, struct fgraph_ops, ops); 4394 + seq_printf(m, " {ent:%pS ret:%pS}", 4395 + (void *)gops->entryfunc, 4396 + (void *)gops->retfunc); 4397 + continue; 4398 + } 4399 + #endif 4400 + if (subops->trampoline) { 4401 + seq_printf(m, " {%pS (%pS)}", 4402 + (void *)subops->trampoline, 4403 + (void *)subops->func); 4404 + add_trampoline_func(m, subops, rec); 4405 + } else { 4406 + seq_printf(m, " {%pS}", 4407 + (void *)subops->func); 4408 + } 4409 + } 4410 + } 4411 + 4376 4412 static int t_show(struct seq_file *m, void *v) 4377 4413 { 4378 4414 struct ftrace_iterator *iter = m->private; ··· 4461 4425 (void *)ops->trampoline, 4462 4426 (void *)ops->func); 4463 4427 add_trampoline_func(m, ops, rec); 4428 + print_subops(m, ops, rec); 4464 4429 ops = ftrace_find_tramp_ops_next(rec, ops); 4465 4430 } while (ops); 4466 4431 } else ··· 4474 4437 if (ops) { 4475 4438 seq_printf(m, "\tops: %pS (%pS)", 4476 4439 ops, ops->func); 4440 + print_subops(m, ops, rec); 4477 4441 } else { 4478 4442 seq_puts(m, "\tops: ERROR!"); 4479 4443 } ··· 5208 5170 void *data; 5209 5171 }; 5210 5172 5173 + /* 5174 + * Note, ftrace_func_mapper is freed by free_ftrace_hash(&mapper->hash). 5175 + * The hash field must be the first field. 5176 + */ 5211 5177 struct ftrace_func_mapper { 5212 - struct ftrace_hash hash; 5178 + struct ftrace_hash hash; /* Must be first! */ 5213 5179 }; 5214 5180 5215 5181 /** ··· 5348 5306 } 5349 5307 } 5350 5308 } 5309 + /* This also frees the mapper itself */ 5351 5310 free_ftrace_hash(&mapper->hash); 5352 5311 } 5353 5312
+2 -6
kernel/trace/pid_list.c
··· 81 81 { 82 82 /* 83 83 * If chunk->data has no lower chunks, it will be the same 84 - * as a zeroed bitmask. Use find_first_bit() to test it 85 - * and if it doesn't find any bits set, then the array 86 - * is empty. 84 + * as a zeroed bitmask. 87 85 */ 88 - int bit = find_first_bit((unsigned long *)chunk->data, 89 - sizeof(chunk->data) * 8); 90 - return bit >= sizeof(chunk->data) * 8; 86 + return bitmap_empty((unsigned long *)chunk->data, BITS_PER_TYPE(chunk->data)); 91 87 } 92 88 93 89 static inline int pid_split(unsigned int pid, unsigned int *upper1,
+18
kernel/trace/ring_buffer.c
··· 4885 4885 } 4886 4886 4887 4887 /** 4888 + * ring_buffer_record_is_on_cpu - return true if the ring buffer can write 4889 + * @buffer: The ring buffer to see if write is enabled 4890 + * @cpu: The CPU to test if the ring buffer can write too 4891 + * 4892 + * Returns true if the ring buffer is in a state that it accepts writes 4893 + * for a particular CPU. 4894 + */ 4895 + bool ring_buffer_record_is_on_cpu(struct trace_buffer *buffer, int cpu) 4896 + { 4897 + struct ring_buffer_per_cpu *cpu_buffer; 4898 + 4899 + cpu_buffer = buffer->buffers[cpu]; 4900 + 4901 + return ring_buffer_record_is_set_on(buffer) && 4902 + !atomic_read(&cpu_buffer->record_disabled); 4903 + } 4904 + 4905 + /** 4888 4906 * ring_buffer_record_disable_cpu - stop all writes into the cpu_buffer 4889 4907 * @buffer: The ring buffer to stop writes to. 4890 4908 * @cpu: The CPU buffer to stop
+172 -42
kernel/trace/trace.c
··· 527 527 TRACE_ITER_ANNOTATE | TRACE_ITER_CONTEXT_INFO | \ 528 528 TRACE_ITER_RECORD_CMD | TRACE_ITER_OVERWRITE | \ 529 529 TRACE_ITER_IRQ_INFO | TRACE_ITER_MARKERS | \ 530 - TRACE_ITER_HASH_PTR | TRACE_ITER_TRACE_PRINTK) 530 + TRACE_ITER_HASH_PTR | TRACE_ITER_TRACE_PRINTK | \ 531 + TRACE_ITER_COPY_MARKER) 531 532 532 533 /* trace_options that are only supported by global_trace */ 533 534 #define TOP_LEVEL_TRACE_FLAGS (TRACE_ITER_PRINTK | \ ··· 536 535 537 536 /* trace_flags that are default zero for instances */ 538 537 #define ZEROED_TRACE_FLAGS \ 539 - (TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK | TRACE_ITER_TRACE_PRINTK) 538 + (TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK | TRACE_ITER_TRACE_PRINTK | \ 539 + TRACE_ITER_COPY_MARKER) 540 540 541 541 /* 542 542 * The global_trace is the descriptor that holds the top-level tracing ··· 548 546 }; 549 547 550 548 static struct trace_array *printk_trace = &global_trace; 549 + 550 + /* List of trace_arrays interested in the top level trace_marker */ 551 + static LIST_HEAD(marker_copies); 551 552 552 553 static __always_inline bool printk_binsafe(struct trace_array *tr) 553 554 { ··· 571 566 printk_trace->trace_flags &= ~TRACE_ITER_TRACE_PRINTK; 572 567 printk_trace = tr; 573 568 tr->trace_flags |= TRACE_ITER_TRACE_PRINTK; 569 + } 570 + 571 + /* Returns true if the status of tr changed */ 572 + static bool update_marker_trace(struct trace_array *tr, int enabled) 573 + { 574 + lockdep_assert_held(&event_mutex); 575 + 576 + if (enabled) { 577 + if (!list_empty(&tr->marker_list)) 578 + return false; 579 + 580 + list_add_rcu(&tr->marker_list, &marker_copies); 581 + tr->trace_flags |= TRACE_ITER_COPY_MARKER; 582 + return true; 583 + } 584 + 585 + if (list_empty(&tr->marker_list)) 586 + return false; 587 + 588 + list_del_init(&tr->marker_list); 589 + tr->trace_flags &= ~TRACE_ITER_COPY_MARKER; 590 + return true; 574 591 } 575 592 576 593 void trace_set_ring_buffer_expanded(struct trace_array *tr) ··· 1641 1614 tr->buffer_disabled = 1; 1642 1615 /* Make the flag seen by readers */ 1643 1616 smp_wmb(); 1617 + } 1618 + 1619 + /** 1620 + * tracer_tracing_disable() - temporary disable the buffer from write 1621 + * @tr: The trace array to disable its buffer for 1622 + * 1623 + * Expects trace_tracing_enable() to re-enable tracing. 1624 + * The difference between this and tracer_tracing_off() is that this 1625 + * is a counter and can nest, whereas, tracer_tracing_off() can 1626 + * be called multiple times and a single trace_tracing_on() will 1627 + * enable it. 1628 + */ 1629 + void tracer_tracing_disable(struct trace_array *tr) 1630 + { 1631 + if (WARN_ON_ONCE(!tr->array_buffer.buffer)) 1632 + return; 1633 + 1634 + ring_buffer_record_disable(tr->array_buffer.buffer); 1635 + } 1636 + 1637 + /** 1638 + * tracer_tracing_enable() - counter part of tracer_tracing_disable() 1639 + * @tr: The trace array that had tracer_tracincg_disable() called on it 1640 + * 1641 + * This is called after tracer_tracing_disable() has been called on @tr, 1642 + * when it's safe to re-enable tracing. 1643 + */ 1644 + void tracer_tracing_enable(struct trace_array *tr) 1645 + { 1646 + if (WARN_ON_ONCE(!tr->array_buffer.buffer)) 1647 + return; 1648 + 1649 + ring_buffer_record_enable(tr->array_buffer.buffer); 1644 1650 } 1645 1651 1646 1652 /** ··· 5142 5082 */ 5143 5083 if (cpumask_test_cpu(cpu, tr->tracing_cpumask) && 5144 5084 !cpumask_test_cpu(cpu, tracing_cpumask_new)) { 5145 - atomic_inc(&per_cpu_ptr(tr->array_buffer.data, cpu)->disabled); 5146 5085 ring_buffer_record_disable_cpu(tr->array_buffer.buffer, cpu); 5147 5086 #ifdef CONFIG_TRACER_MAX_TRACE 5148 5087 ring_buffer_record_disable_cpu(tr->max_buffer.buffer, cpu); ··· 5149 5090 } 5150 5091 if (!cpumask_test_cpu(cpu, tr->tracing_cpumask) && 5151 5092 cpumask_test_cpu(cpu, tracing_cpumask_new)) { 5152 - atomic_dec(&per_cpu_ptr(tr->array_buffer.data, cpu)->disabled); 5153 5093 ring_buffer_record_enable_cpu(tr->array_buffer.buffer, cpu); 5154 5094 #ifdef CONFIG_TRACER_MAX_TRACE 5155 5095 ring_buffer_record_enable_cpu(tr->max_buffer.buffer, cpu); ··· 5281 5223 { 5282 5224 if ((mask == TRACE_ITER_RECORD_TGID) || 5283 5225 (mask == TRACE_ITER_RECORD_CMD) || 5284 - (mask == TRACE_ITER_TRACE_PRINTK)) 5226 + (mask == TRACE_ITER_TRACE_PRINTK) || 5227 + (mask == TRACE_ITER_COPY_MARKER)) 5285 5228 lockdep_assert_held(&event_mutex); 5286 5229 5287 5230 /* do nothing if flag is already set */ ··· 5312 5253 update_printk_trace(&global_trace); 5313 5254 } 5314 5255 } 5256 + 5257 + if (mask == TRACE_ITER_COPY_MARKER) 5258 + update_marker_trace(tr, enabled); 5315 5259 5316 5260 if (enabled) 5317 5261 tr->trace_flags |= mask; ··· 6100 6038 }; 6101 6039 6102 6040 struct trace_scratch { 6041 + unsigned int clock_id; 6103 6042 unsigned long text_addr; 6104 6043 unsigned long nr_entries; 6105 6044 struct trace_mod_entry entries[]; ··· 6129 6066 struct trace_module_delta *module_delta; 6130 6067 struct trace_scratch *tscratch; 6131 6068 struct trace_mod_entry *entry; 6069 + unsigned long raddr; 6132 6070 int idx = 0, nr_entries; 6133 6071 6134 6072 /* If we don't have last boot delta, return the address */ ··· 6143 6079 module_delta = READ_ONCE(tr->module_delta); 6144 6080 if (!module_delta || !tscratch->nr_entries || 6145 6081 tscratch->entries[0].mod_addr > addr) { 6146 - return addr + tr->text_delta; 6082 + raddr = addr + tr->text_delta; 6083 + return __is_kernel(raddr) || is_kernel_core_data(raddr) || 6084 + is_kernel_rodata(raddr) ? raddr : addr; 6147 6085 } 6148 6086 6149 6087 /* Note that entries must be sorted. */ ··· 6216 6150 if (tr->scratch) { 6217 6151 struct trace_scratch *tscratch = tr->scratch; 6218 6152 6153 + tscratch->clock_id = tr->clock_id; 6219 6154 memset(tscratch->entries, 0, 6220 6155 flex_array_size(tscratch, entries, tscratch->nr_entries)); 6221 6156 tscratch->nr_entries = 0; ··· 6925 6858 ret = trace_seq_to_buffer(&iter->seq, 6926 6859 page_address(spd.pages[i]), 6927 6860 min((size_t)trace_seq_used(&iter->seq), 6928 - PAGE_SIZE)); 6861 + (size_t)PAGE_SIZE)); 6929 6862 if (ret < 0) { 6930 6863 __free_page(spd.pages[i]); 6931 6864 break; ··· 7201 7134 7202 7135 #define TRACE_MARKER_MAX_SIZE 4096 7203 7136 7204 - static ssize_t 7205 - tracing_mark_write(struct file *filp, const char __user *ubuf, 7206 - size_t cnt, loff_t *fpos) 7137 + static ssize_t write_marker_to_buffer(struct trace_array *tr, const char __user *ubuf, 7138 + size_t cnt, unsigned long ip) 7207 7139 { 7208 - struct trace_array *tr = filp->private_data; 7209 7140 struct ring_buffer_event *event; 7210 7141 enum event_trigger_type tt = ETT_NONE; 7211 7142 struct trace_buffer *buffer; ··· 7216 7151 /* Used in tracing_mark_raw_write() as well */ 7217 7152 #define FAULTED_STR "<faulted>" 7218 7153 #define FAULTED_SIZE (sizeof(FAULTED_STR) - 1) /* '\0' is already accounted for */ 7219 - 7220 - if (tracing_disabled) 7221 - return -EINVAL; 7222 - 7223 - if (!(tr->trace_flags & TRACE_ITER_MARKERS)) 7224 - return -EINVAL; 7225 - 7226 - if ((ssize_t)cnt < 0) 7227 - return -EINVAL; 7228 - 7229 - if (cnt > TRACE_MARKER_MAX_SIZE) 7230 - cnt = TRACE_MARKER_MAX_SIZE; 7231 7154 7232 7155 meta_size = sizeof(*entry) + 2; /* add '\0' and possible '\n' */ 7233 7156 again: ··· 7249 7196 } 7250 7197 7251 7198 entry = ring_buffer_event_data(event); 7252 - entry->ip = _THIS_IP_; 7199 + entry->ip = ip; 7253 7200 7254 7201 len = __copy_from_user_inatomic(&entry->buf, ubuf, cnt); 7255 7202 if (len) { ··· 7282 7229 } 7283 7230 7284 7231 static ssize_t 7285 - tracing_mark_raw_write(struct file *filp, const char __user *ubuf, 7232 + tracing_mark_write(struct file *filp, const char __user *ubuf, 7286 7233 size_t cnt, loff_t *fpos) 7287 7234 { 7288 7235 struct trace_array *tr = filp->private_data; 7236 + ssize_t written = -ENODEV; 7237 + unsigned long ip; 7238 + 7239 + if (tracing_disabled) 7240 + return -EINVAL; 7241 + 7242 + if (!(tr->trace_flags & TRACE_ITER_MARKERS)) 7243 + return -EINVAL; 7244 + 7245 + if ((ssize_t)cnt < 0) 7246 + return -EINVAL; 7247 + 7248 + if (cnt > TRACE_MARKER_MAX_SIZE) 7249 + cnt = TRACE_MARKER_MAX_SIZE; 7250 + 7251 + /* The selftests expect this function to be the IP address */ 7252 + ip = _THIS_IP_; 7253 + 7254 + /* The global trace_marker can go to multiple instances */ 7255 + if (tr == &global_trace) { 7256 + guard(rcu)(); 7257 + list_for_each_entry_rcu(tr, &marker_copies, marker_list) { 7258 + written = write_marker_to_buffer(tr, ubuf, cnt, ip); 7259 + if (written < 0) 7260 + break; 7261 + } 7262 + } else { 7263 + written = write_marker_to_buffer(tr, ubuf, cnt, ip); 7264 + } 7265 + 7266 + return written; 7267 + } 7268 + 7269 + static ssize_t write_raw_marker_to_buffer(struct trace_array *tr, 7270 + const char __user *ubuf, size_t cnt) 7271 + { 7289 7272 struct ring_buffer_event *event; 7290 7273 struct trace_buffer *buffer; 7291 7274 struct raw_data_entry *entry; ··· 7330 7241 int len; 7331 7242 7332 7243 #define FAULT_SIZE_ID (FAULTED_SIZE + sizeof(int)) 7333 - 7334 - if (tracing_disabled) 7335 - return -EINVAL; 7336 - 7337 - if (!(tr->trace_flags & TRACE_ITER_MARKERS)) 7338 - return -EINVAL; 7339 - 7340 - /* The marker must at least have a tag id */ 7341 - if (cnt < sizeof(unsigned int)) 7342 - return -EINVAL; 7343 7244 7344 7245 size = sizeof(*entry) + cnt; 7345 7246 if (cnt < FAULT_SIZE_ID) ··· 7357 7278 written = cnt; 7358 7279 7359 7280 __buffer_unlock_commit(buffer, event); 7281 + 7282 + return written; 7283 + } 7284 + 7285 + static ssize_t 7286 + tracing_mark_raw_write(struct file *filp, const char __user *ubuf, 7287 + size_t cnt, loff_t *fpos) 7288 + { 7289 + struct trace_array *tr = filp->private_data; 7290 + ssize_t written = -ENODEV; 7291 + 7292 + #define FAULT_SIZE_ID (FAULTED_SIZE + sizeof(int)) 7293 + 7294 + if (tracing_disabled) 7295 + return -EINVAL; 7296 + 7297 + if (!(tr->trace_flags & TRACE_ITER_MARKERS)) 7298 + return -EINVAL; 7299 + 7300 + /* The marker must at least have a tag id */ 7301 + if (cnt < sizeof(unsigned int)) 7302 + return -EINVAL; 7303 + 7304 + /* The global trace_marker_raw can go to multiple instances */ 7305 + if (tr == &global_trace) { 7306 + guard(rcu)(); 7307 + list_for_each_entry_rcu(tr, &marker_copies, marker_list) { 7308 + written = write_raw_marker_to_buffer(tr, ubuf, cnt); 7309 + if (written < 0) 7310 + break; 7311 + } 7312 + } else { 7313 + written = write_raw_marker_to_buffer(tr, ubuf, cnt); 7314 + } 7360 7315 7361 7316 return written; 7362 7317 } ··· 7438 7325 ring_buffer_set_clock(tr->max_buffer.buffer, trace_clocks[i].func); 7439 7326 tracing_reset_online_cpus(&tr->max_buffer); 7440 7327 #endif 7328 + 7329 + if (tr->scratch && !(tr->flags & TRACE_ARRAY_FL_LAST_BOOT)) { 7330 + struct trace_scratch *tscratch = tr->scratch; 7331 + 7332 + tscratch->clock_id = i; 7333 + } 7441 7334 7442 7335 mutex_unlock(&trace_types_lock); 7443 7336 ··· 9670 9551 9671 9552 /* Scan modules to make text delta for modules. */ 9672 9553 module_for_each_mod(make_mod_delta, tr); 9554 + 9555 + /* Set trace_clock as the same of the previous boot. */ 9556 + if (tscratch->clock_id != tr->clock_id) { 9557 + if (tscratch->clock_id >= ARRAY_SIZE(trace_clocks) || 9558 + tracing_set_clock(tr, trace_clocks[tscratch->clock_id].name) < 0) { 9559 + pr_info("the previous trace_clock info is not valid."); 9560 + goto reset; 9561 + } 9562 + } 9673 9563 return; 9674 9564 reset: 9675 9565 /* Invalid trace modules */ ··· 9903 9775 INIT_LIST_HEAD(&tr->events); 9904 9776 INIT_LIST_HEAD(&tr->hist_vars); 9905 9777 INIT_LIST_HEAD(&tr->err_log); 9778 + INIT_LIST_HEAD(&tr->marker_list); 9906 9779 9907 9780 #ifdef CONFIG_MODULES 9908 9781 INIT_LIST_HEAD(&tr->mod_events); ··· 10062 9933 10063 9934 if (printk_trace == tr) 10064 9935 update_printk_trace(&global_trace); 9936 + 9937 + if (update_marker_trace(tr, 0)) 9938 + synchronize_rcu(); 10065 9939 10066 9940 tracing_set_nop(tr); 10067 9941 clear_ftrace_function_probes(tr); ··· 10614 10482 static struct trace_iterator iter; 10615 10483 unsigned int old_userobj; 10616 10484 unsigned long flags; 10617 - int cnt = 0, cpu; 10485 + int cnt = 0; 10618 10486 10619 10487 /* 10620 10488 * Always turn off tracing when we dump. ··· 10631 10499 /* Simulate the iterator */ 10632 10500 trace_init_iter(&iter, tr); 10633 10501 10634 - for_each_tracing_cpu(cpu) { 10635 - atomic_inc(&per_cpu_ptr(iter.array_buffer->data, cpu)->disabled); 10636 - } 10502 + /* While dumping, do not allow the buffer to be enable */ 10503 + tracer_tracing_disable(tr); 10637 10504 10638 10505 old_userobj = tr->trace_flags & TRACE_ITER_SYM_USEROBJ; 10639 10506 ··· 10691 10560 10692 10561 tr->trace_flags |= old_userobj; 10693 10562 10694 - for_each_tracing_cpu(cpu) { 10695 - atomic_dec(&per_cpu_ptr(iter.array_buffer->data, cpu)->disabled); 10696 - } 10563 + tracer_tracing_enable(tr); 10697 10564 local_irq_restore(flags); 10698 10565 } 10699 10566 ··· 11131 11002 INIT_LIST_HEAD(&global_trace.events); 11132 11003 INIT_LIST_HEAD(&global_trace.hist_vars); 11133 11004 INIT_LIST_HEAD(&global_trace.err_log); 11005 + list_add(&global_trace.marker_list, &marker_copies); 11134 11006 list_add(&global_trace.list, &ftrace_trace_arrays); 11135 11007 11136 11008 apply_trace_boot_options();
+23 -7
kernel/trace/trace.h
··· 183 183 * the trace, etc.) 184 184 */ 185 185 struct trace_array_cpu { 186 - atomic_t disabled; 187 - void *buffer_page; /* ring buffer spare */ 186 + local_t disabled; 188 187 189 188 unsigned long entries; 190 189 unsigned long saved_latency; ··· 403 404 struct trace_options *topts; 404 405 struct list_head systems; 405 406 struct list_head events; 407 + struct list_head marker_list; 406 408 struct trace_event_file *trace_marker_file; 407 409 cpumask_var_t tracing_cpumask; /* only trace on set CPUs */ 408 410 /* one per_cpu trace_pipe can be opened by only one user */ ··· 665 665 bool tracer_tracing_is_on(struct trace_array *tr); 666 666 void tracer_tracing_on(struct trace_array *tr); 667 667 void tracer_tracing_off(struct trace_array *tr); 668 + void tracer_tracing_disable(struct trace_array *tr); 669 + void tracer_tracing_enable(struct trace_array *tr); 668 670 struct dentry *trace_create_file(const char *name, 669 671 umode_t mode, 670 672 struct dentry *parent, 671 673 void *data, 672 674 const struct file_operations *fops); 675 + 676 + 677 + /** 678 + * tracer_tracing_is_on_cpu - show real state of ring buffer enabled on for a cpu 679 + * @tr : the trace array to know if ring buffer is enabled 680 + * @cpu: The cpu buffer to check if enabled 681 + * 682 + * Shows real state of the per CPU buffer if it is enabled or not. 683 + */ 684 + static inline bool tracer_tracing_is_on_cpu(struct trace_array *tr, int cpu) 685 + { 686 + if (tr->array_buffer.buffer) 687 + return ring_buffer_record_is_on_cpu(tr->array_buffer.buffer, cpu); 688 + return false; 689 + } 673 690 674 691 int tracing_init_dentry(void); 675 692 ··· 1385 1368 C(MARKERS, "markers"), \ 1386 1369 C(EVENT_FORK, "event-fork"), \ 1387 1370 C(TRACE_PRINTK, "trace_printk_dest"), \ 1371 + C(COPY_MARKER, "copy_trace_marker"),\ 1388 1372 C(PAUSE_ON_TRACE, "pause-on-trace"), \ 1389 1373 C(HASH_PTR, "hash-ptr"), /* Print hashed pointer */ \ 1390 1374 FUNCTION_FLAGS \ ··· 1790 1772 extern void event_enable_unregister_trigger(char *glob, 1791 1773 struct event_trigger_data *test, 1792 1774 struct trace_event_file *file); 1775 + extern struct event_trigger_data * 1776 + trigger_data_alloc(struct event_command *cmd_ops, char *cmd, char *param, 1777 + void *private_data); 1793 1778 extern void trigger_data_free(struct event_trigger_data *data); 1794 1779 extern int event_trigger_init(struct event_trigger_data *data); 1795 1780 extern int trace_event_trigger_enable_disable(struct trace_event_file *file, ··· 1819 1798 extern bool event_trigger_empty_param(const char *param); 1820 1799 extern int event_trigger_separate_filter(char *param_and_filter, char **param, 1821 1800 char **filter, bool param_required); 1822 - extern struct event_trigger_data * 1823 - event_trigger_alloc(struct event_command *cmd_ops, 1824 - char *cmd, 1825 - char *param, 1826 - void *private_data); 1827 1801 extern int event_trigger_parse_num(char *trigger, 1828 1802 struct event_trigger_data *trigger_data); 1829 1803 extern int event_trigger_set_filter(struct event_command *cmd_ops,
+1 -3
kernel/trace/trace_branch.c
··· 32 32 { 33 33 struct trace_array *tr = branch_tracer; 34 34 struct trace_buffer *buffer; 35 - struct trace_array_cpu *data; 36 35 struct ring_buffer_event *event; 37 36 struct trace_branch *entry; 38 37 unsigned long flags; ··· 53 54 54 55 raw_local_irq_save(flags); 55 56 current->trace_recursion |= TRACE_BRANCH_BIT; 56 - data = this_cpu_ptr(tr->array_buffer.data); 57 - if (atomic_read(&data->disabled)) 57 + if (!tracer_tracing_is_on_cpu(tr, raw_smp_processor_id())) 58 58 goto out; 59 59 60 60 trace_ctx = tracing_gen_ctx_flags(flags);
+6 -6
kernel/trace/trace_entries.h
··· 97 97 F_STRUCT( 98 98 __field_struct( struct fgraph_retaddr_ent, graph_ent ) 99 99 __field_packed( unsigned long, graph_ent, func ) 100 - __field_packed( int, graph_ent, depth ) 100 + __field_packed( unsigned int, graph_ent, depth ) 101 101 __field_packed( unsigned long, graph_ent, retaddr ) 102 102 ), 103 103 104 - F_printk("--> %ps (%d) <- %ps", (void *)__entry->func, __entry->depth, 104 + F_printk("--> %ps (%u) <- %ps", (void *)__entry->func, __entry->depth, 105 105 (void *)__entry->retaddr) 106 106 ); 107 107 ··· 124 124 __field_struct( struct ftrace_graph_ret, ret ) 125 125 __field_packed( unsigned long, ret, func ) 126 126 __field_packed( unsigned long, ret, retval ) 127 - __field_packed( int, ret, depth ) 127 + __field_packed( unsigned int, ret, depth ) 128 128 __field_packed( unsigned int, ret, overrun ) 129 129 __field(unsigned long long, calltime ) 130 130 __field(unsigned long long, rettime ) 131 131 ), 132 132 133 - F_printk("<-- %ps (%d) (start: %llx end: %llx) over: %d retval: %lx", 133 + F_printk("<-- %ps (%u) (start: %llx end: %llx) over: %u retval: %lx", 134 134 (void *)__entry->func, __entry->depth, 135 135 __entry->calltime, __entry->rettime, 136 136 __entry->depth, __entry->retval) ··· 146 146 F_STRUCT( 147 147 __field_struct( struct ftrace_graph_ret, ret ) 148 148 __field_packed( unsigned long, ret, func ) 149 - __field_packed( int, ret, depth ) 149 + __field_packed( unsigned int, ret, depth ) 150 150 __field_packed( unsigned int, ret, overrun ) 151 151 __field(unsigned long long, calltime ) 152 152 __field(unsigned long long, rettime ) 153 153 ), 154 154 155 - F_printk("<-- %ps (%d) (start: %llx end: %llx) over: %d", 155 + F_printk("<-- %ps (%u) (start: %llx end: %llx) over: %u", 156 156 (void *)__entry->func, __entry->depth, 157 157 __entry->calltime, __entry->rettime, 158 158 __entry->depth)
+25 -14
kernel/trace/trace_events.c
··· 400 400 return true; 401 401 } 402 402 403 + static void handle_dereference_arg(const char *arg_str, u64 string_flags, int len, 404 + u64 *dereference_flags, int arg, 405 + struct trace_event_call *call) 406 + { 407 + if (string_flags & (1ULL << arg)) { 408 + if (process_string(arg_str, len, call)) 409 + *dereference_flags &= ~(1ULL << arg); 410 + } else if (process_pointer(arg_str, len, call)) 411 + *dereference_flags &= ~(1ULL << arg); 412 + else 413 + pr_warn("TRACE EVENT ERROR: Bad dereference argument: '%.*s'\n", 414 + len, arg_str); 415 + } 416 + 403 417 /* 404 418 * Examine the print fmt of the event looking for unsafe dereference 405 419 * pointers using %p* that could be recorded in the trace event and ··· 577 563 } 578 564 579 565 if (dereference_flags & (1ULL << arg)) { 580 - if (string_flags & (1ULL << arg)) { 581 - if (process_string(fmt + start_arg, e - start_arg, call)) 582 - dereference_flags &= ~(1ULL << arg); 583 - } else if (process_pointer(fmt + start_arg, e - start_arg, call)) 584 - dereference_flags &= ~(1ULL << arg); 566 + handle_dereference_arg(fmt + start_arg, string_flags, 567 + e - start_arg, 568 + &dereference_flags, arg, call); 585 569 } 586 570 587 571 start_arg = i; ··· 590 578 } 591 579 592 580 if (dereference_flags & (1ULL << arg)) { 593 - if (string_flags & (1ULL << arg)) { 594 - if (process_string(fmt + start_arg, i - start_arg, call)) 595 - dereference_flags &= ~(1ULL << arg); 596 - } else if (process_pointer(fmt + start_arg, i - start_arg, call)) 597 - dereference_flags &= ~(1ULL << arg); 581 + handle_dereference_arg(fmt + start_arg, string_flags, 582 + i - start_arg, 583 + &dereference_flags, arg, call); 598 584 } 599 585 600 586 /* ··· 632 622 bool trace_event_ignore_this_pid(struct trace_event_file *trace_file) 633 623 { 634 624 struct trace_array *tr = trace_file->tr; 635 - struct trace_array_cpu *data; 636 625 struct trace_pid_list *no_pid_list; 637 626 struct trace_pid_list *pid_list; 638 627 ··· 641 632 if (!pid_list && !no_pid_list) 642 633 return false; 643 634 644 - data = this_cpu_ptr(tr->array_buffer.data); 645 - 646 - return data->ignore_pid; 635 + /* 636 + * This is recorded at every sched_switch for this task. 637 + * Thus, even if the task migrates the ignore value will be the same. 638 + */ 639 + return this_cpu_read(tr->array_buffer.data->ignore_pid) != 0; 647 640 } 648 641 EXPORT_SYMBOL_GPL(trace_event_ignore_this_pid); 649 642
+150 -29
kernel/trace/trace_events_hist.c
··· 114 114 HIST_FIELD_FN_BUCKET, 115 115 HIST_FIELD_FN_TIMESTAMP, 116 116 HIST_FIELD_FN_CPU, 117 + HIST_FIELD_FN_COMM, 117 118 HIST_FIELD_FN_STRING, 118 119 HIST_FIELD_FN_DYNSTRING, 119 120 HIST_FIELD_FN_RELDYNSTRING, ··· 507 506 HIST_FIELD_FL_CONST = 1 << 18, 508 507 HIST_FIELD_FL_PERCENT = 1 << 19, 509 508 HIST_FIELD_FL_GRAPH = 1 << 20, 509 + HIST_FIELD_FL_COMM = 1 << 21, 510 510 }; 511 511 512 512 struct var_defs { ··· 885 883 int cpu = smp_processor_id(); 886 884 887 885 return cpu; 886 + } 887 + 888 + static u64 hist_field_comm(struct hist_field *hist_field, 889 + struct tracing_map_elt *elt, 890 + struct trace_buffer *buffer, 891 + struct ring_buffer_event *rbe, 892 + void *event) 893 + { 894 + return (u64)(unsigned long)current->comm; 888 895 } 889 896 890 897 /** ··· 1349 1338 field_name = hist_field_name(field->operands[0], ++level); 1350 1339 else if (field->flags & HIST_FIELD_FL_CPU) 1351 1340 field_name = "common_cpu"; 1341 + else if (field->flags & HIST_FIELD_FL_COMM) 1342 + field_name = "common_comm"; 1352 1343 else if (field->flags & HIST_FIELD_FL_EXPR || 1353 1344 field->flags & HIST_FIELD_FL_VAR_REF) { 1354 1345 if (field->system) { ··· 2028 2015 goto out; 2029 2016 } 2030 2017 2018 + if (flags & HIST_FIELD_FL_COMM) { 2019 + hist_field->fn_num = HIST_FIELD_FN_COMM; 2020 + hist_field->size = MAX_FILTER_STR_VAL; 2021 + hist_field->type = "char[]"; 2022 + goto out; 2023 + } 2024 + 2031 2025 if (WARN_ON_ONCE(!field)) 2032 2026 goto out; 2033 2027 ··· 2379 2359 hist_data->attrs->ts_in_usecs = true; 2380 2360 } else if (strcmp(field_name, "common_stacktrace") == 0) { 2381 2361 *flags |= HIST_FIELD_FL_STACKTRACE; 2382 - } else if (strcmp(field_name, "common_cpu") == 0) 2362 + } else if (strcmp(field_name, "common_cpu") == 0) { 2383 2363 *flags |= HIST_FIELD_FL_CPU; 2384 - else if (strcmp(field_name, "hitcount") == 0) 2364 + } else if (strcmp(field_name, "common_comm") == 0) { 2365 + *flags |= HIST_FIELD_FL_COMM | HIST_FIELD_FL_STRING; 2366 + } else if (strcmp(field_name, "hitcount") == 0) 2385 2367 *flags |= HIST_FIELD_FL_HITCOUNT; 2386 2368 else { 2387 2369 field = trace_find_event_field(file->event_call, field_name); ··· 2399 2377 *flags |= HIST_FIELD_FL_CPU; 2400 2378 } else if (field && field->filter_type == FILTER_STACKTRACE) { 2401 2379 *flags |= HIST_FIELD_FL_STACKTRACE; 2380 + } else if (field && field->filter_type == FILTER_COMM) { 2381 + *flags |= HIST_FIELD_FL_COMM | HIST_FIELD_FL_STRING; 2402 2382 } else { 2403 2383 hist_err(tr, HIST_ERR_FIELD_NOT_FOUND, 2404 2384 errpos(field_name)); ··· 4351 4327 return hist_field_timestamp(hist_field, elt, buffer, rbe, event); 4352 4328 case HIST_FIELD_FN_CPU: 4353 4329 return hist_field_cpu(hist_field, elt, buffer, rbe, event); 4330 + case HIST_FIELD_FN_COMM: 4331 + return hist_field_comm(hist_field, elt, buffer, rbe, event); 4354 4332 case HIST_FIELD_FN_STRING: 4355 4333 return hist_field_string(hist_field, elt, buffer, rbe, event); 4356 4334 case HIST_FIELD_FN_DYNSTRING: ··· 5238 5212 size_t size = key_field->size; 5239 5213 5240 5214 if (key_field->flags & HIST_FIELD_FL_STRING) { 5241 - struct ftrace_event_field *field; 5242 5215 5243 - field = key_field->field; 5244 - if (field->filter_type == FILTER_DYN_STRING || 5245 - field->filter_type == FILTER_RDYN_STRING) 5246 - size = *(u32 *)(rec + field->offset) >> 16; 5247 - else if (field->filter_type == FILTER_STATIC_STRING) 5248 - size = field->size; 5216 + if (key_field->flags & HIST_FIELD_FL_COMM) { 5217 + size = strlen((char *)key); 5218 + } else { 5219 + struct ftrace_event_field *field; 5220 + 5221 + field = key_field->field; 5222 + if (field->filter_type == FILTER_DYN_STRING || 5223 + field->filter_type == FILTER_RDYN_STRING) 5224 + size = *(u32 *)(rec + field->offset) >> 16; 5225 + else if (field->filter_type == FILTER_STATIC_STRING) 5226 + size = field->size; 5227 + } 5249 5228 5250 5229 /* ensure NULL-termination */ 5251 5230 if (size > key_field->size - 1) 5252 5231 size = key_field->size - 1; 5253 - 5254 - strncpy(compound_key + key_field->offset, (char *)key, size); 5255 - } else 5256 - memcpy(compound_key + key_field->offset, key, size); 5232 + } 5233 + memcpy(compound_key + key_field->offset, key, size); 5257 5234 } 5258 5235 5259 5236 static void ··· 5275 5246 } 5276 5247 } 5277 5248 5249 + /* 5250 + * The hist_pad structure is used to save information to create 5251 + * a histogram from the histogram trigger. It's too big to store 5252 + * on the stack, so when the histogram trigger is initialized 5253 + * a percpu array of 4 hist_pad structures is allocated. 5254 + * This will cover every context from normal, softirq, irq and NMI 5255 + * in the very unlikely event that a tigger happens at each of 5256 + * these contexts and interrupts a currently active trigger. 5257 + */ 5258 + struct hist_pad { 5259 + unsigned long entries[HIST_STACKTRACE_DEPTH]; 5260 + u64 var_ref_vals[TRACING_MAP_VARS_MAX]; 5261 + char compound_key[HIST_KEY_SIZE_MAX]; 5262 + }; 5263 + 5264 + static struct hist_pad __percpu *hist_pads; 5265 + static DEFINE_PER_CPU(int, hist_pad_cnt); 5266 + static refcount_t hist_pad_ref; 5267 + 5268 + /* One hist_pad for every context (normal, softirq, irq, NMI) */ 5269 + #define MAX_HIST_CNT 4 5270 + 5271 + static int alloc_hist_pad(void) 5272 + { 5273 + lockdep_assert_held(&event_mutex); 5274 + 5275 + if (refcount_read(&hist_pad_ref)) { 5276 + refcount_inc(&hist_pad_ref); 5277 + return 0; 5278 + } 5279 + 5280 + hist_pads = __alloc_percpu(sizeof(struct hist_pad) * MAX_HIST_CNT, 5281 + __alignof__(struct hist_pad)); 5282 + if (!hist_pads) 5283 + return -ENOMEM; 5284 + 5285 + refcount_set(&hist_pad_ref, 1); 5286 + return 0; 5287 + } 5288 + 5289 + static void free_hist_pad(void) 5290 + { 5291 + lockdep_assert_held(&event_mutex); 5292 + 5293 + if (!refcount_dec_and_test(&hist_pad_ref)) 5294 + return; 5295 + 5296 + free_percpu(hist_pads); 5297 + hist_pads = NULL; 5298 + } 5299 + 5300 + static struct hist_pad *get_hist_pad(void) 5301 + { 5302 + struct hist_pad *hist_pad; 5303 + int cnt; 5304 + 5305 + if (WARN_ON_ONCE(!hist_pads)) 5306 + return NULL; 5307 + 5308 + preempt_disable(); 5309 + 5310 + hist_pad = per_cpu_ptr(hist_pads, smp_processor_id()); 5311 + 5312 + if (this_cpu_read(hist_pad_cnt) == MAX_HIST_CNT) { 5313 + preempt_enable(); 5314 + return NULL; 5315 + } 5316 + 5317 + cnt = this_cpu_inc_return(hist_pad_cnt) - 1; 5318 + 5319 + return &hist_pad[cnt]; 5320 + } 5321 + 5322 + static void put_hist_pad(void) 5323 + { 5324 + this_cpu_dec(hist_pad_cnt); 5325 + preempt_enable(); 5326 + } 5327 + 5278 5328 static void event_hist_trigger(struct event_trigger_data *data, 5279 5329 struct trace_buffer *buffer, void *rec, 5280 5330 struct ring_buffer_event *rbe) 5281 5331 { 5282 5332 struct hist_trigger_data *hist_data = data->private_data; 5283 5333 bool use_compound_key = (hist_data->n_keys > 1); 5284 - unsigned long entries[HIST_STACKTRACE_DEPTH]; 5285 - u64 var_ref_vals[TRACING_MAP_VARS_MAX]; 5286 - char compound_key[HIST_KEY_SIZE_MAX]; 5287 5334 struct tracing_map_elt *elt = NULL; 5288 5335 struct hist_field *key_field; 5336 + struct hist_pad *hist_pad; 5289 5337 u64 field_contents; 5290 5338 void *key = NULL; 5291 5339 unsigned int i; ··· 5370 5264 if (unlikely(!rbe)) 5371 5265 return; 5372 5266 5373 - memset(compound_key, 0, hist_data->key_size); 5267 + hist_pad = get_hist_pad(); 5268 + if (!hist_pad) 5269 + return; 5270 + 5271 + memset(hist_pad->compound_key, 0, hist_data->key_size); 5374 5272 5375 5273 for_each_hist_key_field(i, hist_data) { 5376 5274 key_field = hist_data->fields[i]; 5377 5275 5378 5276 if (key_field->flags & HIST_FIELD_FL_STACKTRACE) { 5277 + unsigned long *entries = hist_pad->entries; 5278 + 5379 5279 memset(entries, 0, HIST_STACKTRACE_SIZE); 5380 5280 if (key_field->field) { 5381 5281 unsigned long *stack, n_entries; ··· 5405 5293 } 5406 5294 5407 5295 if (use_compound_key) 5408 - add_to_key(compound_key, key, key_field, rec); 5296 + add_to_key(hist_pad->compound_key, key, key_field, rec); 5409 5297 } 5410 5298 5411 5299 if (use_compound_key) 5412 - key = compound_key; 5300 + key = hist_pad->compound_key; 5413 5301 5414 5302 if (hist_data->n_var_refs && 5415 - !resolve_var_refs(hist_data, key, var_ref_vals, false)) 5416 - return; 5303 + !resolve_var_refs(hist_data, key, hist_pad->var_ref_vals, false)) 5304 + goto out; 5417 5305 5418 5306 elt = tracing_map_insert(hist_data->map, key); 5419 5307 if (!elt) 5420 - return; 5308 + goto out; 5421 5309 5422 - hist_trigger_elt_update(hist_data, elt, buffer, rec, rbe, var_ref_vals); 5310 + hist_trigger_elt_update(hist_data, elt, buffer, rec, rbe, hist_pad->var_ref_vals); 5423 5311 5424 - if (resolve_var_refs(hist_data, key, var_ref_vals, true)) 5425 - hist_trigger_actions(hist_data, elt, buffer, rec, rbe, key, var_ref_vals); 5312 + if (resolve_var_refs(hist_data, key, hist_pad->var_ref_vals, true)) { 5313 + hist_trigger_actions(hist_data, elt, buffer, rec, rbe, 5314 + key, hist_pad->var_ref_vals); 5315 + } 5426 5316 5427 5317 hist_poll_wakeup(); 5318 + 5319 + out: 5320 + put_hist_pad(); 5428 5321 } 5429 5322 5430 5323 static void hist_trigger_stacktrace_print(struct seq_file *m, ··· 6128 6011 6129 6012 if (hist_field->flags & HIST_FIELD_FL_CPU) 6130 6013 seq_puts(m, "common_cpu"); 6014 + if (hist_field->flags & HIST_FIELD_FL_COMM) 6015 + seq_puts(m, "common_comm"); 6131 6016 else if (hist_field->flags & HIST_FIELD_FL_CONST) 6132 6017 seq_printf(m, "%llu", hist_field->constant); 6133 6018 else if (field_name) { ··· 6276 6157 { 6277 6158 struct hist_trigger_data *hist_data = data->private_data; 6278 6159 6160 + if (alloc_hist_pad() < 0) 6161 + return -ENOMEM; 6162 + 6279 6163 if (!data->ref && hist_data->attrs->name) 6280 6164 save_named_trigger(hist_data->attrs->name, data); 6281 6165 ··· 6323 6201 6324 6202 destroy_hist_data(hist_data); 6325 6203 } 6204 + free_hist_pad(); 6326 6205 } 6327 6206 6328 6207 static const struct event_trigger_ops event_hist_trigger_ops = { ··· 6339 6216 6340 6217 save_named_trigger(data->named_data->name, data); 6341 6218 6342 - event_hist_trigger_init(data->named_data); 6343 - 6344 - return 0; 6219 + return event_hist_trigger_init(data->named_data); 6345 6220 } 6346 6221 6347 6222 static void event_hist_trigger_named_free(struct event_trigger_data *data) ··· 6826 6705 return PTR_ERR(hist_data); 6827 6706 } 6828 6707 6829 - trigger_data = event_trigger_alloc(cmd_ops, cmd, param, hist_data); 6708 + trigger_data = trigger_data_alloc(cmd_ops, cmd, param, hist_data); 6830 6709 if (!trigger_data) { 6831 6710 ret = -ENOMEM; 6832 6711 goto out_free;
+27 -37
kernel/trace/trace_events_trigger.c
··· 552 552 lockdep_assert_held(&event_mutex); 553 553 554 554 list_for_each_entry(test, &file->triggers, list) { 555 - if (test->cmd_ops->trigger_type == data->cmd_ops->trigger_type) { 556 - ret = -EEXIST; 557 - goto out; 558 - } 555 + if (test->cmd_ops->trigger_type == data->cmd_ops->trigger_type) 556 + return -EEXIST; 559 557 } 560 558 561 559 if (data->ops->init) { 562 560 ret = data->ops->init(data); 563 561 if (ret < 0) 564 - goto out; 562 + return ret; 565 563 } 566 564 567 565 list_add_rcu(&data->list, &file->triggers); ··· 570 572 list_del_rcu(&data->list); 571 573 update_cond_flag(file); 572 574 } 573 - out: 574 575 return ret; 575 576 } 576 577 ··· 767 770 if (!param_and_filter) { 768 771 if (param_required) 769 772 ret = -EINVAL; 770 - goto out; 773 + return ret; 771 774 } 772 775 773 776 /* ··· 778 781 */ 779 782 if (!param_required && param_and_filter && !isdigit(param_and_filter[0])) { 780 783 *filter = param_and_filter; 781 - goto out; 784 + return ret; 782 785 } 783 786 784 787 /* ··· 796 799 if (!**filter) 797 800 *filter = NULL; 798 801 } 799 - out: 800 802 return ret; 801 803 } 802 804 803 805 /** 804 - * event_trigger_alloc - allocate and init event_trigger_data for a trigger 806 + * trigger_data_alloc - allocate and init event_trigger_data for a trigger 805 807 * @cmd_ops: The event_command operations for the trigger 806 808 * @cmd: The cmd string 807 809 * @param: The param string ··· 811 815 * trigger_ops to assign to the event_trigger_data. @private_data can 812 816 * also be passed in and associated with the event_trigger_data. 813 817 * 814 - * Use event_trigger_free() to free an event_trigger_data object. 818 + * Use trigger_data_free() to free an event_trigger_data object. 815 819 * 816 820 * Return: The trigger_data object success, NULL otherwise 817 821 */ 818 - struct event_trigger_data *event_trigger_alloc(struct event_command *cmd_ops, 819 - char *cmd, 820 - char *param, 821 - void *private_data) 822 + struct event_trigger_data *trigger_data_alloc(struct event_command *cmd_ops, 823 + char *cmd, 824 + char *param, 825 + void *private_data) 822 826 { 823 827 struct event_trigger_data *trigger_data; 824 828 const struct event_trigger_ops *trigger_ops; ··· 985 989 return ret; 986 990 987 991 ret = -ENOMEM; 988 - trigger_data = event_trigger_alloc(cmd_ops, cmd, param, file); 992 + trigger_data = trigger_data_alloc(cmd_ops, cmd, param, file); 989 993 if (!trigger_data) 990 - goto out; 994 + return ret; 991 995 992 996 if (remove) { 993 997 event_trigger_unregister(cmd_ops, file, glob+1, trigger_data); 994 - kfree(trigger_data); 995 - ret = 0; 996 - goto out; 998 + trigger_data_free(trigger_data); 999 + return 0; 997 1000 } 998 1001 999 1002 ret = event_trigger_parse_num(param, trigger_data); ··· 1012 1017 1013 1018 /* Down the counter of trigger_data or free it if not used anymore */ 1014 1019 event_trigger_free(trigger_data); 1015 - out: 1016 1020 return ret; 1017 1021 1018 1022 out_free: 1019 1023 event_trigger_reset_filter(cmd_ops, trigger_data); 1020 - kfree(trigger_data); 1021 - goto out; 1024 + trigger_data_free(trigger_data); 1025 + return ret; 1022 1026 } 1023 1027 1024 1028 /** ··· 1051 1057 s = strsep(&filter_str, " \t"); 1052 1058 1053 1059 if (!strlen(s) || strcmp(s, "if") != 0) 1054 - goto out; 1060 + return ret; 1055 1061 1056 1062 if (!filter_str) 1057 - goto out; 1063 + return ret; 1058 1064 1059 1065 /* The filter is for the 'trigger' event, not the triggered event */ 1060 1066 ret = create_event_filter(file->tr, file->event_call, ··· 1098 1104 ret = -ENOMEM; 1099 1105 } 1100 1106 } 1101 - out: 1102 1107 return ret; 1103 1108 } 1104 1109 ··· 1765 1772 ret = -EINVAL; 1766 1773 event_enable_file = find_event_file(tr, system, event); 1767 1774 if (!event_enable_file) 1768 - goto out; 1775 + return ret; 1769 1776 1770 1777 #ifdef CONFIG_HIST_TRIGGERS 1771 1778 hist = ((strcmp(cmd, ENABLE_HIST_STR) == 0) || ··· 1780 1787 1781 1788 enable_data = kzalloc(sizeof(*enable_data), GFP_KERNEL); 1782 1789 if (!enable_data) 1783 - goto out; 1790 + return ret; 1784 1791 1785 1792 enable_data->hist = hist; 1786 1793 enable_data->enable = enable; 1787 1794 enable_data->file = event_enable_file; 1788 1795 1789 - trigger_data = event_trigger_alloc(cmd_ops, cmd, param, enable_data); 1796 + trigger_data = trigger_data_alloc(cmd_ops, cmd, param, enable_data); 1790 1797 if (!trigger_data) { 1791 1798 kfree(enable_data); 1792 - goto out; 1799 + return ret; 1793 1800 } 1794 1801 1795 1802 if (remove) { ··· 1797 1804 kfree(trigger_data); 1798 1805 kfree(enable_data); 1799 1806 ret = 0; 1800 - goto out; 1807 + return ret; 1801 1808 } 1802 1809 1803 1810 /* Up the trigger_data count to make sure nothing frees it on failure */ ··· 1827 1834 goto out_disable; 1828 1835 1829 1836 event_trigger_free(trigger_data); 1830 - out: 1831 1837 return ret; 1832 1838 out_disable: 1833 1839 trace_event_enable_disable(event_enable_file, 0, 1); ··· 1837 1845 event_trigger_free(trigger_data); 1838 1846 kfree(enable_data); 1839 1847 1840 - goto out; 1848 + return ret; 1841 1849 } 1842 1850 1843 1851 int event_enable_register_trigger(char *glob, ··· 1857 1865 (test->cmd_ops->trigger_type == 1858 1866 data->cmd_ops->trigger_type) && 1859 1867 (test_enable_data->file == enable_data->file)) { 1860 - ret = -EEXIST; 1861 - goto out; 1868 + return -EEXIST; 1862 1869 } 1863 1870 } 1864 1871 1865 1872 if (data->ops->init) { 1866 1873 ret = data->ops->init(data); 1867 1874 if (ret < 0) 1868 - goto out; 1875 + return ret; 1869 1876 } 1870 1877 1871 1878 list_add_rcu(&data->list, &file->triggers); ··· 1875 1884 list_del_rcu(&data->list); 1876 1885 update_cond_flag(file); 1877 1886 } 1878 - out: 1879 1887 return ret; 1880 1888 } 1881 1889
+7 -17
kernel/trace/trace_functions.c
··· 209 209 struct ftrace_ops *op, struct ftrace_regs *fregs) 210 210 { 211 211 struct trace_array *tr = op->private; 212 - struct trace_array_cpu *data; 213 212 unsigned int trace_ctx; 214 213 int bit; 215 214 ··· 223 224 224 225 trace_ctx = tracing_gen_ctx_dec(); 225 226 226 - data = this_cpu_ptr(tr->array_buffer.data); 227 - if (!atomic_read(&data->disabled)) 228 - trace_function(tr, ip, parent_ip, trace_ctx, NULL); 227 + trace_function(tr, ip, parent_ip, trace_ctx, NULL); 229 228 230 229 ftrace_test_recursion_unlock(bit); 231 230 } ··· 233 236 struct ftrace_ops *op, struct ftrace_regs *fregs) 234 237 { 235 238 struct trace_array *tr = op->private; 236 - struct trace_array_cpu *data; 237 239 unsigned int trace_ctx; 238 240 int bit; 239 - int cpu; 240 241 241 242 if (unlikely(!tr->function_enabled)) 242 243 return; ··· 245 250 246 251 trace_ctx = tracing_gen_ctx(); 247 252 248 - cpu = smp_processor_id(); 249 - data = per_cpu_ptr(tr->array_buffer.data, cpu); 250 - if (!atomic_read(&data->disabled)) 251 - trace_function(tr, ip, parent_ip, trace_ctx, fregs); 253 + trace_function(tr, ip, parent_ip, trace_ctx, fregs); 252 254 253 255 ftrace_test_recursion_unlock(bit); 254 256 } ··· 291 299 parent_ip = function_get_true_parent_ip(parent_ip, fregs); 292 300 cpu = raw_smp_processor_id(); 293 301 data = per_cpu_ptr(tr->array_buffer.data, cpu); 294 - disabled = atomic_inc_return(&data->disabled); 302 + disabled = local_inc_return(&data->disabled); 295 303 296 304 if (likely(disabled == 1)) { 297 305 trace_ctx = tracing_gen_ctx_flags(flags); ··· 303 311 __trace_stack(tr, trace_ctx, skip); 304 312 } 305 313 306 - atomic_dec(&data->disabled); 314 + local_dec(&data->disabled); 307 315 local_irq_restore(flags); 308 316 } 309 317 ··· 344 352 { 345 353 struct trace_func_repeats *last_info; 346 354 struct trace_array *tr = op->private; 347 - struct trace_array_cpu *data; 348 355 unsigned int trace_ctx; 349 356 int bit; 350 357 ··· 355 364 return; 356 365 357 366 parent_ip = function_get_true_parent_ip(parent_ip, fregs); 358 - data = this_cpu_ptr(tr->array_buffer.data); 359 - if (atomic_read(&data->disabled)) 367 + if (!tracer_tracing_is_on(tr)) 360 368 goto out; 361 369 362 370 /* ··· 402 412 parent_ip = function_get_true_parent_ip(parent_ip, fregs); 403 413 cpu = raw_smp_processor_id(); 404 414 data = per_cpu_ptr(tr->array_buffer.data, cpu); 405 - disabled = atomic_inc_return(&data->disabled); 415 + disabled = local_inc_return(&data->disabled); 406 416 407 417 if (likely(disabled == 1)) { 408 418 last_info = per_cpu_ptr(tr->last_func_repeats, cpu); ··· 417 427 } 418 428 419 429 out: 420 - atomic_dec(&data->disabled); 430 + local_dec(&data->disabled); 421 431 local_irq_restore(flags); 422 432 } 423 433
+9 -29
kernel/trace/trace_functions_graph.c
··· 202 202 { 203 203 unsigned long *task_var = fgraph_get_task_var(gops); 204 204 struct trace_array *tr = gops->private; 205 - struct trace_array_cpu *data; 206 205 struct fgraph_times *ftimes; 207 206 unsigned int trace_ctx; 208 - long disabled; 209 207 int ret = 0; 210 - int cpu; 211 208 212 209 if (*task_var & TRACE_GRAPH_NOTRACE) 213 210 return 0; ··· 254 257 if (tracing_thresh) 255 258 return 1; 256 259 257 - preempt_disable_notrace(); 258 - cpu = raw_smp_processor_id(); 259 - data = per_cpu_ptr(tr->array_buffer.data, cpu); 260 - disabled = atomic_read(&data->disabled); 261 - if (likely(!disabled)) { 262 - trace_ctx = tracing_gen_ctx(); 263 - if (IS_ENABLED(CONFIG_FUNCTION_GRAPH_RETADDR) && 264 - tracer_flags_is_set(TRACE_GRAPH_PRINT_RETADDR)) { 265 - unsigned long retaddr = ftrace_graph_top_ret_addr(current); 266 - ret = __trace_graph_retaddr_entry(tr, trace, trace_ctx, retaddr); 267 - } else { 268 - ret = __graph_entry(tr, trace, trace_ctx, fregs); 269 - } 260 + trace_ctx = tracing_gen_ctx(); 261 + if (IS_ENABLED(CONFIG_FUNCTION_GRAPH_RETADDR) && 262 + tracer_flags_is_set(TRACE_GRAPH_PRINT_RETADDR)) { 263 + unsigned long retaddr = ftrace_graph_top_ret_addr(current); 264 + ret = __trace_graph_retaddr_entry(tr, trace, trace_ctx, retaddr); 265 + } else { 266 + ret = __graph_entry(tr, trace, trace_ctx, fregs); 270 267 } 271 - preempt_enable_notrace(); 272 268 273 269 return ret; 274 270 } ··· 341 351 { 342 352 unsigned long *task_var = fgraph_get_task_var(gops); 343 353 struct trace_array *tr = gops->private; 344 - struct trace_array_cpu *data; 345 354 struct fgraph_times *ftimes; 346 355 unsigned int trace_ctx; 347 356 u64 calltime, rettime; 348 - long disabled; 349 357 int size; 350 - int cpu; 351 358 352 359 rettime = trace_clock_local(); 353 360 ··· 363 376 364 377 calltime = ftimes->calltime; 365 378 366 - preempt_disable_notrace(); 367 - cpu = raw_smp_processor_id(); 368 - data = per_cpu_ptr(tr->array_buffer.data, cpu); 369 - disabled = atomic_read(&data->disabled); 370 - if (likely(!disabled)) { 371 - trace_ctx = tracing_gen_ctx(); 372 - __trace_graph_return(tr, trace, trace_ctx, calltime, rettime); 373 - } 374 - preempt_enable_notrace(); 379 + trace_ctx = tracing_gen_ctx(); 380 + __trace_graph_return(tr, trace, trace_ctx, calltime, rettime); 375 381 } 376 382 377 383 static void trace_graph_thresh_return(struct ftrace_graph_ret *trace,
+27 -20
kernel/trace/trace_irqsoff.c
··· 123 123 return 0; 124 124 125 125 *data = per_cpu_ptr(tr->array_buffer.data, cpu); 126 - disabled = atomic_inc_return(&(*data)->disabled); 126 + disabled = local_inc_return(&(*data)->disabled); 127 127 128 128 if (likely(disabled == 1)) 129 129 return 1; 130 130 131 - atomic_dec(&(*data)->disabled); 131 + local_dec(&(*data)->disabled); 132 132 133 133 return 0; 134 134 } ··· 152 152 153 153 trace_function(tr, ip, parent_ip, trace_ctx, fregs); 154 154 155 - atomic_dec(&data->disabled); 155 + local_dec(&data->disabled); 156 156 } 157 157 #endif /* CONFIG_FUNCTION_TRACER */ 158 158 ··· 209 209 210 210 trace_ctx = tracing_gen_ctx_flags(flags); 211 211 ret = __trace_graph_entry(tr, trace, trace_ctx); 212 - atomic_dec(&data->disabled); 212 + local_dec(&data->disabled); 213 213 214 214 return ret; 215 215 } ··· 238 238 239 239 trace_ctx = tracing_gen_ctx_flags(flags); 240 240 __trace_graph_return(tr, trace, trace_ctx, *calltime, rettime); 241 - atomic_dec(&data->disabled); 241 + local_dec(&data->disabled); 242 242 } 243 243 244 244 static struct fgraph_ops fgraph_ops = { ··· 397 397 int cpu; 398 398 struct trace_array *tr = irqsoff_trace; 399 399 struct trace_array_cpu *data; 400 + long disabled; 400 401 401 402 if (!tracer_enabled || !tracing_is_enabled()) 402 403 return; ··· 409 408 410 409 data = per_cpu_ptr(tr->array_buffer.data, cpu); 411 410 412 - if (unlikely(!data) || atomic_read(&data->disabled)) 411 + if (unlikely(!data) || local_read(&data->disabled)) 413 412 return; 414 413 415 - atomic_inc(&data->disabled); 414 + disabled = local_inc_return(&data->disabled); 416 415 417 - data->critical_sequence = max_sequence; 418 - data->preempt_timestamp = ftrace_now(cpu); 419 - data->critical_start = parent_ip ? : ip; 416 + if (disabled == 1) { 417 + data->critical_sequence = max_sequence; 418 + data->preempt_timestamp = ftrace_now(cpu); 419 + data->critical_start = parent_ip ? : ip; 420 420 421 - __trace_function(tr, ip, parent_ip, tracing_gen_ctx()); 421 + __trace_function(tr, ip, parent_ip, tracing_gen_ctx()); 422 422 423 - per_cpu(tracing_cpu, cpu) = 1; 423 + per_cpu(tracing_cpu, cpu) = 1; 424 + } 424 425 425 - atomic_dec(&data->disabled); 426 + local_dec(&data->disabled); 426 427 } 427 428 428 429 static nokprobe_inline void ··· 434 431 struct trace_array *tr = irqsoff_trace; 435 432 struct trace_array_cpu *data; 436 433 unsigned int trace_ctx; 434 + long disabled; 437 435 438 436 cpu = raw_smp_processor_id(); 439 437 /* Always clear the tracing cpu on stopping the trace */ ··· 449 445 data = per_cpu_ptr(tr->array_buffer.data, cpu); 450 446 451 447 if (unlikely(!data) || 452 - !data->critical_start || atomic_read(&data->disabled)) 448 + !data->critical_start || local_read(&data->disabled)) 453 449 return; 454 450 455 - atomic_inc(&data->disabled); 451 + disabled = local_inc_return(&data->disabled); 456 452 457 - trace_ctx = tracing_gen_ctx(); 458 - __trace_function(tr, ip, parent_ip, trace_ctx); 459 - check_critical_timing(tr, data, parent_ip ? : ip, cpu); 460 - data->critical_start = 0; 461 - atomic_dec(&data->disabled); 453 + if (disabled == 1) { 454 + trace_ctx = tracing_gen_ctx(); 455 + __trace_function(tr, ip, parent_ip, trace_ctx); 456 + check_critical_timing(tr, data, parent_ip ? : ip, cpu); 457 + data->critical_start = 0; 458 + } 459 + 460 + local_dec(&data->disabled); 462 461 } 463 462 464 463 /* start and stop critical timings used to for stoppage (in idle) */
+2 -7
kernel/trace/trace_kdb.c
··· 98 98 long cpu_file; 99 99 int err; 100 100 int cnt; 101 - int cpu; 102 101 103 102 if (argc > 2) 104 103 return KDB_ARGCOUNT; ··· 119 120 trace_init_global_iter(&iter); 120 121 iter.buffer_iter = buffer_iter; 121 122 122 - for_each_tracing_cpu(cpu) { 123 - atomic_inc(&per_cpu_ptr(iter.array_buffer->data, cpu)->disabled); 124 - } 123 + tracer_tracing_disable(iter.tr); 125 124 126 125 /* A negative skip_entries means skip all but the last entries */ 127 126 if (skip_entries < 0) { ··· 132 135 133 136 ftrace_dump_buf(skip_entries, cpu_file); 134 137 135 - for_each_tracing_cpu(cpu) { 136 - atomic_dec(&per_cpu_ptr(iter.array_buffer->data, cpu)->disabled); 137 - } 138 + tracer_tracing_enable(iter.tr); 138 139 139 140 kdb_trap_printk--; 140 141
+2 -10
kernel/trace/trace_mmiotrace.c
··· 291 291 device_initcall(init_mmio_trace); 292 292 293 293 static void __trace_mmiotrace_rw(struct trace_array *tr, 294 - struct trace_array_cpu *data, 295 294 struct mmiotrace_rw *rw) 296 295 { 297 296 struct trace_buffer *buffer = tr->array_buffer.buffer; ··· 314 315 void mmio_trace_rw(struct mmiotrace_rw *rw) 315 316 { 316 317 struct trace_array *tr = mmio_trace_array; 317 - struct trace_array_cpu *data = per_cpu_ptr(tr->array_buffer.data, smp_processor_id()); 318 - __trace_mmiotrace_rw(tr, data, rw); 318 + __trace_mmiotrace_rw(tr, rw); 319 319 } 320 320 321 321 static void __trace_mmiotrace_map(struct trace_array *tr, 322 - struct trace_array_cpu *data, 323 322 struct mmiotrace_map *map) 324 323 { 325 324 struct trace_buffer *buffer = tr->array_buffer.buffer; ··· 341 344 void mmio_trace_mapping(struct mmiotrace_map *map) 342 345 { 343 346 struct trace_array *tr = mmio_trace_array; 344 - struct trace_array_cpu *data; 345 - 346 - preempt_disable(); 347 - data = per_cpu_ptr(tr->array_buffer.data, smp_processor_id()); 348 - __trace_mmiotrace_map(tr, data, map); 349 - preempt_enable(); 347 + __trace_mmiotrace_map(tr, map); 350 348 } 351 349 352 350 int mmio_trace_printk(const char *fmt, va_list args)
+5 -4
kernel/trace/trace_osnoise.c
··· 2302 2302 * osnoise_cpus_write - Write function for "cpus" entry 2303 2303 * @filp: The active open file structure 2304 2304 * @ubuf: The user buffer that contains the value to write 2305 - * @cnt: The maximum number of bytes to write to "file" 2305 + * @count: The maximum number of bytes to write to "file" 2306 2306 * @ppos: The current position in @file 2307 2307 * 2308 2308 * This function provides a write implementation for the "cpus" ··· 2320 2320 { 2321 2321 cpumask_var_t osnoise_cpumask_new; 2322 2322 int running, err; 2323 - char buf[256]; 2323 + char *buf __free(kfree) = NULL; 2324 2324 2325 - if (count >= 256) 2326 - return -EINVAL; 2325 + buf = kmalloc(count, GFP_KERNEL); 2326 + if (!buf) 2327 + return -ENOMEM; 2327 2328 2328 2329 if (copy_from_user(buf, ubuf, count)) 2329 2330 return -EFAULT;
+42 -18
kernel/trace/trace_output.c
··· 938 938 struct list_head *head) 939 939 { 940 940 struct ftrace_event_field *field; 941 + struct trace_array *tr = iter->tr; 942 + unsigned long long laddr; 943 + unsigned long addr; 941 944 int offset; 942 945 int len; 943 946 int ret; ··· 977 974 case FILTER_PTR_STRING: 978 975 if (!iter->fmt_size) 979 976 trace_iter_expand_format(iter); 980 - pos = *(void **)pos; 981 - ret = strncpy_from_kernel_nofault(iter->fmt, pos, 977 + addr = trace_adjust_address(tr, *(unsigned long *)pos); 978 + ret = strncpy_from_kernel_nofault(iter->fmt, (void *)addr, 982 979 iter->fmt_size); 983 980 if (ret < 0) 984 981 trace_seq_printf(&iter->seq, "(0x%px)", pos); ··· 987 984 pos, iter->fmt); 988 985 break; 989 986 case FILTER_TRACE_FN: 990 - pos = *(void **)pos; 991 - trace_seq_printf(&iter->seq, "%pS", pos); 987 + addr = trace_adjust_address(tr, *(unsigned long *)pos); 988 + trace_seq_printf(&iter->seq, "%pS", (void *)addr); 992 989 break; 993 990 case FILTER_CPU: 994 991 case FILTER_OTHER: ··· 1018 1015 break; 1019 1016 } 1020 1017 1021 - trace_seq_printf(&iter->seq, "0x%x (%d)", 1022 - *(unsigned int *)pos, 1023 - *(unsigned int *)pos); 1018 + addr = *(unsigned int *)pos; 1019 + 1020 + /* Some fields reference offset from _stext. */ 1021 + if (!strcmp(field->name, "caller_offs") || 1022 + !strcmp(field->name, "parent_offs")) { 1023 + unsigned long ip; 1024 + 1025 + ip = addr + (unsigned long)_stext; 1026 + ip = trace_adjust_address(tr, ip); 1027 + trace_seq_printf(&iter->seq, "%pS ", (void *)ip); 1028 + } 1029 + 1030 + if (sizeof(long) == 4) { 1031 + addr = trace_adjust_address(tr, addr); 1032 + trace_seq_printf(&iter->seq, "%pS (%d)", 1033 + (void *)addr, (int)addr); 1034 + } else { 1035 + trace_seq_printf(&iter->seq, "0x%x (%d)", 1036 + (unsigned int)addr, (int)addr); 1037 + } 1024 1038 break; 1025 1039 case 8: 1026 - trace_seq_printf(&iter->seq, "0x%llx (%lld)", 1027 - *(unsigned long long *)pos, 1028 - *(unsigned long long *)pos); 1040 + laddr = *(unsigned long long *)pos; 1041 + if (sizeof(long) == 8) { 1042 + laddr = trace_adjust_address(tr, (unsigned long)laddr); 1043 + trace_seq_printf(&iter->seq, "%pS (%lld)", 1044 + (void *)(long)laddr, laddr); 1045 + } else { 1046 + trace_seq_printf(&iter->seq, "0x%llx (%lld)", laddr, laddr); 1047 + } 1029 1048 break; 1030 1049 default: 1031 1050 trace_seq_puts(&iter->seq, "<INVALID-SIZE>"); ··· 1111 1086 } 1112 1087 1113 1088 static void print_fn_trace(struct trace_seq *s, unsigned long ip, 1114 - unsigned long parent_ip, long delta, 1115 - unsigned long *args, int flags) 1089 + unsigned long parent_ip, unsigned long *args, 1090 + struct trace_array *tr, int flags) 1116 1091 { 1117 - ip += delta; 1118 - parent_ip += delta; 1092 + ip = trace_adjust_address(tr, ip); 1093 + parent_ip = trace_adjust_address(tr, parent_ip); 1119 1094 1120 1095 seq_print_ip_sym(s, ip, flags); 1121 1096 if (args) ··· 1144 1119 else 1145 1120 args = NULL; 1146 1121 1147 - print_fn_trace(s, field->ip, field->parent_ip, iter->tr->text_delta, 1148 - args, flags); 1122 + print_fn_trace(s, field->ip, field->parent_ip, args, iter->tr, flags); 1149 1123 trace_seq_putc(s, '\n'); 1150 1124 1151 1125 return trace_handle_return(s); ··· 1730 1706 1731 1707 trace_assign_type(field, iter->ent); 1732 1708 1733 - ip = field->ip + iter->tr->text_delta; 1709 + ip = trace_adjust_address(iter->tr, field->ip); 1734 1710 1735 1711 seq_print_ip_sym(s, ip, flags); 1736 1712 trace_seq_printf(s, ": %s", field->buf); ··· 1816 1792 1817 1793 trace_assign_type(field, iter->ent); 1818 1794 1819 - print_fn_trace(s, field->ip, field->parent_ip, iter->tr->text_delta, NULL, flags); 1795 + print_fn_trace(s, field->ip, field->parent_ip, NULL, iter->tr, flags); 1820 1796 trace_seq_printf(s, " (repeats: %u, last_ts:", field->count); 1821 1797 trace_print_time(s, iter, 1822 1798 iter->ts - FUNC_REPEATS_GET_DELTA_TS(field));
+9 -9
kernel/trace/trace_sched_wakeup.c
··· 83 83 goto out_enable; 84 84 85 85 *data = per_cpu_ptr(tr->array_buffer.data, cpu); 86 - disabled = atomic_inc_return(&(*data)->disabled); 86 + disabled = local_inc_return(&(*data)->disabled); 87 87 if (unlikely(disabled != 1)) 88 88 goto out; 89 89 90 90 return 1; 91 91 92 92 out: 93 - atomic_dec(&(*data)->disabled); 93 + local_dec(&(*data)->disabled); 94 94 95 95 out_enable: 96 96 preempt_enable_notrace(); ··· 144 144 *calltime = trace_clock_local(); 145 145 146 146 ret = __trace_graph_entry(tr, trace, trace_ctx); 147 - atomic_dec(&data->disabled); 147 + local_dec(&data->disabled); 148 148 preempt_enable_notrace(); 149 149 150 150 return ret; ··· 173 173 return; 174 174 175 175 __trace_graph_return(tr, trace, trace_ctx, *calltime, rettime); 176 - atomic_dec(&data->disabled); 176 + local_dec(&data->disabled); 177 177 178 178 preempt_enable_notrace(); 179 179 return; ··· 243 243 trace_function(tr, ip, parent_ip, trace_ctx, fregs); 244 244 local_irq_restore(flags); 245 245 246 - atomic_dec(&data->disabled); 246 + local_dec(&data->disabled); 247 247 preempt_enable_notrace(); 248 248 } 249 249 ··· 471 471 472 472 /* disable local data, not wakeup_cpu data */ 473 473 cpu = raw_smp_processor_id(); 474 - disabled = atomic_inc_return(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 474 + disabled = local_inc_return(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 475 475 if (likely(disabled != 1)) 476 476 goto out; 477 477 ··· 508 508 arch_spin_unlock(&wakeup_lock); 509 509 local_irq_restore(flags); 510 510 out: 511 - atomic_dec(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 511 + local_dec(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 512 512 } 513 513 514 514 static void __wakeup_reset(struct trace_array *tr) ··· 563 563 (!dl_task(p) && (p->prio >= wakeup_prio || p->prio >= current->prio))) 564 564 return; 565 565 566 - disabled = atomic_inc_return(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 566 + disabled = local_inc_return(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 567 567 if (unlikely(disabled != 1)) 568 568 goto out; 569 569 ··· 610 610 out_locked: 611 611 arch_spin_unlock(&wakeup_lock); 612 612 out: 613 - atomic_dec(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 613 + local_dec(&per_cpu_ptr(wakeup_trace->array_buffer.data, cpu)->disabled); 614 614 } 615 615 616 616 static void start_wakeup_tracer(struct trace_array *tr)
+1 -1
kernel/trace/trace_stack.c
··· 542 542 int len; 543 543 544 544 if ((len = str_has_prefix(str, "_filter="))) 545 - strncpy(stack_trace_filter_buf, str + len, COMMAND_LINE_SIZE); 545 + strscpy(stack_trace_filter_buf, str + len); 546 546 547 547 stack_tracer_enabled = 1; 548 548 return 1;
+1 -1
tools/testing/selftests/bpf/progs/raw_tp_null.c
··· 10 10 int tid; 11 11 int i; 12 12 13 - SEC("tp_btf/bpf_testmod_test_raw_tp_null") 13 + SEC("tp_btf/bpf_testmod_test_raw_tp_null_tp") 14 14 int BPF_PROG(test_raw_tp_null, struct sk_buff *skb) 15 15 { 16 16 struct task_struct *task = bpf_get_current_task_btf();
+1 -1
tools/testing/selftests/bpf/progs/raw_tp_null_fail.c
··· 8 8 char _license[] SEC("license") = "GPL"; 9 9 10 10 /* Ensure module parameter has PTR_MAYBE_NULL */ 11 - SEC("tp_btf/bpf_testmod_test_raw_tp_null") 11 + SEC("tp_btf/bpf_testmod_test_raw_tp_null_tp") 12 12 __failure __msg("R1 invalid mem access 'trusted_ptr_or_null_'") 13 13 int test_raw_tp_null_bpf_testmod_test_raw_tp_null_arg_1(void *ctx) { 14 14 asm volatile("r1 = *(u64 *)(r1 +0); r1 = *(u64 *)(r1 +0);" ::: __clobber_all);
+2 -2
tools/testing/selftests/bpf/progs/test_module_attach.c
··· 19 19 20 20 __u32 raw_tp_bare_write_sz = 0; 21 21 22 - SEC("raw_tp/bpf_testmod_test_write_bare") 22 + SEC("raw_tp/bpf_testmod_test_write_bare_tp") 23 23 int BPF_PROG(handle_raw_tp_bare, 24 24 struct task_struct *task, struct bpf_testmod_test_write_ctx *write_ctx) 25 25 { ··· 31 31 int raw_tp_writable_bare_early_ret = 0; 32 32 int raw_tp_writable_bare_out_val = 0; 33 33 34 - SEC("raw_tp.w/bpf_testmod_test_writable_bare") 34 + SEC("raw_tp.w/bpf_testmod_test_writable_bare_tp") 35 35 int BPF_PROG(handle_raw_tp_writable_bare, 36 36 struct bpf_testmod_test_writable_ctx *writable) 37 37 {
+2 -2
tools/testing/selftests/bpf/progs/test_tp_btf_nullable.c
··· 6 6 #include "../test_kmods/bpf_testmod.h" 7 7 #include "bpf_misc.h" 8 8 9 - SEC("tp_btf/bpf_testmod_test_nullable_bare") 9 + SEC("tp_btf/bpf_testmod_test_nullable_bare_tp") 10 10 __failure __msg("R1 invalid mem access 'trusted_ptr_or_null_'") 11 11 int BPF_PROG(handle_tp_btf_nullable_bare1, struct bpf_testmod_test_read_ctx *nullable_ctx) 12 12 { 13 13 return nullable_ctx->len; 14 14 } 15 15 16 - SEC("tp_btf/bpf_testmod_test_nullable_bare") 16 + SEC("tp_btf/bpf_testmod_test_nullable_bare_tp") 17 17 int BPF_PROG(handle_tp_btf_nullable_bare2, struct bpf_testmod_test_read_ctx *nullable_ctx) 18 18 { 19 19 if (nullable_ctx)
+4 -4
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
··· 417 417 418 418 (void)bpf_testmod_test_arg_ptr_to_struct(&struct_arg1_2); 419 419 420 - (void)trace_bpf_testmod_test_raw_tp_null(NULL); 420 + (void)trace_bpf_testmod_test_raw_tp_null_tp(NULL); 421 421 422 422 bpf_testmod_test_struct_ops3(); 423 423 ··· 435 435 if (bpf_testmod_loop_test(101) > 100) 436 436 trace_bpf_testmod_test_read(current, &ctx); 437 437 438 - trace_bpf_testmod_test_nullable_bare(NULL); 438 + trace_bpf_testmod_test_nullable_bare_tp(NULL); 439 439 440 440 /* Magic number to enable writable tp */ 441 441 if (len == 64) { 442 442 struct bpf_testmod_test_writable_ctx writable = { 443 443 .val = 1024, 444 444 }; 445 - trace_bpf_testmod_test_writable_bare(&writable); 445 + trace_bpf_testmod_test_writable_bare_tp(&writable); 446 446 if (writable.early_ret) 447 447 return snprintf(buf, len, "%d\n", writable.val); 448 448 } ··· 474 474 .len = len, 475 475 }; 476 476 477 - trace_bpf_testmod_test_write_bare(current, &ctx); 477 + trace_bpf_testmod_test_write_bare_tp(current, &ctx); 478 478 479 479 return -EIO; /* always fail */ 480 480 }