summaryrefslogtreecommitdiff
path: root/tools/tracing
AgeCommit message (Collapse)Author
2025-03-27Merge tag 'trace-tools-v6.15' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing tooling updates from Steven Rostedt: - Allow RTLA to collect data via BPF The current implementation of rtla uses libtracefs and libtraceevent to pull sample events generated by the timerlat tracer from the trace buffer. rtla then processes the sample by updating the histogram and summary (current, maximum, minimum, and sum values) as well as checks if tracing has been stopped due to threshold overflow. In use cases where a large number of samples is being generated, that is, with measurements running on many CPUs and with a low interval, this sample processing design causes a significant CPU load on the rtla side. Furthermore, with >100 CPUs and 100us interval, rtla was reported as not being able to keep up with the samples and dropping most of them, leading to it being unusable. Change the way the timerlat trace processes samples by attaching a BPF program to the trace event using the BPF skeleton feature of bpftool. Unlike the current implementation, the BPF implementation does not check whether tracing is stopped (in BPF mode, tracing is always off to improve performance), but waits for a write to a BPF ringbuffer instead. This allows rtla to exit immediately when a threshold is violated, without waiting for the next iteration of the while loop. If the requirements for the BPF implementation are not met, either at build time or at run time, the current implementation is used as fallback. Which implementation is being used can be seen when running rtla timerlat with "-D" option. rtla can be forced to run in non-BPF mode by setting the RTLA_NO_BPF option to 1, for debugging purposes. - Fix LD_FLAGS from being dropped in build - Refactor code to remove duplication of save_trace_to_file - Always set options and do not rely on default settings Do not rely on the default kernel settings of the tracers when starting. They could have been changed by the user which gives inconsistent results. Always set the options that rtla expects. - Add creation of ctags and TAGS for traversing code * tag 'trace-tools-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: rtla: Add the ability to create ctags and etags rtla/tests: Test setting default options rtla/tests: Reset osnoise options before check rtla: Always set all tracer options rtla/osnoise: Set OSNOISE_WORKLOAD to true rtla: Unify apply_config between top and hist rtla/osnoise: Unify params struct rtla: Fix segfault in save_trace_to_file call tools/build: Use SYSTEM_BPFTOOL for system bpftool rtla: Refactor save_trace_to_file tools/rv: Keep user LDFLAGS in build rtla/timerlat: Test BPF mode rtla/timerlat_top: Use BPF to collect samples rtla/timerlat_top: Move divisor to update rtla/timerlat_hist: Use BPF to collect samples rtla/timerlat: Add BPF skeleton to collect samples rtla: Add optional dependency on BPF tooling tools/build: Add bpftool-skeletons feature test rtla/timerlat: Unify params struct
2025-03-26rtla: Add the ability to create ctags and etagsJohn Kacur
- Add the ability to create and remove ctags and etags, using the following make tags make TAGS make tags_clean - fix a comment in Makefile.rtla with the correct spelling and don't imply that the ability to create an rtla tarball will be removed Cc: Tomas Glozar <tglozar@redhat.com> Cc: "Luis Claudio R . Goncalves" <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250321175053.29048-1-jkacur@redhat.com Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla/tests: Test setting default optionsTomas Glozar
Add function to test engine to test with pre-set osnoise options, and use it to test whether osnoise period (as an example) is set correctly. The test works by pre-setting a high period of 10 minutes and stop on threshold. Thus, it is easy to check whether rtla is properly resetting the period to default: if it is, the test will complete on time, since the first sample will overflow the threshold. If not, it will time out. Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-7-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla/tests: Reset osnoise options before checkTomas Glozar
Remove any dangling tracing instances from previous improperly exited runs of rtla, and reset osnoise options to default before running a test case. This ensures that the test results are deterministic. Specific test cases checked that rtla behaves correctly even when the tracer state is not clean will be added later. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla: Always set all tracer optionsTomas Glozar
rtla currently only sets tracer options that are explicitly set by the user, with the exception of OSNOISE_WORKLOAD. This leads to improper behavior in case rtla is run with those options not set to the default value. rtla does reset them to the original value upon exiting, but that does not protect it from starting with non-default values set either by an improperly exited rtla or by another user of the tracers. For example, after running this command: $ echo 1 > /sys/kernel/tracing/osnoise/stop_tracing_us all runs of rtla will stop at the 1us threshold, even if not requested by the user: $ rtla osnoise hist Index CPU-000 CPU-001 1 8 5 2 5 9 3 1 2 4 6 1 5 2 1 6 0 1 8 1 1 12 0 1 14 1 0 15 1 0 over: 0 0 count: 25 21 min: 1 1 avg: 3.68 3.05 max: 15 12 rtla osnoise hit stop tracing Fix the problem by setting the default value for all tracer options if the user has not provided their own value. For most of the options, it's enough to just drop the if clause checking for the value being set. For cpus, "all" is used as the default value, and for osnoise default period and runtime, default values of the osnoise_data variable in trace_osnoise.c are used. Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-5-tglozar@redhat.com Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla/osnoise: Set OSNOISE_WORKLOAD to trueTomas Glozar
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples: $ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options $ rtla osnoise hist -d 10s Index over: 0 count: 0 min: 0 avg: 0 max: 0 This situation can also happen when running rtla-osnoise after an improperly exited rtla-timerlat run. Set OSNOISE_WORKLOAD in rtla-osnoise, too, similarly to what we already did for timerlat in commit 217f0b1e990e ("rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads") and commit d8d866171a41 ("rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads"). Note that there is no user workload mode for rtla-osnoise yet, so OSNOISE_WORKLOAD is always set to true. Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-4-tglozar@redhat.com Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla: Unify apply_config between top and histTomas Glozar
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly the same. Move common part from them into separate functions osnoise_apply_config and timerlat_apply_config. For rtla-timerlat, also unify params->user_hist and params->user_top into one field called params->user_data, and move several fields used only by timerlat-top into the top-only section of struct timerlat_params. Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla/osnoise: Unify params structTomas Glozar
Instead of having separate structs osnoise_top_params and osnoise_hist_params, use one struct osnoise_params for both. This allows code using the structs to be shared between osnoise-top and osnoise-hist. Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26rtla: Fix segfault in save_trace_to_file callTomas Glozar
Running rtla with exit on threshold, but without saving trace leads to a segmenetation fault: $ rtla timerlat hist -T 10 ... Max timerlat IRQ latency from idle: 4.29 us in cpu 0 Segmentation fault This is caused by null pointer deference in the call of save_trace_to_file, which attempts to dereference an uninitialized osnoise_tool variable: save_trace_to_file(record->trace.inst, params->trace_output); ^ this is uninitialized if params->trace_output is not set Fix this by not attempting to dereference "record" if it is NULL and passing NULL instead. As a safety measure, the first field is also checked for NULL inside save_trace_to_file. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Link: https://lore.kernel.org/20250313141034.299117-1-tglozar@redhat.com Fixes: dc4d4e7c72d1 ("rtla: Refactor save_trace_to_file") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-26tools/build: Use SYSTEM_BPFTOOL for system bpftoolTomas Glozar
The feature test for system bpftool uses BPFTOOL as the variable to set its path, defaulting to just "bpftool" if not set by the user. This conflicts with selftests and a few other utilities, which expect BPFTOOL to be set to the in-tree bpftool path by default. For example, bpftool selftests fail to build: $ make -C tools/testing/selftests/bpf/ make: Entering directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf' make: *** No rule to make target 'bpftool', needed by '/home/tglozar/dev/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop. make: Leaving directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf' Fix the problem by renaming the variable used for system bpftool from BPFTOOL to SYSTEM_BPFTOOL, so that the new usage does not conflict with the existing one of BPFTOOL. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250326004018.248357-1-tglozar@redhat.com Fixes: 8a635c3856dd ("tools/build: Add bpftool-skeletons feature test") Closes: https://lore.kernel.org/linux-kernel/5df6968a-2e5f-468e-b457-fc201535dd4c@linux.ibm.com/ Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com> Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com> Suggested-by: Quentin Monnet <qmo@kernel.org> Acked-by: Quentin Monnet <qmo@kernel.org> Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla: Refactor save_trace_to_fileCosta Shulyupin
The functions osnoise_hist_main(), osnoise_top_main(), timerlat_hist_main(), and timerlat_top_main() are lengthy and contain duplicated code. Refactor by consolidating the duplicate lines into the save_trace_to_file() function. Cc: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: John Kacur <jkacur@redhat.com> Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com> Cc: Eder Zulian <ezulian@redhat.com> Cc: Dan Carpenter <dan.carpenter@linaro.org> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250219115138.406075-1-costa.shul@redhat.com Signed-off-by: Costa Shulyupin <costa.shul@redhat.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Tested-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat: Test BPF modeTomas Glozar
Using the RTLA_NO_BPF environmental variable, execute rtla-timerlat tests both with and without BPF support to cover both paths. If rtla is built without BPF or the osnoise:timerlat_sample trace event is not available, test only the non-BPF path. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-9-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat_top: Use BPF to collect samplesTomas Glozar
Collect samples using BPF program instead of pulling them from tracefs. If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fails to load for whatever reason, rtla falls back to the old implementation. The collection of samples using the BPF program is fully self-contained and requires no activity of the userspace part of rtla during the measurement. Thus, rtla only pulls the summary from the BPF map and displays it every second, improving the performance. In --aa-only mode, the BPF program does not collect any data and only signalizes the end of tracing to userspace. An optimization that re-used the main trace instance for auto-analysis in aa-only mode was dropped, as rtla no longer turns tracing on in the main trace instance, making it useless for auto-analysis. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-8-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat_top: Move divisor to updateTomas Glozar
Unlike timerlat-hist, timerlat-top applies the output divisor used to set ns/us mode when printing results instead of applying it when collecting the samples. Move the application of the divisor from timerlat_top_print into timerlat_top_update to make it consistent with timerlat-hist. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-7-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat_hist: Use BPF to collect samplesTomas Glozar
Collect samples using BPF program instead of pulling them from tracefs. If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fails to load for whatever reason, rtla falls back to the old implementation. The collection of samples using the BPF program is fully self-contained and requires no activity of the userspace part of rtla during the measurement. Thus, instead of waking up every second to collect samples, rtla simply sleeps until woken up by a signal or threshold overflow. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat: Add BPF skeleton to collect samplesTomas Glozar
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint and collects both the summary and the histogram (if requested) into BPF maps (one map of each kind per context). The program is designed to be used for both timerlat-top and timerlat-hist. If using with timerlat-top, the "entries" parameter is set to zero, which prevents the BPF program from recording histogram entries. In that case, the maps for histograms do not have to be created, as the BPF verifier will identify the code using them as unreachable. An IRQ or thread latency threshold might be supplied to stop recording if hit, similar to the timerlat tracer threshold, which stops ftrace tracing if hit. A BPF ringbuffer is used to signal threshold overflow to userspace. In aa-only mode, this is the only function of the BPF program. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla: Add optional dependency on BPF toolingTomas Glozar
If tooling required for building BPF CO-RE skeletons is present (that is, libbpf, clang with BPF CO-RE support, and bpftool), turn on HAVE_BPF_SKEL flag. Those requirements are similar to what perf requires, with the difference of using system libbpf and bpftool instead of in-tree versions. rtla can be forcefully built without BPF skeleton support by setting BUILD_BPF_SKEL=0 manually; in that case, a warning is displayed. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-4-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-03-04rtla/timerlat: Unify params structTomas Glozar
Instead of having separate structs timerlat_top_params and timerlat_hist_params, use one struct timerlat_params for both. This allows code using the structs to be shared between timerlat-top and timerlat-hist. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Cc: Clark Williams <williams@redhat.com> Link: https://lore.kernel.org/20250218145859.27762-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-02-25Merge tag 'perf-tools-fixes-for-v6.14-2-2025-02-25' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools Pull perf tools fixes from Arnaldo Carvalho de Melo: - Fix tools/ quiet build Makefile infrastructure that was broken when working on tools/perf/ without testing on other tools/ living utilities. * tag 'perf-tools-fixes-for-v6.14-2-2025-02-25' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools: tools: Remove redundant quiet setup tools: Unify top-level quiet infrastructure
2025-02-18tools: Remove redundant quiet setupCharlie Jenkins
Q is exported from Makefile.include so it is not necessary to manually set it. Reviewed-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Acked-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Quentin Monnet <qmo@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Benjamin Tissoires <bentiss@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Daniel Lezcano <daniel.lezcano@linaro.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Hao Luo <haoluo@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jiri Kosina <jikos@kernel.org> Cc: John Fastabend <john.fastabend@gmail.com> Cc: Josh Poimboeuf <jpoimboe@kernel.org> Cc: KP Singh <kpsingh@kernel.org> Cc: Lukasz Luba <lukasz.luba@arm.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Martin KaFai Lau <martin.lau@linux.dev> Cc: Mykola Lysenko <mykolal@fb.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rafael J. Wysocki <rafael@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Song Liu <song@kernel.org> Cc: Stanislav Fomichev <sdf@google.com> Cc: Steven Rostedt (VMware) <rostedt@goodmis.org> Cc: Yonghong Song <yonghong.song@linux.dev> Cc: Zhang Rui <rui.zhang@intel.com> Link: https://lore.kernel.org/r/20250213-quiet_tools-v3-2-07de4482a581@rivosinc.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2025-01-26Merge tag 'trace-tools-v6.14' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull rv and tools/rtla updates from Steven Rostedt: - Add a test suite to test the tool Add a small test suite that can be used to test rtla's basic features to at least have something to test when applying changes. - Automate manual steps in monitor creation While creating a new monitor in RV, besides generating code from dot2k, there are a few manual steps which can be tedious and error prone, like adding the tracepoints, makefile lines and kconfig, or selecting events that start the monitor in the initial state. Updates were made to try and automate as much as possible among those steps to make creating a new RV monitor much quicker. It is still requires to select proper tracepoints, this step is harder to automate in a general way and, in several cases, would still need user intervention. - Have rtla timerlat hist and top set OSNOISE_WORKLOAD flag Have both rtla-timerlat-hist and rtla-timerlat-top set OSNOISE_WORKLOAD to the proper value ("on" when running with -k, "off" when running with -u) every time the option is available instead of setting it only when running with -u. This prevents rtla timerlat -k from giving no results when NO_OSNOISE_WORKLOAD is set, either manually or by an abnormally exited earlier run of rtla timerlat -u. - Stop rtla timerlat on signal properly when overloaded There is an issue where if rtla is run on machines with a high number of CPUs (100+), timerlat can generate more samples than rtla is able to process via tracefs_iterate_raw_events. This is especially common when the interval is set to 100us (rteval and cyclictest default) as opposed to the rtla default of 1000us, but also happens with the rtla default. Currently, this leads to rtla hanging and having to be terminated with SIGTERM. SIGINT setting stop_tracing is not enough, since more and more events are coming and tracefs_iterate_raw_events never exits. To fix this: Stop the timerlat tracer on SIGINT/SIGALRM to ensure no more events are generated when rtla is supposed to exit. Also on receiving SIGINT/SIGALRM twice, abort iteration immediately with tracefs_iterate_stop, making rtla exit right away instead of waiting for all events to be processed. - Account for missed events Due to tracefs buffer overflow, it can happen that rtla misses events, making the tracing results inaccurate. Count both the number of missed events and the total number of processed events, and display missed events as well as their percentage. The numbers are displayed for both osnoise and timerlat, even though for the earlier, missed events are generally not expected. For hist, the number is displayed at the end of the run; for top, it is displayed on each printing of the top table. - Changes to make osnoise more robust There was a dependency in the code that the first field of the osnoise_tool structure was the trace field. If that that ever changed, then the code work break. Change the code to encapsulate this dependency where the code that uses the structure does not have this dependency. * tag 'trace-tools-v6.14' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (22 commits) rtla: Report missed event count rtla: Add function to report missed events rtla: Count all processed events rtla: Count missed trace events tools/rtla: Add osnoise_trace_is_off() rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads rtla/osnoise: Distinguish missing workload option rtla/timerlat_top: Abort event processing on second signal rtla/timerlat_hist: Abort event processing on second signal rtla/timerlat_top: Stop timerlat tracer on signal rtla/timerlat_hist: Stop timerlat tracer on signal rtla: Add trace_instance_stop tools/rtla: Add basic test suite verification/dot2k: Implement event type detection verification/dot2k: Auto patch current kernel source verification/dot2k: Simplify manual steps in monitor creation rv: Simplify manual steps in monitor creation verification/dot2k: Add support for name and description options verification/dot2k: More robust template variables ...
2025-01-24rtla: Report missed event countTomas Glozar
Print how many events were missed by trace buffer overflow in the main instance at the end of the run (for hist) or during the run (for top). Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250123142339.990300-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Tested-by: Gabriele Monaco <gmonaco@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla: Add function to report missed eventsTomas Glozar
Add osnoise_report_missed_events to be used to report the number of missed events either during or after an osnoise or timerlat run. Also, display the percentage of missed events compared to the total number of received events. If an unknown number of missed events was reported during the run, the entire number of missed events is reported as unknown. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250123142339.990300-4-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla: Count all processed eventsTomas Glozar
Add a field processed_events to struct trace_instance and increment it in collect_registered_events, regardless of whether a handler is registered for the event. The purpose is to calculate the percentage of events that were missed due to tracefs buffer overflow. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250123142339.990300-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla: Count missed trace eventsTomas Glozar
Add function collect_missed_events to trace.c to act as a callback for tracefs_follow_missed_events, summing the number of total missed events into a new field missing_events of struct trace_instance. In case record->missed_events is negative, trace->missed_events is set to UINT64_MAX to signify an unknown number of events was missed. The callback is activated on initialization of the trace instance. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250123142339.990300-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24tools/rtla: Add osnoise_trace_is_off()Costa Shulyupin
All of the users of trace_is_off() passes in &record->trace as the second parameter, where record is a pointer to a struct osnoise_tool. This record could be NULL and there is a hidden dependency that the trace field is the first field to allow &record->trace to work with a NULL record pointer. In order to make this code a bit more robust, as record shouldn't be dereferenced if it is NULL, even if the code does work, create a new function called osnoise_trace_is_off() that takes the pointer to a struct osnoise_tool as its second parameter. This way it can properly test if it is NULL before it dereferences it. The old function trace_is_off() is removed and the function osnoise_trace_is_off() is added into osnoise.c which is what the struct osnoise_tool is associated with. Cc: John Kacur <jkacur@redhat.com> Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com> Cc: Eder Zulian <ezulian@redhat.com> Cc: Dan Carpenter <dan.carpenter@linaro.org> Cc: Tomas Glozar <tglozar@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250115180055.2136815-1-costa.shul@redhat.com Signed-off-by: Costa Shulyupin <costa.shul@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat top -u ^\Quit (core dumped) $ rtla timerlat top -k -d 1s Timer Latency 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-4-tglozar@redhat.com Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat hist -u ^\Quit (core dumped) $ rtla timerlat hist -k -d 1s Index over: count: min: avg: max: ALL: IRQ Thr Usr count: 0 0 0 min: - - - avg: - - - max: - - - The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-3-tglozar@redhat.com Fixes: ed774f7481fa ("rtla/timerlat_hist: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/osnoise: Distinguish missing workload optionTomas Glozar
osnoise_set_workload returns -1 for both missing OSNOISE_WORKLOAD option and failure in setting the option. Return -1 for missing and -2 for failure to distinguish them. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_top: Abort event processing on second signalTomas Glozar
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately. This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_hist: Abort event processing on second signalTomas Glozar
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately. This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_top: Stop timerlat tracer on signalTomas Glozar
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs. In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-4-tglozar@redhat.com Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla/timerlat_hist: Stop timerlat tracer on signalTomas Glozar
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs. In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-3-tglozar@redhat.com Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-24rtla: Add trace_instance_stopTomas Glozar
Support not only turning trace on for the timerlat tracer, but also turning it off. This will be used in subsequent patches to stop the timerlat tracer without also wiping the trace buffer. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2025-01-23tools/rtla: Add basic test suiteTomas Glozar
Implement a simple TAP-based test engine in bash and a few basic tests using it, to be used to check for bugs and regressions. A new "check" target is added to the rtla Makefile that runs the test suite using the "prove" command implemented by Test::Harness. The only test format currently supported is running rtla with defined command arguments per test, checking its exit code. In case the exit code is non-zero, the output of rtla is displayed, together with the exit code. The test cases are adopted from rtla tests in the Continuous Kernel Integration (CKI) project [1] with the authors' approval. [1] https://gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/blob/main/rt-tests/us/rtla/ Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Chang Yin <cyin@redhat.com> Cc: Qiao Zhao <qzhao@redhat.com> Link: https://lore.kernel.org/20250120135630.802111-1-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-12-27rtla/timerlat: Fix histogram ALL for zero samplesTomas Glozar
rtla timerlat hist currently computers the minimum, maximum and average latency even in cases when there are zero samples. This leads to nonsensical values being calculated for maximum and minimum, and to divide by zero for average. A similar bug is fixed by 01b05fc0e5f3 ("rtla/timerlat: Fix histogram report when a cpu count is 0") but the bug still remains for printing the sum over all CPUs in timerlat_print_stats_all. The issue can be reproduced with this command: $ rtla timerlat hist -U -d 1s Index over: count: min: avg: max: Floating point exception (core dumped) (There are always no samples with -U unless the user workload is created.) Fix the bug by omitting max/min/avg when sample count is zero, displaying a dash instead, just like we already do for the individual CPUs. The logic is moved into a new function called format_summary_value, which is used for both the individual CPUs and for the overall summary. Cc: stable@vger.kernel.org Link: https://lore.kernel.org/20241127134130.51171-1-tglozar@redhat.com Fixes: 1462501c7a8 ("rtla/timerlat: Add a summary for hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-11-22Merge tag 'trace-tools-v6.13' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing tools updates from Steven Rostedt: - Add ':' to getopt option 'trace-buffer-size' in timerlat_hist for consistency - Remove unused sched_getattr define - Rename sched_setattr() helper to syscall_sched_setattr() to avoid conflicts - Update counters to long from int to avoid overflow - Add libcpupower dependency detection - Add --deepest-idle-state to timerlat to limit deep idle sleeps - Other minor clean ups and documentation changes * tag 'trace-tools-v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: verification/dot2: Improve dot parser robustness tools/rtla: Improve exception handling in timerlat_load.py tools/rtla: Enhance argument parsing in timerlat_load.py tools/rtla: Improve code readability in timerlat_load.py rtla/timerlat: Do not set params->user_workload with -U rtla: Documentation: Mention --deepest-idle-state rtla/timerlat: Add --deepest-idle-state for hist rtla/timerlat: Add --deepest-idle-state for top rtla/utils: Add idle state disabling via libcpupower rtla: Add optional dependency on libcpupower tools/build: Add libcpupower dependency detection rtla/timerlat: Make timerlat_hist_cpu->*_count unsigned long long rtla/timerlat: Make timerlat_top_cpu->*_count unsigned long long tools/rtla: fix collision with glibc sched_attr/sched_set_attr tools/rtla: drop __NR_sched_getattr rtla: Fix consistency in getopt_long for timerlat_hist rv: Fix a typo tools/rv: Correct the grammatical errors in the comments tools/rv: Correct the grammatical errors in the comments rtla: use the definition for stdout fd when calling isatty()
2024-11-19tools/rtla: Improve exception handling in timerlat_load.pyfurkanonder
The enhancements made to timerlat_load.py are intended to improve the script's exception handling. Summary of the changes: - Specific exceptions are now caught for CPU affinity and priority settings, with clearer error messages provided. - The timerlat file descriptor opening now includes handling for PermissionError and OSError, with informative messages. - In the infinite loop, generic exceptions have been replaced with specific types like KeyboardInterrupt and IOError, improving feedback. Before: $ sudo python timerlat_load.py 122 Error setting affinity After: $ sudo python timerlat_load.py 122 Error setting affinity: [Errno 22] Invalid argument Before: $ sudo python timerlat_load.py 1 -p 950 Error setting priority After: $ sudo python timerlat_load.py 1 -p 950 Error setting priority: [Errno 22] Invalid argument Before: $ python timerlat_load.py 1 Error opening timerlat fd, did you run timerlat -U? After: $ python timerlat_load.py 1 Permission denied. Please check your access rights. Cc: "lgoncalv@redhat.com" <lgoncalv@redhat.com> Cc: "jkacur@redhat.com" <jkacur@redhat.com> Link: https://lore.kernel.org/Q_k1s4hBtUy2px8ou0QKenjEK2_T_LoV8IxAE79aBakBogb-7uHp2fpET3oWtI1t3dy8uKjWeRzQOdKNzIzOOpyM4OjutJOriZ9TrGY6b-g=@protonmail.com Signed-off-by: Furkan Onder <furkanonder@protonmail.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-11-19tools/rtla: Enhance argument parsing in timerlat_load.pyfurkanonder
The enhancements made to timerlat_load.py are aimed at improving the clarity of argument parsing. Summary of Changes: - The cpu argument is now specified as an integer type in the argument parser to enforce input validation, and the construction of affinity_mask has been simplified to directly use the integer value of args.cpu. - The prio argument is similarly updated to be of integer type for consistency and validation, eliminating the need for the conversion of args.prio to an integer, as this is now handled by the argument parser. Cc: "jkacur@redhat.com" <jkacur@redhat.com> Cc: "lgoncalv@redhat.com" <lgoncalv@redhat.com> Link: https://lore.kernel.org/QfgO7ayKD9dsLk8_ZDebkAV0OF7wla7UmasbP9CBmui_sChOeizy512t3RqCHTjvQoUBUDP8dwEOVCdHQ5KvVNEiP69CynMY94SFDERWl94=@protonmail.com Signed-off-by: Furkan Onder <furkanonder@protonmail.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-11-19tools/rtla: Improve code readability in timerlat_load.pyfurkanonder
The enhancements made to timerlat_load.py are intended to improve the script's robustness and readability. Summary of the changes: - Unnecessary semicolons at the end of lines have been removed. - Parentheses surrounding the if statement checking args.prio have been eliminated. - String concatenation for constructing timerlat_path has been replaced with an f-string. - Spacing in a multiplication expression has been adjusted for improved clarity. Cc: "jkacur@redhat.com" <jkacur@redhat.com> Cc: "lgoncalv@redhat.com" <lgoncalv@redhat.com> Link: https://lore.kernel.org/j2B-ted7pv3TaldTyqfIHrMmjq2fVyBFgnu3TskiQJsyRzy9loPTVVJoqHnrCWu5T88MDIFc612jUglH6Sxkdg9LN-I1XuITmoL70uECmus=@protonmail.com Signed-off-by: Furkan Onder <furkanonder@protonmail.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-11-19rtla/timerlat: Do not set params->user_workload with -UTomas Glozar
Since commit fb9e90a67ee9 ("rtla/timerlat: Make user-space threads the default"), rtla-timerlat has been defaulting to params->user_workload if neither that or params->kernel_workload is set. This has unintentionally made -U, which sets only params->user_hist/top but not params->user_workload, to behave like -u unless -k is set, preventing the user from running a custom workload. Example: $ rtla timerlat hist -U -c 0 & [1] 7413 $ python sample/timerlat_load.py 0 Error opening timerlat fd, did you run timerlat -U? $ ps | grep timerlatu 7415 pts/4 00:00:00 timerlatu/0 Fix the issue by checking for params->user_top/hist instead of params->user_workload when setting default thread mode. Link: https://lore.kernel.org/20241021123140.14652-1-tglozar@redhat.com Fixes: fb9e90a67ee9 ("rtla/timerlat: Make user-space threads the default") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-17rtla: Documentation: Mention --deepest-idle-stateTomas Glozar
Add --deepest-idle-state to manpage and mention libcpupower dependency in README.txt. Link: https://lore.kernel.org/20241017140914.3200454-7-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-17rtla/timerlat: Add --deepest-idle-state for histTomas Glozar
Support limiting deepest idle state also for timerlat-hist. Link: https://lore.kernel.org/20241017140914.3200454-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-17rtla/timerlat: Add --deepest-idle-state for topTomas Glozar
Add option to limit deepest idle state on CPUs where timerlat is running for the duration of the workload. Link: https://lore.kernel.org/20241017140914.3200454-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-17rtla/utils: Add idle state disabling via libcpupowerTomas Glozar
Add functions to utils.c to disable idle states through functions of libcpupower. This will serve as the basis for disabling idle states per cpu when running timerlat. Link: https://lore.kernel.org/20241017140914.3200454-4-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-17rtla: Add optional dependency on libcpupowerTomas Glozar
If libcpupower is present, set HAVE_LIBCPUPOWER_SUPPORT macro to allow features depending on libcpupower in rtla. Link: https://lore.kernel.org/20241017140914.3200454-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-11rtla/timerlat: Make timerlat_hist_cpu->*_count unsigned long longTomas Glozar
Do the same fix as in previous commit also for timerlat-hist. Link: https://lore.kernel.org/20241011121015.2868751-2-tglozar@redhat.com Reported-by: Attila Fazekas <afazekas@redhat.com> Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-11rtla/timerlat: Make timerlat_top_cpu->*_count unsigned long longTomas Glozar
Most fields of struct timerlat_top_cpu are unsigned long long, but the fields {irq,thread,user}_count are int (32-bit signed). This leads to overflow when tracing on a large number of CPUs for a long enough time: $ rtla timerlat top -a20 -c 1-127 -d 12h ... 0 12:00:00 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 1 #43200096 | 0 0 1 2 | 3 2 6 12 ... 127 #43200096 | 0 0 1 2 | 3 2 5 11 ALL #119144 e4 | 0 5 4 | 2 28 16 The average latency should be 0-1 for IRQ and 5-6 for thread, but is reported as 5 and 28, about 4 to 5 times more, due to the count overflowing when summed over all CPUs: 43200096 * 127 = 5486412192, however, 1191444898 (= 5486412192 mod MAX_INT) is reported instead, as seen on the last line of the output, and the averages are thus ~4.6 times higher than they should be (5486412192 / 1191444898 = ~4.6). Fix the issue by changing {irq,thread,user}_count fields to unsigned long long, similarly to other fields in struct timerlat_top_cpu and to the count variable in timerlat_top_print_sum. Link: https://lore.kernel.org/20241011121015.2868751-1-tglozar@redhat.com Reported-by: Attila Fazekas <afazekas@redhat.com> Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-10tools/rtla: fix collision with glibc sched_attr/sched_set_attrJan Stancek
glibc commit 21571ca0d703 ("Linux: Add the sched_setattr and sched_getattr functions") now also provides 'struct sched_attr' and sched_setattr() which collide with the ones from rtla. In file included from src/trace.c:11: src/utils.h:49:8: error: redefinition of ‘struct sched_attr’ 49 | struct sched_attr { | ^~~~~~~~~~ In file included from /usr/include/bits/sched.h:60, from /usr/include/sched.h:43, from /usr/include/tracefs/tracefs.h:10, from src/trace.c:4: /usr/include/linux/sched/types.h:98:8: note: originally defined here 98 | struct sched_attr { | ^~~~~~~~~~ Define 'struct sched_attr' conditionally, similar to what strace did: https://lore.kernel.org/all/20240930222913.3981407-1-raj.khem@gmail.com/ and rename rtla's version of sched_setattr() to avoid collision. Link: https://lore.kernel.org/8088f66a7a57c1b209cd8ae0ae7c336a7f8c930d.1728572865.git.jstancek@redhat.com Signed-off-by: Jan Stancek <jstancek@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2024-10-10tools/rtla: drop __NR_sched_getattrJan Stancek
It's not used since commit 084ce16df0f0 ("tools/rtla: Remove unused sched_getattr() function"). Link: https://lore.kernel.org/c355dc9ad23470098d6a8d0f31fbd702551c9ea8.1728552769.git.jstancek@redhat.com Signed-off-by: Jan Stancek <jstancek@redhat.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>