Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

docs: Add debugging section to process

This idea was formed after noticing that new developers experience
certain difficulty to navigate within the multitude of different
debugging options in the Kernel and while there often is good
documentation for the tools, the developer has to know first that they
exist and where to find them.
Add a general debugging section to the Kernel documentation, as an
easily locatable entry point to other documentation and as a general
guideline for the topic.

Signed-off-by: Sebastian Fricke <sebastian.fricke@collabora.com>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/20241028-media_docs_improve_v3-v3-1-edf5c5b3746f@collabora.com

authored by

Sebastian Fricke and committed by
Jonathan Corbet
a037699d d8c949c5

+573 -3
+223
Documentation/process/debugging/driver_development_debugging_guide.rst
··· 1 + .. SPDX-License-Identifier: GPL-2.0 2 + 3 + ======================================== 4 + Debugging advice for driver development 5 + ======================================== 6 + 7 + This document serves as a general starting point and lookup for debugging 8 + device drivers. 9 + While this guide focuses on debugging that requires re-compiling the 10 + module/kernel, the :doc:`userspace debugging guide 11 + </process/debugging/userspace_debugging_guide>` will guide 12 + you through tools like dynamic debug, ftrace and other tools useful for 13 + debugging issues and behavior. 14 + For general debugging advice, see the :doc:`general advice document 15 + </process/debugging/index>`. 16 + 17 + .. contents:: 18 + :depth: 3 19 + 20 + The following sections show you the available tools. 21 + 22 + printk() & friends 23 + ------------------ 24 + 25 + These are derivatives of printf() with varying destinations and support for 26 + being dynamically turned on or off, or lack thereof. 27 + 28 + Simple printk() 29 + ~~~~~~~~~~~~~~~ 30 + 31 + The classic, can be used to great effect for quick and dirty development 32 + of new modules or to extract arbitrary necessary data for troubleshooting. 33 + 34 + Prerequisite: ``CONFIG_PRINTK`` (usually enabled by default) 35 + 36 + **Pros**: 37 + 38 + - No need to learn anything, simple to use 39 + - Easy to modify exactly to your needs (formatting of the data (See: 40 + :doc:`/core-api/printk-formats`), visibility in the log) 41 + - Can cause delays in the execution of the code (beneficial to confirm whether 42 + timing is a factor) 43 + 44 + **Cons**: 45 + 46 + - Requires rebuilding the kernel/module 47 + - Can cause delays in the execution of the code (which can cause issues to be 48 + not reproducible) 49 + 50 + For the full documentation see :doc:`/core-api/printk-basics` 51 + 52 + Trace_printk 53 + ~~~~~~~~~~~~ 54 + 55 + Prerequisite: ``CONFIG_DYNAMIC_FTRACE`` & ``#include <linux/ftrace.h>`` 56 + 57 + It is a tiny bit less comfortable to use than printk(), because you will have 58 + to read the messages from the trace file (See: :ref:`read_ftrace_log` 59 + instead of from the kernel log, but very useful when printk() adds unwanted 60 + delays into the code execution, causing issues to be flaky or hidden.) 61 + 62 + If the processing of this still causes timing issues then you can try 63 + trace_puts(). 64 + 65 + For the full Documentation see trace_printk() 66 + 67 + dev_dbg 68 + ~~~~~~~ 69 + 70 + Print statement, which can be targeted by 71 + :ref:`process/debugging/userspace_debugging_guide:dynamic debug` that contains 72 + additional information about the device used within the context. 73 + 74 + **When is it appropriate to leave a debug print in the code?** 75 + 76 + Permanent debug statements have to be useful for a developer to troubleshoot 77 + driver misbehavior. Judging that is a bit more of an art than a science, but 78 + some guidelines are in the :ref:`Coding style guidelines 79 + <process/coding-style:13) printing kernel messages>`. In almost all cases the 80 + debug statements shouldn't be upstreamed, as a working driver is supposed to be 81 + silent. 82 + 83 + Custom printk 84 + ~~~~~~~~~~~~~ 85 + 86 + Example:: 87 + 88 + #define core_dbg(fmt, arg...) do { \ 89 + if (core_debug) \ 90 + printk(KERN_DEBUG pr_fmt("core: " fmt), ## arg); \ 91 + } while (0) 92 + 93 + **When should you do this?** 94 + 95 + It is better to just use a pr_debug(), which can later be turned on/off with 96 + dynamic debug. Additionally, a lot of drivers activate these prints via a 97 + variable like ``core_debug`` set by a module parameter. However, Module 98 + parameters `are not recommended anymore 99 + <https://lore.kernel.org/all/2024032757-surcharge-grime-d3dd@gregkh>`_. 100 + 101 + Ftrace 102 + ------ 103 + 104 + Creating a custom Ftrace tracepoint 105 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 106 + 107 + A tracepoint adds a hook into your code that will be called and logged when the 108 + tracepoint is enabled. This can be used, for example, to trace hitting a 109 + conditional branch or to dump the internal state at specific points of the code 110 + flow during a debugging session. 111 + 112 + Here is a basic description of :ref:`how to implement new tracepoints 113 + <trace/tracepoints:usage>`. 114 + 115 + For the full event tracing documentation see :doc:`/trace/events` 116 + 117 + For the full Ftrace documentation see :doc:`/trace/ftrace` 118 + 119 + DebugFS 120 + ------- 121 + 122 + Prerequisite: ``CONFIG_DEBUG_FS` & `#include <linux/debugfs.h>`` 123 + 124 + DebugFS differs from the other approaches of debugging, as it doesn't write 125 + messages to the kernel log nor add traces to the code. Instead it allows the 126 + developer to handle a set of files. 127 + With these files you can either store values of variables or make 128 + register/memory dumps or you can make these files writable and modify 129 + values/settings in the driver. 130 + 131 + Possible use-cases among others: 132 + 133 + - Store register values 134 + - Keep track of variables 135 + - Store errors 136 + - Store settings 137 + - Toggle a setting like debug on/off 138 + - Error injection 139 + 140 + This is especially useful, when the size of a data dump would be hard to digest 141 + as part of the general kernel log (for example when dumping raw bitstream data) 142 + or when you are not interested in all the values all the time, but with the 143 + possibility to inspect them. 144 + 145 + The general idea is: 146 + 147 + - Create a directory during probe (``struct dentry *parent = 148 + debugfs_create_dir("my_driver", NULL);``) 149 + - Create a file (``debugfs_create_u32("my_value", 444, parent, &my_variable);``) 150 + 151 + - In this example the file is found in 152 + ``/sys/kernel/debug/my_driver/my_value`` (with read permissions for 153 + user/group/all) 154 + - any read of the file will return the current contents of the variable 155 + ``my_variable`` 156 + 157 + - Clean up the directory when removing the device 158 + (``debugfs_remove_recursive(parent);``) 159 + 160 + For the full documentation see :doc:`/filesystems/debugfs`. 161 + 162 + KASAN, UBSAN, lockdep and other error checkers 163 + ---------------------------------------------- 164 + 165 + KASAN (Kernel Address Sanitizer) 166 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 167 + 168 + Prerequisite: ``CONFIG_KASAN`` 169 + 170 + KASAN is a dynamic memory error detector that helps to find use-after-free and 171 + out-of-bounds bugs. It uses compile-time instrumentation to check every memory 172 + access. 173 + 174 + For the full documentation see :doc:`/dev-tools/kasan`. 175 + 176 + UBSAN (Undefined Behavior Sanitizer) 177 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 178 + 179 + Prerequisite: ``CONFIG_UBSAN`` 180 + 181 + UBSAN relies on compiler instrumentation and runtime checks to detect undefined 182 + behavior. It is designed to find a variety of issues, including signed integer 183 + overflow, array index out of bounds, and more. 184 + 185 + For the full documentation see :doc:`/dev-tools/ubsan` 186 + 187 + lockdep (Lock Dependency Validator) 188 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 189 + 190 + Prerequisite: ``CONFIG_DEBUG_LOCKDEP`` 191 + 192 + lockdep is a runtime lock dependency validator that detects potential deadlocks 193 + and other locking-related issues in the kernel. 194 + It tracks lock acquisitions and releases, building a dependency graph that is 195 + analyzed for potential deadlocks. 196 + lockdep is especially useful for validating the correctness of lock ordering in 197 + the kernel. 198 + 199 + PSI (Pressure stall information tracking) 200 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 201 + 202 + Prerequisite: ``CONFIG_PSI`` 203 + 204 + PSI is a measurement tool to identify excessive overcommits on hardware 205 + resources, that can cause performance disruptions or even OOM kills. 206 + 207 + device coredump 208 + --------------- 209 + 210 + Prerequisite: ``#include <linux/devcoredump.h>`` 211 + 212 + Provides the infrastructure for a driver to provide arbitrary data to userland. 213 + It is most often used in conjunction with udev or similar userland application 214 + to listen for kernel uevents, which indicate that the dump is ready. Udev has 215 + rules to copy that file somewhere for long-term storage and analysis, as by 216 + default, the data for the dump is automatically cleaned up after 5 minutes. 217 + That data is analyzed with driver-specific tools or GDB. 218 + 219 + You can find an example implementation at: 220 + `drivers/media/platform/qcom/venus/core.c 221 + <https://elixir.bootlin.com/linux/v6.11.6/source/drivers/media/platform/qcom/venus/core.c#L30>`__ 222 + 223 + **Copyright** ©2024 : Collabora
+65
Documentation/process/debugging/index.rst
··· 1 + .. SPDX-License-Identifier: GPL-2.0 2 + 3 + ============================================ 4 + Debugging advice for Linux Kernel developers 5 + ============================================ 6 + 7 + .. toctree:: 8 + :maxdepth: 1 9 + 10 + driver_development_debugging_guide 11 + userspace_debugging_guide 12 + 13 + .. only:: subproject and html 14 + 15 + Indices 16 + ======= 17 + 18 + * :ref:`genindex` 19 + 20 + General debugging advice 21 + ======================== 22 + 23 + Depending on the issue, a different set of tools is available to track down the 24 + problem or even to realize whether there is one in the first place. 25 + 26 + As a first step you have to figure out what kind of issue you want to debug. 27 + Depending on the answer, your methodology and choice of tools may vary. 28 + 29 + Do I need to debug with limited access? 30 + --------------------------------------- 31 + 32 + Do you have limited access to the machine or are you unable to stop the running 33 + execution? 34 + 35 + In this case your debugging capability depends on built-in debugging support of 36 + provided distribution kernel. 37 + The :doc:`/process/debugging/userspace_debugging_guide` provides a brief 38 + overview over a range of possible debugging tools in that situation. You can 39 + check the capability of your kernel, in most cases, by looking into config file 40 + within the /boot directory. 41 + 42 + Do I have root access to the system? 43 + ------------------------------------ 44 + 45 + Are you easily able to replace the module in question or to install a new 46 + kernel? 47 + 48 + In that case your range of available tools is a lot bigger, you can find the 49 + tools in the :doc:`/process/debugging/driver_development_debugging_guide`. 50 + 51 + Is timing a factor? 52 + ------------------- 53 + 54 + It is important to understand if the problem you want to debug manifests itself 55 + consistently (i.e. given a set of inputs you always get the same, incorrect 56 + output), or inconsistently. If it manifests itself inconsistently, some timing 57 + factor might be at play. If inserting delays into the code does change the 58 + behavior, then quite likely timing is a factor. 59 + 60 + When timing does alter the outcome of the code execution using a simple 61 + printk() for debugging purposes may not work, a similar alternative is to use 62 + trace_printk() , which logs the debug messages to the trace file instead of the 63 + kernel log. 64 + 65 + **Copyright** ©2024 : Collabora
+280
Documentation/process/debugging/userspace_debugging_guide.rst
··· 1 + .. SPDX-License-Identifier: GPL-2.0 2 + 3 + ========================== 4 + Userspace debugging advice 5 + ========================== 6 + 7 + This document provides a brief overview of common tools to debug the Linux 8 + Kernel from userspace. 9 + For debugging advice aimed at driver developers go :doc:`here 10 + </process/debugging/driver_development_debugging_guide>`. 11 + For general debugging advice, see :doc:`general advice document 12 + </process/debugging/index>`. 13 + 14 + .. contents:: 15 + :depth: 3 16 + 17 + The following sections show you the available tools. 18 + 19 + Dynamic debug 20 + ------------- 21 + 22 + Mechanism to filter what ends up in the kernel log by dis-/en-abling log 23 + messages. 24 + 25 + Prerequisite: ``CONFIG_DYNAMIC_DEBUG`` 26 + 27 + Dynamic debug is only able to target: 28 + 29 + - pr_debug() 30 + - dev_dbg() 31 + - print_hex_dump_debug() 32 + - print_hex_dump_bytes() 33 + 34 + Therefore the usability of this tool is, as of now, quite limited as there is 35 + no uniform rule for adding debug prints to the codebase, resulting in a variety 36 + of ways these prints are implemented. 37 + 38 + Also, note that most debug statements are implemented as a variation of 39 + dprintk(), which have to be activated via a parameter in respective module, 40 + dynamic debug is unable to do that step for you. 41 + 42 + Here is one example, that enables all available pr_debug()'s within the file:: 43 + 44 + $ alias ddcmd='echo $* > /proc/dynamic_debug/control' 45 + $ ddcmd '-p; file v4l2-h264.c +p' 46 + $ grep =p /proc/dynamic_debug/control 47 + drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =p 48 + "ref_pic_list_b%u (cur_poc %u%c) %s" 49 + drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =p 50 + "ref_pic_list_p (cur_poc %u%c) %s\n" 51 + 52 + **When should you use this over Ftrace ?** 53 + 54 + - When the code contains one of the valid print statements (see above) or when 55 + you have added multiple pr_debug() statements during development 56 + - When timing is not an issue, meaning if multiple pr_debug() statements in 57 + the code won't cause delays 58 + - When you care more about receiving specific log messages than tracing the 59 + pattern of how a function is called 60 + 61 + For the full documentation see :doc:`/admin-guide/dynamic-debug-howto` 62 + 63 + Ftrace 64 + ------ 65 + 66 + Prerequisite: ``CONFIG_DYNAMIC_FTRACE`` 67 + 68 + This tool uses the tracefs file system for the control files and output files. 69 + That file system will be mounted as a ``tracing`` directory, which can be found 70 + in either ``/sys/kernel/`` or ``/sys/debug/kernel/``. 71 + 72 + Some of the most important operations for debugging are: 73 + 74 + - You can perform a function trace by adding a function name to the 75 + ``set_ftrace_filter`` file (which accepts any function name found within the 76 + ``available_filter_functions`` file) or you can specifically disable certain 77 + functions by adding their names to the ``set_ftrace_notrace`` file (more info 78 + at: :ref:`trace/ftrace:dynamic ftrace`). 79 + - In order to find out where calls originate from you can activate the 80 + ``func_stack_trace`` option under ``options/func_stack_trace``. 81 + - Tracing the children of a function call and showing the return values are 82 + possible by adding the desired function in the ``set_graph_function`` file 83 + (requires config ``FUNCTION_GRAPH_RETVAL``); more info at 84 + :ref:`trace/ftrace:dynamic ftrace with the function graph tracer`. 85 + 86 + For the full Ftrace documentation see :doc:`/trace/ftrace` 87 + 88 + Or you could also trace for specific events by :ref:`using event tracing 89 + <trace/events:2. using event tracing>`, which can be defined as described here: 90 + :ref:`Creating a custom Ftrace tracepoint 91 + <process/debugging/driver_development_debugging_guide:ftrace>`. 92 + 93 + For the full Ftrace event tracing documentation see :doc:`/trace/events` 94 + 95 + .. _read_ftrace_log: 96 + 97 + Reading the ftrace log 98 + ~~~~~~~~~~~~~~~~~~~~~~ 99 + 100 + The ``trace`` file can be read just like any other file (``cat``, ``tail``, 101 + ``head``, ``vim``, etc.), the size of the file is limited by the 102 + ``buffer_size_kb`` (``echo 1000 > buffer_size_kb``). The 103 + :ref:`trace/ftrace:trace_pipe` will behave similarly to the ``trace`` file, but 104 + whenever you read from the file the content is consumed. 105 + 106 + Kernelshark 107 + ~~~~~~~~~~~ 108 + 109 + A GUI interface to visualize the traces as a graph and list view from the 110 + output of the `trace-cmd 111 + <https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/>`__ application. 112 + 113 + For the full documentation see `<https://kernelshark.org/Documentation.html>`__ 114 + 115 + Perf & alternatives 116 + ------------------- 117 + 118 + The tools mentioned above provide ways to inspect kernel code, results, 119 + variable values, etc. Sometimes you have to find out first where to look and 120 + for those cases, a box of performance tracking tools can help you to frame the 121 + issue. 122 + 123 + Why should you do a performance analysis? 124 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 125 + 126 + A performance analysis is a good first step when among other reasons: 127 + 128 + - you cannot define the issue 129 + - you do not know where it occurs 130 + - the running system should not be interrupted or it is a remote system, where 131 + you cannot install a new module/kernel 132 + 133 + How to do a simple analysis with linux tools? 134 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 135 + 136 + For the start of a performance analysis, you can start with the usual tools 137 + like: 138 + 139 + - ``top`` / ``htop`` / ``atop`` (*get an overview of the system load, see 140 + spikes on specific processes*) 141 + - ``mpstat -P ALL`` (*look at the load distribution among CPUs*) 142 + - ``iostat -x`` (*observe input and output devices utilization and performance*) 143 + - ``vmstat`` (*overview of memory usage on the system*) 144 + - ``pidstat`` (*similar to* ``vmstat`` *but per process, to dial it down to the 145 + target*) 146 + - ``strace -tp $PID`` (*once you know the process, you can figure out how it 147 + communicates with the Kernel*) 148 + 149 + These should help to narrow down the areas to look at sufficiently. 150 + 151 + Diving deeper with perf 152 + ~~~~~~~~~~~~~~~~~~~~~~~ 153 + 154 + The **perf** tool provides a series of metrics and events to further dial down 155 + on issues. 156 + 157 + Prerequisite: build or install perf on your system 158 + 159 + Gather statistics data for finding all files starting with ``gcc`` in ``/usr``:: 160 + 161 + # perf stat -d find /usr -name 'gcc*' | wc -l 162 + 163 + Performance counter stats for 'find /usr -name gcc*': 164 + 165 + 1277.81 msec task-clock # 0.997 CPUs utilized 166 + 9 context-switches # 7.043 /sec 167 + 1 cpu-migrations # 0.783 /sec 168 + 704 page-faults # 550.943 /sec 169 + 766548897 cycles # 0.600 GHz (97.15%) 170 + 798285467 instructions # 1.04 insn per cycle (97.15%) 171 + 57582731 branches # 45.064 M/sec (2.85%) 172 + 3842573 branch-misses # 6.67% of all branches (97.15%) 173 + 281616097 L1-dcache-loads # 220.390 M/sec (97.15%) 174 + 4220975 L1-dcache-load-misses # 1.50% of all L1-dcache accesses (97.15%) 175 + <not supported> LLC-loads 176 + <not supported> LLC-load-misses 177 + 178 + 1.281746009 seconds time elapsed 179 + 180 + 0.508796000 seconds user 181 + 0.773209000 seconds sys 182 + 183 + 184 + 52 185 + 186 + The availability of events and metrics depends on the system you are running. 187 + 188 + For the full documentation see 189 + `<https://perf.wiki.kernel.org/index.php/Main_Page>`__ 190 + 191 + Perfetto 192 + ~~~~~~~~ 193 + 194 + A set of tools to measure and analyze how well applications and systems perform. 195 + You can use it to: 196 + 197 + * identify bottlenecks 198 + * optimize code 199 + * make software run faster and more efficiently. 200 + 201 + **What is the difference between perfetto and perf?** 202 + 203 + * perf is tool as part of and specialized for the Linux Kernel and has CLI user 204 + interface. 205 + * perfetto cross-platform performance analysis stack, has extended 206 + functionality into userspace and provides a WEB user interface. 207 + 208 + For the full documentation see `<https://perfetto.dev/docs/>`__ 209 + 210 + Kernel panic analysis tools 211 + --------------------------- 212 + 213 + To capture the crash dump please use ``Kdump`` & ``Kexec``. Below you can find 214 + some advice for analysing the data. 215 + 216 + For the full documentation see the :doc:`/admin-guide/kdump/kdump` 217 + 218 + In order to find the corresponding line in the code you can use `faddr2line 219 + <https://elixir.bootlin.com/linux/v6.11.6/source/scripts/faddr2line>`__; note 220 + that you need to enable ``CONFIG_DEBUG_INFO`` for that to work. 221 + 222 + An alternative to using ``faddr2line`` is the use of ``objdump`` (and its 223 + derivatives for the different platforms like ``aarch64-linux-gnu-objdump``). 224 + Take this line as an example: 225 + 226 + ``[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]``. 227 + 228 + We can find the corresponding line of code by executing:: 229 + 230 + aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vdec.ko | grep rkvdec_device_run\>: -A 40 231 + 0000000000000ac8 <rkvdec_device_run>: 232 + ac8: d503201f nop 233 + acc: d503201f nop 234 + { 235 + ad0: d503233f paciasp 236 + ad4: a9bd7bfd stp x29, x30, [sp, #-48]! 237 + ad8: 910003fd mov x29, sp 238 + adc: a90153f3 stp x19, x20, [sp, #16] 239 + ae0: a9025bf5 stp x21, x22, [sp, #32] 240 + const struct rkvdec_coded_fmt_desc *desc = ctx->coded_fmt_desc; 241 + ae4: f9411814 ldr x20, [x0, #560] 242 + struct rkvdec_dev *rkvdec = ctx->dev; 243 + ae8: f9418015 ldr x21, [x0, #768] 244 + if (WARN_ON(!desc)) 245 + aec: b4000654 cbz x20, bb4 <rkvdec_device_run+0xec> 246 + ret = pm_runtime_resume_and_get(rkvdec->dev); 247 + af0: f943d2b6 ldr x22, [x21, #1952] 248 + ret = __pm_runtime_resume(dev, RPM_GET_PUT); 249 + af4: aa0003f3 mov x19, x0 250 + af8: 52800081 mov w1, #0x4 // #4 251 + afc: aa1603e0 mov x0, x22 252 + b00: 94000000 bl 0 <__pm_runtime_resume> 253 + if (ret < 0) { 254 + b04: 37f80340 tbnz w0, #31, b6c <rkvdec_device_run+0xa4> 255 + dev_warn(rkvdec->dev, "Not good\n"); 256 + b08: f943d2a0 ldr x0, [x21, #1952] 257 + b0c: 90000001 adrp x1, 0 <rkvdec_try_ctrl-0x8> 258 + b10: 91000021 add x1, x1, #0x0 259 + b14: 94000000 bl 0 <_dev_warn> 260 + *bad = 1; 261 + b18: d2800001 mov x1, #0x0 // #0 262 + ... 263 + 264 + Meaning, in this line from the crash dump:: 265 + 266 + [ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec] 267 + 268 + I can take the ``0x50`` as offset, which I have to add to the base address 269 + of the corresponding function, which I find in this line:: 270 + 271 + 0000000000000ac8 <rkvdec_device_run>: 272 + 273 + The result of ``0xac8 + 0x50 = 0xb18`` 274 + And when I search for that address within the function I get the 275 + following line:: 276 + 277 + *bad = 1; 278 + b18: d2800001 mov x1, #0x0 279 + 280 + **Copyright** ©2024 : Collabora
+5 -3
Documentation/process/index.rst
··· 72 72 Dealing with bugs 73 73 ----------------- 74 74 75 - Bugs are a fact of life; it is important that we handle them properly. 76 - The documents below describe our policies around the handling of a couple 77 - of special classes of bugs: regressions and security problems. 75 + Bugs are a fact of life; it is important that we handle them properly. The 76 + documents below provide general advice about debugging and describe our 77 + policies around the handling of a couple of special classes of bugs: 78 + regressions and security problems. 78 79 79 80 .. toctree:: 80 81 :maxdepth: 1 81 82 83 + debugging/index 82 84 handling-regressions 83 85 security-bugs 84 86 cve