Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
xz: Documentation/staging/xz.rst: Revise thoroughly
Add SPDX license identifier.
Omit xz_dec_test info. That isn't relevant to developers of non-XZ code.
Revise the docs about xzkern and add xzkern_with_size. The latter was
added to scripts/Makefile.lib in the commit 7ce7e984ab2b ("kbuild: rename
cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}").
Omit contact info as MAINTAINERS has it.
Omit other info that is outdated or not relevant in the kernel context.
Include the xz_dec kernel-doc from include/linux/xz.h.
Link: https://lkml.kernel.org/r/20240721133633.47721-8-lasse.collin@tukaani.org
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Reviewed-by: Sam James <sam@gentoo.org>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Rui Li <me@lirui.org>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Joel Stanley <joel@jms.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Jubin Zhong <zhongjubin@huawei.com>
Cc: Jules Maselbas <jmaselbas@zdiv.net>
Cc: Krzysztof Kozlowski <krzk@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Simon Glass <sjg@chromium.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
authored by
Lasse Collin
and committed by
Andrew Morton
64167246
0f2c5996
+58
-84
+58
-84
Documentation/staging/xz.rst
+58
-84
Documentation/staging/xz.rst
···
1
+
.. SPDX-License-Identifier: 0BSD
2
+
1
3
============================
2
4
XZ data compression in Linux
3
5
============================
···
8
6
============
9
7
10
8
XZ is a general purpose data compression format with high compression
11
-
ratio and relatively fast decompression. The primary compression
12
-
algorithm (filter) is LZMA2. Additional filters can be used to improve
13
-
compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
14
-
improve compression ratio of executable data.
9
+
ratio. The XZ decompressor in Linux is called XZ Embedded. It supports
10
+
the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters
11
+
for executable code. CRC32 is supported for integrity checking.
15
12
16
-
The XZ decompressor in Linux is called XZ Embedded. It supports
17
-
the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
18
-
for integrity checking. The home page of XZ Embedded is at
19
-
<https://tukaani.org/xz/embedded.html>, where you can find the
20
-
latest version and also information about using the code outside
21
-
the Linux kernel.
13
+
See the `XZ Embedded`_ home page for the latest version which includes
14
+
a few optional extra features that aren't required in the Linux kernel
15
+
and information about using the code outside the Linux kernel.
22
16
23
-
For userspace, XZ Utils provide a zlib-like compression library
24
-
and a gzip-like command line tool. XZ Utils can be downloaded from
25
-
<https://tukaani.org/xz/>.
17
+
For userspace, `XZ Utils`_ provide a zlib-like compression library
18
+
and a gzip-like command line tool.
19
+
20
+
.. _XZ Embedded: https://tukaani.org/xz/embedded.html
21
+
.. _XZ Utils: https://tukaani.org/xz/
26
22
27
23
XZ related components in the kernel
28
24
===================================
29
25
30
26
The xz_dec module provides XZ decompressor with single-call (buffer
31
-
to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
32
-
module is documented in include/linux/xz.h.
33
-
34
-
The xz_dec_test module is for testing xz_dec. xz_dec_test is not
35
-
useful unless you are hacking the XZ decompressor. xz_dec_test
36
-
allocates a char device major dynamically to which one can write
37
-
.xz files from userspace. The decompressed output is thrown away.
38
-
Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
39
-
See the xz_dec_test source code for the details.
27
+
to buffer) and multi-call (stateful) APIs in include/linux/xz.h.
40
28
41
29
For decompressing the kernel image, initramfs, and initrd, there
42
30
is a wrapper function in lib/decompress_unxz.c. Its API is the
43
31
same as in other decompress_*.c files, which is defined in
44
32
include/linux/decompress/generic.h.
45
33
46
-
scripts/xz_wrap.sh is a wrapper for the xz command line tool found
47
-
from XZ Utils. The wrapper sets compression options to values suitable
48
-
for compressing the kernel image.
34
+
For kernel makefiles, three commands are provided for use with
35
+
``$(call if_changed)``. They require the xz tool from XZ Utils.
49
36
50
-
For kernel makefiles, two commands are provided for use with
51
-
$(call if_needed). The kernel image should be compressed with
52
-
$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
53
-
dictionary. It will also append a four-byte trailer containing the
54
-
uncompressed size of the file, which is needed by the boot code.
55
-
Other things should be compressed with $(call if_needed,xzmisc)
56
-
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
37
+
- ``$(call if_changed,xzkern)`` is for compressing the kernel image.
38
+
It runs the script scripts/xz_wrap.sh which uses arch-optimized
39
+
options and a big LZMA2 dictionary.
40
+
41
+
- ``$(call if_changed,xzkern_with_size)`` is like ``xzkern`` above but
42
+
this also appends a four-byte trailer containing the uncompressed size
43
+
of the file. The trailer is needed by the boot code on some archs.
44
+
45
+
- Other things can be compressed with ``$(call if_needed,xzmisc)``
46
+
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
57
47
58
48
Notes on compression options
59
49
============================
60
50
61
-
Since the XZ Embedded supports only streams with no integrity check or
62
-
CRC32, make sure that you don't use some other integrity check type
63
-
when encoding files that are supposed to be decoded by the kernel. With
64
-
liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
65
-
when encoding. With the xz command line tool, use --check=none or
66
-
--check=crc32.
51
+
Since the XZ Embedded supports only streams with CRC32 or no integrity
52
+
check, make sure that you don't use some other integrity check type
53
+
when encoding files that are supposed to be decoded by the kernel.
54
+
With liblzma from XZ Utils, you need to use either ``LZMA_CHECK_CRC32``
55
+
or ``LZMA_CHECK_NONE`` when encoding. With the ``xz`` command line tool,
56
+
use ``--check=crc32`` or ``--check=none`` to override the default
57
+
``--check=crc64``.
67
58
68
59
Using CRC32 is strongly recommended unless there is some other layer
69
60
which will verify the integrity of the uncompressed data anyway.
···
66
71
disable it) for the actual uncompressed data.
67
72
68
73
In userspace, LZMA2 is typically used with dictionary sizes of several
69
-
megabytes. The decoder needs to have the dictionary in RAM, thus big
70
-
dictionaries cannot be used for files that are intended to be decoded
71
-
by the kernel. 1 MiB is probably the maximum reasonable dictionary
72
-
size for in-kernel use (maybe more is OK for initramfs). The presets
73
-
in XZ Utils may not be optimal when creating files for the kernel,
74
-
so don't hesitate to use custom settings. Example::
74
+
megabytes. The decoder needs to have the dictionary in RAM:
75
75
76
-
xz --check=crc32 --lzma2=dict=512KiB inputfile
76
+
- In multi-call mode the dictionary is allocated as part of the
77
+
decoder state. The reasonable maximum dictionary size for in-kernel
78
+
use will depend on the target hardware: a few megabytes is fine for
79
+
desktop systems while 64 KiB to 1 MiB might be more appropriate on
80
+
some embedded systems.
77
81
78
-
An exception to above dictionary size limitation is when the decoder
79
-
is used in single-call mode. Decompressing the kernel itself is an
80
-
example of this situation. In single-call mode, the memory usage
81
-
doesn't depend on the dictionary size, and it is perfectly fine to
82
-
use a big dictionary: for maximum compression, the dictionary should
83
-
be at least as big as the uncompressed data itself.
82
+
- In single-call mode the output buffer is used as the dictionary
83
+
buffer. That is, the size of the dictionary doesn't affect the
84
+
decompressor memory usage at all. Only the base data structures
85
+
are allocated which take a little less than 30 KiB of memory.
86
+
For the best compression, the dictionary should be at least
87
+
as big as the uncompressed data. A notable example of single-call
88
+
mode is decompressing the kernel itself (except on PowerPC).
84
89
85
-
Future plans
86
-
============
90
+
The compression presets in XZ Utils may not be optimal when creating
91
+
files for the kernel, so don't hesitate to use custom settings to,
92
+
for example, set the dictionary size. Also, xz may produce a smaller
93
+
file in single-threaded mode so setting that explicitly is recommended.
94
+
Example::
87
95
88
-
Creating a limited XZ encoder may be considered if people think it is
89
-
useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
90
-
the fastest settings, so it isn't clear if LZMA2 encoder is wanted
91
-
into the kernel.
96
+
xz --threads=1 --check=crc32 --lzma2=dict=512KiB inputfile
92
97
93
-
Support for limited random-access reading is planned for the
94
-
decompression code. I don't know if it could have any use in the
95
-
kernel, but I know that it would be useful in some embedded projects
96
-
outside the Linux kernel.
98
+
xz_dec API
99
+
==========
97
100
98
-
Conformance to the .xz file format specification
99
-
================================================
101
+
This is available with ``#include <linux/xz.h>``.
100
102
101
-
There are a couple of corner cases where things have been simplified
102
-
at expense of detecting errors as early as possible. These should not
103
-
matter in practice all, since they don't cause security issues. But
104
-
it is good to know this if testing the code e.g. with the test files
105
-
from XZ Utils.
103
+
``XZ_EXTERN`` is a macro used in the preboot code. Ignore it when
104
+
reading this documentation.
106
105
107
-
Reporting bugs
108
-
==============
109
-
110
-
Before reporting a bug, please check that it's not fixed already
111
-
at upstream. See <https://tukaani.org/xz/embedded.html> to get the
112
-
latest code.
113
-
114
-
Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
115
-
Freenode and talk to Larhzu. I don't actively read LKML or other
116
-
kernel-related mailing lists, so if there's something I should know,
117
-
you should email to me personally or use IRC.
118
-
119
-
Don't bother Igor Pavlov with questions about the XZ implementation
120
-
in the kernel or about XZ Utils. While these two implementations
121
-
include essential code that is directly based on Igor Pavlov's code,
122
-
these implementations aren't maintained nor supported by him.
106
+
.. kernel-doc:: include/linux/xz.h