Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge branch 'add-btf-layout-to-btf'
Alan Maguire says:
====================
Add BTF layout to BTF
Update struct btf_header to add a new "layout" section containing
a description of how to parse the BTF kinds known about at BTF
encoding time. This provides the opportunity for tools that might
not know all of these kinds - as is the case when older tools run
on more newly-generated BTF - to still parse the BTF provided,
even if it cannot all be used.
The ideas here were discussed at [1], with further discussion
at [2].
Patches for pahole will enable the layout addition during BTF
generation are at [3], but even absent these the addition of the
layout feature in the final patch in this series should not break
anything since such unknown features are simply ignored during pahole
BTF generation.
Separately tested sanitization of BTF location info with separate
small series which simulates feature absence to support testing of
features for older kernels; will follow up with that shortly.
Changes since v15 [4]:
- Fixed endian issues for layout section by swapping flags fields
where needed (sashiko.dev, patch 2)
- Fixed string size issue with swapped endian case, use btf->magic
for comparison to determine endian mismatch (bpf review bot,
sashiko.dev, patch 6)
Changes since v14 [5]:
- Fix potential overflow for swapped endian case (BPF review bot,
patch 2)
- Add global: keyword to libbpf.map (sashiko.dev, patch 4)
- Fix endian issues in sanitization; we use the endian safe
btf->hdr and check for endian mismatch between it and raw original
BTF header to inform how we write the change str_off. Also fix
potential truncation issues due to not including hdr->type_off
(sashiko.dev, patch 6)
- Fix issues with selftests raw BTF file interactions (sashiko.dev,
patch 8)
- Drop feature test test since it will be covered by another series
Changes since v13: [6]:
- add feature check/sanitization of BTF with layout info (Andrii,
patch 6)
- added feature check test for layout support (patch 9)
Changes since v12: [7]:
- add logging of layout off/len to kernel header logging (review bot,
patch 6)
- add mode to open() in selftest (review bot, patch 7)
Changes since v11 [8]:
- Revert unneeded changes to btf__new_empty() (Eduard, review bot,
patch 4)
- Reorder btf_parse_layout_sec() checks to ensure min size check
occurs before multiple check (review bot, patch 6)
Changes since v10 [9]:
- deal with read overflow with small header (review bot, patch 2)
- validate layout length is a multiple of sizeof(struct btf_layout)
(review bot, patch 6)
- fix comment style (Alexei, patches 4,7)
- remove bpftool BTF metadata subcommands for now (Alexei)
Changes since v9: [10]:
- fix memcpy header size overrun (review bot, patch 2)
- return size computation directly (Andrii, patch 333)
- revert to original unknown kind logging (Alexei/review bot,
patch 6)
- gap-checking logic can be simplified now that we have
4-byte aligned types and layout together (patch 6)
- fix naming of layout offset, unconditionally emit a layouts
array in json (Quentin, review bot, patch 8)
- fix metadata output in man page to include flags (review bot,
patch 9)
Changes since v8: [11]:
- updated name from "kind_layout" to "layout" (Andrii)
- moved layout info to inbetween types and strings since
both types and layout info align on 4 bytes (Andrii)
- use embedded btf_header (Eduard)
- when consulting layout, fall back to base BTF if none found in
split BTF; this will allow us to only encode layout info in
vmlinux rather than repeating it for each module.
Changes since v7: [12]:
- Fixed comment style in UAPI headers (Mykyta, patch 1)
- Simplify calcuation of header size using min() (Mykyta, patch 2)
- simplify computation of bounds for kind (Mykyta, patch 3)
- Added utility functions for updating type, string offsets when
data is added; this simplifies the code and encapsulates such
updates more clearly (patch 2)
Changes since v6: [13]:
- BPF review bot caught some memory leaks around freeing
of kind layout; more importantly, it noted that we were
breaking with the contiguous BTF representation for
btf_new_empty_opts(). Doing so meant that freeing kind_layout
could not be predicated on having btf->modifiable set, so
adpoted the contiguous raw data layout for BTF to be
consistent with type/string storage (patches 2,4)
- Moved checks for kind overflow prior to referencing kinds
to avoid any risk of overrun (patches 3, 8)
- Tightened up kind layout header offset/len header validation
to catch invalid combinations early in btf_parse_hdr()
(patch 2)
- Fixed selftest to verify calloc success (patch 7)
Changes since v5: [14]:
- removed flags field from kind layout; it is not really workable
since we would have to define semantics of all possible future
flags today to be usable. Instead stick to parsing only, which
means each kind just needs the length of the singular and
vlen-specified objects (Alexei)
- added documentation for bpftool BTF metadata dump (Quentin, patch 9)
Changes since v4: [15]:
- removed CRC generation since it is not needed to handle modules
built at different time than kernel; distilled base BTF supports
this now
- fixed up bpftool display of empty kind names, comment/documentation
indentation (Quentin, patches 8, 9)
Changes since v3 [16]:
- fixed mismerge issues with kbuild changes for BTF generation
(patches 9, 14)
- fixed a few small issues in libbpf with kind layout representation
(patches 2, 4)
Changes since v2 [17]:
- drop "optional" kind flag (Andrii, patch 1)
- allocate "struct btf_header" for struct btf to ensure
we can always access new fields (Andrii, patch 2)
- use an internal BTF kind array in btf.c to simplify
kind encoding (Andrii, patch 2)
- drop use of kind layout information for in-kernel parsing,
since the kernel needs to be strict in what it accepts
(Andrii, patch 6)
- added CRC verification for BTF objects and for matching
with base object (Alexei, patches 7,8)
- fixed bpftool json output (Quentin, patch 10)
- added standalone module BTF support, tests (patches 13-17)
Changes since RFC
- Terminology change from meta -> kind_layout
(Alexei and Andrii)
- Simplify representation, removing meta header
and just having kind layout section (Alexei)
- Fixed bpftool to have JSON support, support
prefix match, documented changes (Quentin)
- Separated metadata opts into add_kind_layout
and add_crc
- Added additional positive/negative tests
to cover basic unknown kind, one with an
info_sz object following it and one with
N elem_sz elements following it.
- Updated pahole-flags to use help output
rather than version to see if features
are present
[1] https://lore.kernel.org/bpf/CAEf4BzYjWHRdNNw4B=eOXOs_ONrDwrgX4bn=Nuc1g8JPFC34MA@mail.gmail.com/
[2] https://lore.kernel.org/bpf/20230531201936.1992188-1-alan.maguire@oracle.com/
[3] https://lore.kernel.org/dwarves/20260226085240.1908874-1-alan.maguire@oracle.com/
[4] https://lore.kernel.org/bpf/20260324174450.1570809-1-alan.maguire@oracle.com/
[5] https://lore.kernel.org/bpf/20260318132927.1142388-1-alan.maguire@oracle.com/
[6] https://lore.kernel.org/bpf/20260306113630.1281527-1-alan.maguire@oracle.com/
[7] https://lore.kernel.org/bpf/20260303182003.117483-1-alan.maguire@oracle.com/
[8] https://lore.kernel.org/bpf/20260302114059.3697879-1-alan.maguire@oracle.com/
[9] https://lore.kernel.org/bpf/20260227100426.2585191-1-alan.maguire@oracle.com/
[10] https://lore.kernel.org/bpf/20260226085624.1909682-1-alan.maguire@oracle.com/
[11] https://lore.kernel.org/bpf/20251215091730.1188790-1-alan.maguire@oracle.com/
[12] https://lore.kernel.org/dwarves/20251211164646.1219122-1-alan.maguire@oracle.com/
[13] https://lore.kernel.org/bpf/20251210203243.814529-1-alan.maguire@oracle.com/
[14] https://lore.kernel.org/bpf/20250528095743.791722-1-alan.maguire@oracle.com/
[15] https://lore.kernel.org/bpf/20231112124834.388735-1-alan.maguire@oracle.com/
[16] https://lore.kernel.org/bpf/20231110110304.63910-1-alan.maguire@oracle.com/
[17] https://lore.kernel.org/bpf/20230616171728.530116-1-alan.maguire@oracle.com/
====================
Link: https://patch.msgid.link/20260326145444.2076244-1-alan.maguire@oracle.com
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
+892
-211
+12
include/uapi/linux/btf.h
+12
include/uapi/linux/btf.h
···
8
8
#define BTF_MAGIC 0xeB9F
9
9
#define BTF_VERSION 1
10
10
11
+
/*
12
+
* BTF layout section consists of a struct btf_layout for each known
13
+
* kind at BTF encoding time.
14
+
*/
15
+
struct btf_layout {
16
+
__u8 info_sz; /* size of singular element after btf_type */
17
+
__u8 elem_sz; /* size of each of btf_vlen(t) elements */
18
+
__u16 flags; /* currently unused */
19
+
};
20
+
11
21
struct btf_header {
12
22
__u16 magic;
13
23
__u8 version;
···
29
19
__u32 type_len; /* length of type section */
30
20
__u32 str_off; /* offset of string section */
31
21
__u32 str_len; /* length of string section */
22
+
__u32 layout_off; /* offset of layout section */
23
+
__u32 layout_len; /* length of layout section */
32
24
};
33
25
34
26
/* Max # of type identifier */
+56
-4
kernel/bpf/btf.c
+56
-4
kernel/bpf/btf.c
···
270
270
struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
271
271
struct btf_struct_metas *struct_meta_tab;
272
272
struct btf_struct_ops_tab *struct_ops_tab;
273
+
struct btf_layout *layout;
273
274
274
275
/* split BTF support */
275
276
struct btf *base_btf;
···
1708
1707
__btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
1709
1708
__btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
1710
1709
__btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
1710
+
if (hdr->hdr_len >= sizeof(struct btf_header) &&
1711
+
btf_data_size >= hdr->hdr_len) {
1712
+
__btf_verifier_log(log, "layout_off: %u\n", hdr->layout_off);
1713
+
__btf_verifier_log(log, "layout_len: %u\n", hdr->layout_len);
1714
+
}
1711
1715
__btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
1712
1716
}
1713
1717
···
5532
5526
start = btf->nohdr_data + hdr->str_off;
5533
5527
end = start + hdr->str_len;
5534
5528
5535
-
if (end != btf->data + btf->data_size) {
5529
+
if (hdr->hdr_len < sizeof(struct btf_header) &&
5530
+
end != btf->data + btf->data_size) {
5536
5531
btf_verifier_log(env, "String section is not at the end");
5537
5532
return -EINVAL;
5538
5533
}
···
5554
5547
return 0;
5555
5548
}
5556
5549
5550
+
static int btf_parse_layout_sec(struct btf_verifier_env *env)
5551
+
{
5552
+
const struct btf_header *hdr = &env->btf->hdr;
5553
+
struct btf *btf = env->btf;
5554
+
void *start, *end;
5555
+
5556
+
if (hdr->hdr_len < sizeof(struct btf_header) ||
5557
+
hdr->layout_len == 0)
5558
+
return 0;
5559
+
5560
+
/* Layout section must align to 4 bytes */
5561
+
if (hdr->layout_off & (sizeof(u32) - 1)) {
5562
+
btf_verifier_log(env, "Unaligned layout_off");
5563
+
return -EINVAL;
5564
+
}
5565
+
start = btf->nohdr_data + hdr->layout_off;
5566
+
end = start + hdr->layout_len;
5567
+
5568
+
if (hdr->layout_len < sizeof(struct btf_layout)) {
5569
+
btf_verifier_log(env, "Layout section is too small");
5570
+
return -EINVAL;
5571
+
}
5572
+
if (hdr->layout_len % sizeof(struct btf_layout) != 0) {
5573
+
btf_verifier_log(env, "layout_len is not multiple of %zu",
5574
+
sizeof(struct btf_layout));
5575
+
return -EINVAL;
5576
+
}
5577
+
if (end > btf->data + btf->data_size) {
5578
+
btf_verifier_log(env, "Layout section is too big");
5579
+
return -EINVAL;
5580
+
}
5581
+
btf->layout = start;
5582
+
5583
+
return 0;
5584
+
}
5585
+
5557
5586
static const size_t btf_sec_info_offset[] = {
5558
5587
offsetof(struct btf_header, type_off),
5559
5588
offsetof(struct btf_header, str_off),
5589
+
offsetof(struct btf_header, layout_off)
5560
5590
};
5561
5591
5562
5592
static int btf_sec_info_cmp(const void *a, const void *b)
···
5609
5565
{
5610
5566
struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
5611
5567
u32 total, expected_total, i;
5568
+
u32 nr_secs = ARRAY_SIZE(btf_sec_info_offset);
5612
5569
const struct btf_header *hdr;
5613
5570
const struct btf *btf;
5614
5571
5615
5572
btf = env->btf;
5616
5573
hdr = &btf->hdr;
5617
5574
5575
+
if (hdr->hdr_len < sizeof(struct btf_header) || hdr->layout_len == 0)
5576
+
nr_secs--;
5577
+
5618
5578
/* Populate the secs from hdr */
5619
-
for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++)
5579
+
for (i = 0; i < nr_secs; i++)
5620
5580
secs[i] = *(struct btf_sec_info *)((void *)hdr +
5621
5581
btf_sec_info_offset[i]);
5622
5582
5623
-
sort(secs, ARRAY_SIZE(btf_sec_info_offset),
5583
+
sort(secs, nr_secs,
5624
5584
sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
5625
5585
5626
5586
/* Check for gaps and overlap among sections */
5627
5587
total = 0;
5628
5588
expected_total = btf_data_size - hdr->hdr_len;
5629
-
for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++) {
5589
+
for (i = 0; i < nr_secs; i++) {
5630
5590
if (expected_total < secs[i].off) {
5631
5591
btf_verifier_log(env, "Invalid section offset");
5632
5592
return -EINVAL;
···
5983
5935
btf->nohdr_data = btf->data + btf->hdr.hdr_len;
5984
5936
5985
5937
err = btf_parse_str_sec(env);
5938
+
if (err)
5939
+
goto errout;
5940
+
5941
+
err = btf_parse_layout_sec(env);
5986
5942
if (err)
5987
5943
goto errout;
5988
5944
+2
scripts/Makefile.btf
+2
scripts/Makefile.btf
+12
tools/include/uapi/linux/btf.h
+12
tools/include/uapi/linux/btf.h
···
8
8
#define BTF_MAGIC 0xeB9F
9
9
#define BTF_VERSION 1
10
10
11
+
/*
12
+
* BTF layout section consists of a struct btf_layout for each known
13
+
* kind at BTF encoding time.
14
+
*/
15
+
struct btf_layout {
16
+
__u8 info_sz; /* size of singular element after btf_type */
17
+
__u8 elem_sz; /* size of each of btf_vlen(t) elements */
18
+
__u16 flags; /* currently unused */
19
+
};
20
+
11
21
struct btf_header {
12
22
__u16 magic;
13
23
__u8 version;
···
29
19
__u32 type_len; /* length of type section */
30
20
__u32 str_off; /* offset of string section */
31
21
__u32 str_len; /* length of string section */
22
+
__u32 layout_off; /* offset of layout section */
23
+
__u32 layout_len; /* length of layout section */
32
24
};
33
25
34
26
/* Max # of type identifier */
+431
-167
tools/lib/bpf/btf.c
+431
-167
tools/lib/bpf/btf.c
···
29
29
30
30
static struct btf_type btf_void;
31
31
32
+
/*
33
+
* Describe how kinds are laid out; some have a singular element following the "struct btf_type",
34
+
* some have BTF_INFO_VLEN(t->info) elements. Specify sizes for both. Flags are currently unused.
35
+
* Kind layout can be optionally added to the BTF representation in a dedicated section to
36
+
* facilitate parsing. New kinds must be added here.
37
+
*/
38
+
static struct btf_layout layouts[NR_BTF_KINDS] = {
39
+
/* singular element size vlen element(s) size flags */
40
+
[BTF_KIND_UNKN] = { 0, 0, 0 },
41
+
[BTF_KIND_INT] = { sizeof(__u32), 0, 0 },
42
+
[BTF_KIND_PTR] = { 0, 0, 0 },
43
+
[BTF_KIND_ARRAY] = { sizeof(struct btf_array), 0, 0 },
44
+
[BTF_KIND_STRUCT] = { 0, sizeof(struct btf_member), 0 },
45
+
[BTF_KIND_UNION] = { 0, sizeof(struct btf_member), 0 },
46
+
[BTF_KIND_ENUM] = { 0, sizeof(struct btf_enum), 0 },
47
+
[BTF_KIND_FWD] = { 0, 0, 0 },
48
+
[BTF_KIND_TYPEDEF] = { 0, 0, 0 },
49
+
[BTF_KIND_VOLATILE] = { 0, 0, 0 },
50
+
[BTF_KIND_CONST] = { 0, 0, 0 },
51
+
[BTF_KIND_RESTRICT] = { 0, 0, 0 },
52
+
[BTF_KIND_FUNC] = { 0, 0, 0 },
53
+
[BTF_KIND_FUNC_PROTO] = { 0, sizeof(struct btf_param), 0 },
54
+
[BTF_KIND_VAR] = { sizeof(struct btf_var), 0, 0 },
55
+
[BTF_KIND_DATASEC] = { 0, sizeof(struct btf_var_secinfo), 0 },
56
+
[BTF_KIND_FLOAT] = { 0, 0, 0 },
57
+
[BTF_KIND_DECL_TAG] = { sizeof(struct btf_decl_tag), 0, 0 },
58
+
[BTF_KIND_TYPE_TAG] = { 0, 0, 0 },
59
+
[BTF_KIND_ENUM64] = { 0, sizeof(struct btf_enum64), 0 },
60
+
};
61
+
32
62
struct btf {
33
63
/* raw BTF data in native endianness */
34
64
void *raw_data;
···
70
40
71
41
/*
72
42
* When BTF is loaded from an ELF or raw memory it is stored
73
-
* in a contiguous memory block. The hdr, type_data, and, strs_data
43
+
* in a contiguous memory block. The type_data, layout and strs_data
74
44
* point inside that memory region to their respective parts of BTF
75
45
* representation:
76
46
*
77
-
* +--------------------------------+
78
-
* | Header | Types | Strings |
79
-
* +--------------------------------+
80
-
* ^ ^ ^
81
-
* | | |
82
-
* hdr | |
83
-
* types_data-+ |
84
-
* strs_data------------+
47
+
* +----------------------------------------+---------------+
48
+
* | Header | Types | Optional layout | Strings |
49
+
* +--------------------------------------------------------+
50
+
* ^ ^ ^ ^
51
+
* | | | |
52
+
* raw_data | | |
53
+
* types_data-+ | |
54
+
* layout---------------+ |
55
+
* strs_data--------------------------------+
56
+
*
57
+
* A separate struct btf_header is embedded as btf->hdr,
58
+
* and header information is copied into it. This allows us
59
+
* to handle header data for various header formats; the original,
60
+
* the extended header with layout info, etc.
85
61
*
86
62
* If BTF data is later modified, e.g., due to types added or
87
63
* removed, BTF deduplication performed, etc, this contiguous
88
-
* representation is broken up into three independently allocated
89
-
* memory regions to be able to modify them independently.
64
+
* representation is broken up into four independent memory
65
+
* regions.
66
+
*
90
67
* raw_data is nulled out at that point, but can be later allocated
91
68
* and cached again if user calls btf__raw_data(), at which point
92
-
* raw_data will contain a contiguous copy of header, types, and
93
-
* strings:
69
+
* raw_data will contain a contiguous copy of header, types, optional
70
+
* layout and strings. layout optionally points to a
71
+
* btf_layout array - this allows us to encode information about
72
+
* the kinds known at encoding time. If layout is NULL no
73
+
* layout information is encoded.
94
74
*
95
-
* +----------+ +---------+ +-----------+
96
-
* | Header | | Types | | Strings |
97
-
* +----------+ +---------+ +-----------+
98
-
* ^ ^ ^
99
-
* | | |
100
-
* hdr | |
101
-
* types_data----+ |
102
-
* strset__data(strs_set)-----+
75
+
* +----------+ +---------+ +-----------+ +-----------+
76
+
* | Header | | Types | | Layout | | Strings |
77
+
* +----------+ +---------+ +-----------+ +-----------+
78
+
* ^ ^ ^ ^
79
+
* | | | |
80
+
* hdr | | |
81
+
* types_data----+ | |
82
+
* layout---------------------+ |
83
+
* strset__data(strs_set)---------------------+
103
84
*
104
-
* +----------+---------+-----------+
105
-
* | Header | Types | Strings |
106
-
* raw_data----->+----------+---------+-----------+
85
+
* +----------+---------+-------------------+-----------+
86
+
* | Header | Types | Optional Layout | Strings |
87
+
* raw_data----->+----------+---------+-------------------+-----------+
107
88
*/
108
-
struct btf_header *hdr;
89
+
struct btf_header hdr;
109
90
110
91
void *types_data;
111
92
size_t types_data_cap; /* used size stored in hdr->type_len */
···
165
124
166
125
/* whether raw_data is a (read-only) mmap */
167
126
bool raw_data_is_mmap;
127
+
128
+
/* is BTF modifiable? i.e. is it split into separate sections as described above? */
129
+
bool modifiable;
130
+
/* does BTF have header information we do not support? If so, disallow
131
+
* modification.
132
+
*/
133
+
bool has_hdr_extra;
134
+
/* Points either at raw kind layout data in parsed BTF (if present), or
135
+
* at an allocated kind layout array when BTF is modifiable.
136
+
*/
137
+
void *layout;
168
138
169
139
/* BTF object FD, if loaded into kernel */
170
140
int fd;
···
268
216
return 0;
269
217
}
270
218
271
-
static void btf_bswap_hdr(struct btf_header *h)
219
+
static void btf_bswap_hdr(struct btf_header *h, __u32 hdr_len)
272
220
{
273
221
h->magic = bswap_16(h->magic);
274
222
h->hdr_len = bswap_32(h->hdr_len);
···
276
224
h->type_len = bswap_32(h->type_len);
277
225
h->str_off = bswap_32(h->str_off);
278
226
h->str_len = bswap_32(h->str_len);
227
+
/* May be operating on raw data with hdr_len that does not include below fields */
228
+
if (hdr_len >= sizeof(struct btf_header)) {
229
+
h->layout_off = bswap_32(h->layout_off);
230
+
h->layout_len = bswap_32(h->layout_len);
231
+
}
279
232
}
280
233
281
234
static int btf_parse_hdr(struct btf *btf)
282
235
{
283
-
struct btf_header *hdr = btf->hdr;
284
-
__u32 meta_left;
236
+
struct btf_header *hdr = btf->raw_data;
237
+
__u32 hdr_len, meta_left;
285
238
286
-
if (btf->raw_size < sizeof(struct btf_header)) {
239
+
if (btf->raw_size < offsetofend(struct btf_header, str_len)) {
287
240
pr_debug("BTF header not found\n");
288
241
return -EINVAL;
289
242
}
290
243
244
+
hdr_len = hdr->hdr_len;
245
+
291
246
if (hdr->magic == bswap_16(BTF_MAGIC)) {
292
247
btf->swapped_endian = true;
293
-
if (bswap_32(hdr->hdr_len) != sizeof(struct btf_header)) {
248
+
hdr_len = bswap_32(hdr->hdr_len);
249
+
if (hdr_len < offsetofend(struct btf_header, str_len)) {
294
250
pr_warn("Can't load BTF with non-native endianness due to unsupported header length %u\n",
295
-
bswap_32(hdr->hdr_len));
251
+
hdr_len);
296
252
return -ENOTSUP;
297
253
}
298
-
btf_bswap_hdr(hdr);
299
254
} else if (hdr->magic != BTF_MAGIC) {
300
255
pr_debug("Invalid BTF magic: %x\n", hdr->magic);
301
256
return -EINVAL;
302
257
}
303
258
304
-
if (btf->raw_size < hdr->hdr_len) {
259
+
if (btf->raw_size < hdr_len) {
305
260
pr_debug("BTF header len %u larger than data size %u\n",
306
-
hdr->hdr_len, btf->raw_size);
261
+
hdr_len, btf->raw_size);
307
262
return -EINVAL;
308
263
}
309
264
310
-
meta_left = btf->raw_size - hdr->hdr_len;
311
-
if (meta_left < (long long)hdr->str_off + hdr->str_len) {
265
+
if (btf->swapped_endian)
266
+
btf_bswap_hdr(hdr, hdr_len);
267
+
268
+
memcpy(&btf->hdr, hdr, min((size_t)hdr_len, sizeof(struct btf_header)));
269
+
270
+
/* If unknown header data is found, modification is prohibited in
271
+
* btf_ensure_modifiable().
272
+
*/
273
+
if (hdr_len > sizeof(struct btf_header)) {
274
+
__u8 *h = (__u8 *)hdr;
275
+
__u32 i;
276
+
277
+
for (i = sizeof(struct btf_header); i < hdr_len; i++) {
278
+
if (!h[i])
279
+
continue;
280
+
btf->has_hdr_extra = true;
281
+
pr_debug("Unknown BTF header data at offset %u; modification is disallowed\n",
282
+
i);
283
+
break;
284
+
}
285
+
}
286
+
287
+
meta_left = btf->raw_size - hdr_len;
288
+
if (meta_left < (long long)btf->hdr.str_off + btf->hdr.str_len) {
312
289
pr_debug("Invalid BTF total size: %u\n", btf->raw_size);
313
290
return -EINVAL;
314
291
}
315
292
316
-
if ((long long)hdr->type_off + hdr->type_len > hdr->str_off) {
293
+
if ((long long)btf->hdr.type_off + btf->hdr.type_len > btf->hdr.str_off) {
317
294
pr_debug("Invalid BTF data sections layout: type data at %u + %u, strings data at %u + %u\n",
318
-
hdr->type_off, hdr->type_len, hdr->str_off, hdr->str_len);
295
+
btf->hdr.type_off, btf->hdr.type_len, btf->hdr.str_off,
296
+
btf->hdr.str_len);
319
297
return -EINVAL;
320
298
}
321
299
322
-
if (hdr->type_off % 4) {
300
+
if (btf->hdr.type_off % 4) {
323
301
pr_debug("BTF type section is not aligned to 4 bytes\n");
324
302
return -EINVAL;
325
303
}
326
304
305
+
if (btf->hdr.layout_len == 0)
306
+
return 0;
307
+
308
+
/* optional layout section sits between types and strings */
309
+
if (btf->hdr.layout_off % 4) {
310
+
pr_debug("BTF layout section is not aligned to 4 bytes\n");
311
+
return -EINVAL;
312
+
}
313
+
if (btf->hdr.layout_off < (long long)btf->hdr.type_off + btf->hdr.type_len) {
314
+
pr_debug("Invalid BTF data sections layout: type data at %u + %u, layout data at %u + %u\n",
315
+
btf->hdr.type_off, btf->hdr.type_len,
316
+
btf->hdr.layout_off, btf->hdr.layout_len);
317
+
return -EINVAL;
318
+
}
319
+
if ((long long)btf->hdr.layout_off + btf->hdr.layout_len > btf->hdr.str_off ||
320
+
btf->hdr.layout_off > btf->hdr.str_off) {
321
+
pr_debug("Invalid BTF data sections layout: layout data at %u + %u, strings data at %u\n",
322
+
btf->hdr.layout_off, btf->hdr.layout_len, btf->hdr.str_off);
323
+
return -EINVAL;
324
+
}
327
325
return 0;
328
326
}
329
327
330
328
static int btf_parse_str_sec(struct btf *btf)
331
329
{
332
-
const struct btf_header *hdr = btf->hdr;
333
330
const char *start = btf->strs_data;
334
-
const char *end = start + btf->hdr->str_len;
331
+
const char *end = start + btf->hdr.str_len;
335
332
336
-
if (btf->base_btf && hdr->str_len == 0)
333
+
if (btf->base_btf && btf->hdr.str_len == 0)
337
334
return 0;
338
-
if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_STR_OFFSET || end[-1]) {
335
+
if (!btf->hdr.str_len || btf->hdr.str_len - 1 > BTF_MAX_STR_OFFSET || end[-1]) {
339
336
pr_debug("Invalid BTF string section\n");
340
337
return -EINVAL;
341
338
}
···
395
294
return 0;
396
295
}
397
296
398
-
static int btf_type_size(const struct btf_type *t)
297
+
static int btf_parse_layout_sec(struct btf *btf)
298
+
{
299
+
if (!btf->hdr.layout_len)
300
+
return 0;
301
+
302
+
if (btf->hdr.layout_len % sizeof(struct btf_layout) != 0) {
303
+
pr_debug("Invalid BTF kind layout section\n");
304
+
return -EINVAL;
305
+
}
306
+
btf->layout = btf->raw_data + btf->hdr.hdr_len + btf->hdr.layout_off;
307
+
308
+
if (btf->swapped_endian) {
309
+
struct btf_layout *l, *end = btf->layout + btf->hdr.layout_len;
310
+
311
+
for (l = btf->layout; l < end; l++)
312
+
l->flags = bswap_16(l->flags);
313
+
}
314
+
315
+
return 0;
316
+
}
317
+
318
+
/* for unknown kinds, consult kind layout. */
319
+
static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t)
320
+
{
321
+
__u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout);
322
+
struct btf_layout *l = btf->layout;
323
+
__u16 vlen = btf_vlen(t);
324
+
__u32 kind = btf_kind(t);
325
+
326
+
/* Fall back to base BTF if needed as they share layout information */
327
+
if (!l) {
328
+
struct btf *base_btf = btf->base_btf;
329
+
330
+
if (base_btf) {
331
+
l = base_btf->layout;
332
+
l_cnt = base_btf->hdr.layout_len / sizeof(struct btf_layout);
333
+
}
334
+
}
335
+
if (!l || kind >= l_cnt) {
336
+
pr_debug("Unsupported BTF_KIND: %u\n", btf_kind(t));
337
+
return -EINVAL;
338
+
}
339
+
if (l[kind].info_sz % 4) {
340
+
pr_debug("Unsupported info_sz %u for kind %u\n",
341
+
l[kind].info_sz, kind);
342
+
return -EINVAL;
343
+
}
344
+
if (l[kind].elem_sz % 4) {
345
+
pr_debug("Unsupported elem_sz %u for kind %u\n",
346
+
l[kind].elem_sz, kind);
347
+
return -EINVAL;
348
+
}
349
+
350
+
return sizeof(struct btf_type) + l[kind].info_sz + vlen * l[kind].elem_sz;
351
+
}
352
+
353
+
static int btf_type_size(const struct btf *btf, const struct btf_type *t)
399
354
{
400
355
const int base_size = sizeof(struct btf_type);
401
356
__u16 vlen = btf_vlen(t);
···
487
330
case BTF_KIND_DECL_TAG:
488
331
return base_size + sizeof(struct btf_decl_tag);
489
332
default:
490
-
pr_debug("Unsupported BTF_KIND:%u\n", btf_kind(t));
491
-
return -EINVAL;
333
+
return btf_type_size_unknown(btf, t);
492
334
}
493
335
}
494
336
···
577
421
578
422
static int btf_parse_type_sec(struct btf *btf)
579
423
{
580
-
struct btf_header *hdr = btf->hdr;
581
424
void *next_type = btf->types_data;
582
-
void *end_type = next_type + hdr->type_len;
425
+
void *end_type = next_type + btf->hdr.type_len;
583
426
int err, type_size;
584
427
585
428
while (next_type + sizeof(struct btf_type) <= end_type) {
586
429
if (btf->swapped_endian)
587
430
btf_bswap_type_base(next_type);
588
431
589
-
type_size = btf_type_size(next_type);
432
+
type_size = btf_type_size(btf, next_type);
590
433
if (type_size < 0)
591
434
return type_size;
592
435
if (next_type + type_size > end_type) {
···
746
591
break;
747
592
}
748
593
default:
749
-
pr_warn("btf: type [%u]: unrecognized kind %u\n", id, kind);
750
-
return -EINVAL;
594
+
/* Kind may be represented in kind layout information. */
595
+
if (btf_type_size_unknown(btf, t) < 0) {
596
+
pr_warn("btf: type [%u]: unrecognized kind %u\n", id, kind);
597
+
return -EINVAL;
598
+
}
599
+
break;
751
600
}
752
601
return 0;
753
602
}
···
1171
1012
1172
1013
static bool btf_is_modifiable(const struct btf *btf)
1173
1014
{
1174
-
return (void *)btf->hdr != btf->raw_data;
1015
+
/* BTF is modifiable if split into multiple sections */
1016
+
return btf->modifiable;
1175
1017
}
1176
1018
1177
1019
static void btf_free_raw_data(struct btf *btf)
···
1196
1036
1197
1037
if (btf_is_modifiable(btf)) {
1198
1038
/* if BTF was modified after loading, it will have a split
1199
-
* in-memory representation for header, types, and strings
1039
+
* in-memory representation for types, strings and layout
1200
1040
* sections, so we need to free all of them individually. It
1201
1041
* might still have a cached contiguous raw data present,
1202
1042
* which will be unconditionally freed below.
1203
1043
*/
1204
-
free(btf->hdr);
1205
1044
free(btf->types_data);
1206
1045
strset__free(btf->strs_set);
1046
+
free(btf->layout);
1207
1047
}
1208
1048
btf_free_raw_data(btf);
1209
1049
free(btf->raw_data_swapped);
···
1213
1053
free(btf);
1214
1054
}
1215
1055
1216
-
static struct btf *btf_new_empty(struct btf *base_btf)
1056
+
static struct btf *btf_new_empty(struct btf_new_opts *opts)
1217
1057
{
1058
+
bool add_layout = OPTS_GET(opts, add_layout, false);
1059
+
struct btf *base_btf = OPTS_GET(opts, base_btf, NULL);
1060
+
struct btf_header *hdr;
1218
1061
struct btf *btf;
1219
1062
1220
1063
btf = calloc(1, sizeof(*btf));
···
1235
1072
if (base_btf) {
1236
1073
btf->base_btf = base_btf;
1237
1074
btf->start_id = btf__type_cnt(base_btf);
1238
-
btf->start_str_off = base_btf->hdr->str_len + base_btf->start_str_off;
1075
+
btf->start_str_off = base_btf->hdr.str_len + base_btf->start_str_off;
1239
1076
btf->swapped_endian = base_btf->swapped_endian;
1240
1077
}
1241
1078
1242
1079
/* +1 for empty string at offset 0 */
1243
1080
btf->raw_size = sizeof(struct btf_header) + (base_btf ? 0 : 1);
1081
+
if (add_layout)
1082
+
btf->raw_size += sizeof(layouts);
1244
1083
btf->raw_data = calloc(1, btf->raw_size);
1245
1084
if (!btf->raw_data) {
1246
1085
free(btf);
1247
1086
return ERR_PTR(-ENOMEM);
1248
1087
}
1249
1088
1250
-
btf->hdr = btf->raw_data;
1251
-
btf->hdr->hdr_len = sizeof(struct btf_header);
1252
-
btf->hdr->magic = BTF_MAGIC;
1253
-
btf->hdr->version = BTF_VERSION;
1089
+
hdr = btf->raw_data;
1090
+
hdr->hdr_len = sizeof(struct btf_header);
1091
+
hdr->magic = BTF_MAGIC;
1092
+
hdr->version = BTF_VERSION;
1254
1093
1255
-
btf->types_data = btf->raw_data + btf->hdr->hdr_len;
1256
-
btf->strs_data = btf->raw_data + btf->hdr->hdr_len;
1257
-
btf->hdr->str_len = base_btf ? 0 : 1; /* empty string at offset 0 */
1094
+
btf->types_data = btf->raw_data + hdr->hdr_len;
1095
+
btf->strs_data = btf->raw_data + hdr->hdr_len;
1096
+
hdr->str_len = base_btf ? 0 : 1; /* empty string at offset 0 */
1097
+
1098
+
if (add_layout) {
1099
+
hdr->layout_len = sizeof(layouts);
1100
+
btf->layout = layouts;
1101
+
/*
1102
+
* No need to swap endianness here as btf_get_raw_data()
1103
+
* will do this for us if btf->swapped_endian is true.
1104
+
*/
1105
+
memcpy(btf->raw_data + hdr->hdr_len, layouts, sizeof(layouts));
1106
+
btf->strs_data += sizeof(layouts);
1107
+
hdr->str_off += sizeof(layouts);
1108
+
}
1109
+
1110
+
memcpy(&btf->hdr, hdr, sizeof(*hdr));
1258
1111
1259
1112
return btf;
1260
1113
}
···
1282
1103
1283
1104
struct btf *btf__new_empty_split(struct btf *base_btf)
1284
1105
{
1285
-
return libbpf_ptr(btf_new_empty(base_btf));
1106
+
LIBBPF_OPTS(btf_new_opts, opts);
1107
+
1108
+
opts.base_btf = base_btf;
1109
+
1110
+
return libbpf_ptr(btf_new_empty(&opts));
1111
+
}
1112
+
1113
+
struct btf *btf__new_empty_opts(struct btf_new_opts *opts)
1114
+
{
1115
+
if (!OPTS_VALID(opts, btf_new_opts))
1116
+
return libbpf_err_ptr(-EINVAL);
1117
+
1118
+
return libbpf_ptr(btf_new_empty(opts));
1286
1119
}
1287
1120
1288
1121
static struct btf *btf_new(const void *data, __u32 size, struct btf *base_btf, bool is_mmap)
···
1315
1124
if (base_btf) {
1316
1125
btf->base_btf = base_btf;
1317
1126
btf->start_id = btf__type_cnt(base_btf);
1318
-
btf->start_str_off = base_btf->hdr->str_len + base_btf->start_str_off;
1127
+
btf->start_str_off = base_btf->hdr.str_len + base_btf->start_str_off;
1319
1128
}
1320
1129
1321
1130
if (is_mmap) {
···
1332
1141
1333
1142
btf->raw_size = size;
1334
1143
1335
-
btf->hdr = btf->raw_data;
1336
1144
err = btf_parse_hdr(btf);
1337
1145
if (err)
1338
1146
goto done;
1339
1147
1340
-
btf->strs_data = btf->raw_data + btf->hdr->hdr_len + btf->hdr->str_off;
1341
-
btf->types_data = btf->raw_data + btf->hdr->hdr_len + btf->hdr->type_off;
1148
+
btf->strs_data = btf->raw_data + btf->hdr.hdr_len + btf->hdr.str_off;
1149
+
btf->types_data = btf->raw_data + btf->hdr.hdr_len + btf->hdr.type_off;
1342
1150
1343
1151
err = btf_parse_str_sec(btf);
1152
+
err = err ?: btf_parse_layout_sec(btf);
1344
1153
err = err ?: btf_parse_type_sec(btf);
1345
1154
err = err ?: btf_sanity_check(btf);
1346
1155
if (err)
···
1792
1601
1793
1602
static void *btf_get_raw_data(const struct btf *btf, __u32 *size, bool swap_endian)
1794
1603
{
1795
-
struct btf_header *hdr = btf->hdr;
1604
+
const struct btf_header *hdr = &btf->hdr;
1796
1605
struct btf_type *t;
1797
1606
void *data, *p;
1798
1607
__u32 data_sz;
···
1805
1614
}
1806
1615
1807
1616
data_sz = hdr->hdr_len + hdr->type_len + hdr->str_len;
1617
+
if (btf->layout)
1618
+
data_sz += hdr->layout_len;
1619
+
1808
1620
data = calloc(1, data_sz);
1809
1621
if (!data)
1810
1622
return NULL;
1811
1623
p = data;
1812
1624
1813
-
memcpy(p, hdr, hdr->hdr_len);
1625
+
memcpy(p, hdr, min((__u32)sizeof(struct btf_header), hdr->hdr_len));
1814
1626
if (swap_endian)
1815
-
btf_bswap_hdr(p);
1627
+
btf_bswap_hdr(p, hdr->hdr_len);
1816
1628
p += hdr->hdr_len;
1817
1629
1818
1630
memcpy(p, btf->types_data, hdr->type_len);
···
1833
1639
}
1834
1640
p += hdr->type_len;
1835
1641
1642
+
if (btf->layout) {
1643
+
memcpy(p, btf->layout, hdr->layout_len);
1644
+
if (swap_endian) {
1645
+
struct btf_layout *l, *end = p + hdr->layout_len;
1646
+
1647
+
for (l = p; l < end ; l++)
1648
+
l->flags = bswap_16(l->flags);
1649
+
}
1650
+
p += hdr->layout_len;
1651
+
}
1652
+
1836
1653
memcpy(p, btf_strs_data(btf), hdr->str_len);
1837
-
p += hdr->str_len;
1838
1654
1839
1655
*size = data_sz;
1840
1656
return data;
···
1879
1675
{
1880
1676
if (offset < btf->start_str_off)
1881
1677
return btf__str_by_offset(btf->base_btf, offset);
1882
-
else if (offset - btf->start_str_off < btf->hdr->str_len)
1678
+
else if (offset - btf->start_str_off < btf->hdr.str_len)
1883
1679
return btf_strs_data(btf) + (offset - btf->start_str_off);
1884
1680
else
1885
1681
return errno = EINVAL, NULL;
···
1987
1783
}
1988
1784
1989
1785
/* Ensure BTF is ready to be modified (by splitting into a three memory
1990
-
* regions for header, types, and strings). Also invalidate cached
1786
+
* regions for types, strings and layout. Also invalidate cached
1991
1787
* raw_data, if any.
1992
1788
*/
1993
1789
static int btf_ensure_modifiable(struct btf *btf)
1994
1790
{
1995
-
void *hdr, *types;
1791
+
void *types, *layout = NULL;
1996
1792
struct strset *set = NULL;
1997
1793
int err = -ENOMEM;
1998
1794
···
2002
1798
return 0;
2003
1799
}
2004
1800
2005
-
/* split raw data into three memory regions */
2006
-
hdr = malloc(btf->hdr->hdr_len);
2007
-
types = malloc(btf->hdr->type_len);
2008
-
if (!hdr || !types)
2009
-
goto err_out;
1801
+
if (btf->has_hdr_extra) {
1802
+
/* Additional BTF header data was found; not safe to modify. */
1803
+
return -EOPNOTSUPP;
1804
+
}
2010
1805
2011
-
memcpy(hdr, btf->hdr, btf->hdr->hdr_len);
2012
-
memcpy(types, btf->types_data, btf->hdr->type_len);
1806
+
/* split raw data into memory regions; btf->hdr is done already. */
1807
+
types = malloc(btf->hdr.type_len);
1808
+
if (!types)
1809
+
goto err_out;
1810
+
memcpy(types, btf->types_data, btf->hdr.type_len);
1811
+
1812
+
if (btf->hdr.layout_len) {
1813
+
layout = malloc(btf->hdr.layout_len);
1814
+
if (!layout)
1815
+
goto err_out;
1816
+
memcpy(layout, btf->raw_data + btf->hdr.hdr_len + btf->hdr.layout_off,
1817
+
btf->hdr.layout_len);
1818
+
}
2013
1819
2014
1820
/* build lookup index for all strings */
2015
-
set = strset__new(BTF_MAX_STR_OFFSET, btf->strs_data, btf->hdr->str_len);
1821
+
set = strset__new(BTF_MAX_STR_OFFSET, btf->strs_data, btf->hdr.str_len);
2016
1822
if (IS_ERR(set)) {
2017
1823
err = PTR_ERR(set);
2018
1824
goto err_out;
2019
1825
}
2020
1826
2021
1827
/* only when everything was successful, update internal state */
2022
-
btf->hdr = hdr;
2023
1828
btf->types_data = types;
2024
-
btf->types_data_cap = btf->hdr->type_len;
1829
+
btf->types_data_cap = btf->hdr.type_len;
2025
1830
btf->strs_data = NULL;
2026
1831
btf->strs_set = set;
1832
+
if (layout)
1833
+
btf->layout = layout;
2027
1834
/* if BTF was created from scratch, all strings are guaranteed to be
2028
1835
* unique and deduplicated
2029
1836
*/
2030
-
if (btf->hdr->str_len == 0)
1837
+
if (btf->hdr.str_len == 0)
2031
1838
btf->strs_deduped = true;
2032
-
if (!btf->base_btf && btf->hdr->str_len == 1)
1839
+
if (!btf->base_btf && btf->hdr.str_len == 1)
2033
1840
btf->strs_deduped = true;
2034
1841
2035
1842
/* invalidate raw_data representation */
2036
1843
btf_invalidate_raw_data(btf);
2037
1844
1845
+
btf->modifiable = true;
1846
+
2038
1847
return 0;
2039
1848
2040
1849
err_out:
2041
1850
strset__free(set);
2042
-
free(hdr);
2043
1851
free(types);
1852
+
free(layout);
2044
1853
return err;
2045
1854
}
2046
1855
···
2066
1849
int btf__find_str(struct btf *btf, const char *s)
2067
1850
{
2068
1851
int off;
1852
+
int err;
2069
1853
2070
1854
if (btf->base_btf) {
2071
1855
off = btf__find_str(btf->base_btf, s);
···
2075
1857
}
2076
1858
2077
1859
/* BTF needs to be in a modifiable state to build string lookup index */
2078
-
if (btf_ensure_modifiable(btf))
2079
-
return libbpf_err(-ENOMEM);
1860
+
err = btf_ensure_modifiable(btf);
1861
+
if (err)
1862
+
return libbpf_err(err);
2080
1863
2081
1864
off = strset__find_str(btf->strs_set, s);
2082
1865
if (off < 0)
···
2094
1875
int btf__add_str(struct btf *btf, const char *s)
2095
1876
{
2096
1877
int off;
1878
+
int err;
2097
1879
2098
1880
if (btf->base_btf) {
2099
1881
off = btf__find_str(btf->base_btf, s);
···
2102
1882
return off;
2103
1883
}
2104
1884
2105
-
if (btf_ensure_modifiable(btf))
2106
-
return libbpf_err(-ENOMEM);
1885
+
err = btf_ensure_modifiable(btf);
1886
+
if (err)
1887
+
return libbpf_err(err);
2107
1888
2108
1889
off = strset__add_str(btf->strs_set, s);
2109
1890
if (off < 0)
2110
1891
return libbpf_err(off);
2111
1892
2112
-
btf->hdr->str_len = strset__data_size(btf->strs_set);
1893
+
btf->hdr.str_len = strset__data_size(btf->strs_set);
2113
1894
2114
1895
return btf->start_str_off + off;
2115
1896
}
···
2118
1897
static void *btf_add_type_mem(struct btf *btf, size_t add_sz)
2119
1898
{
2120
1899
return libbpf_add_mem(&btf->types_data, &btf->types_data_cap, 1,
2121
-
btf->hdr->type_len, UINT_MAX, add_sz);
1900
+
btf->hdr.type_len, UINT_MAX, add_sz);
2122
1901
}
2123
1902
2124
1903
static void btf_type_inc_vlen(struct btf_type *t)
···
2126
1905
t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));
2127
1906
}
2128
1907
1908
+
static void btf_hdr_update_type_len(struct btf *btf, int new_len)
1909
+
{
1910
+
btf->hdr.type_len = new_len;
1911
+
if (btf->layout) {
1912
+
btf->hdr.layout_off = btf->hdr.type_off + new_len;
1913
+
btf->hdr.str_off = btf->hdr.layout_off + btf->hdr.layout_len;
1914
+
} else {
1915
+
btf->hdr.str_off = btf->hdr.type_off + new_len;
1916
+
}
1917
+
}
1918
+
1919
+
static void btf_hdr_update_str_len(struct btf *btf, int new_len)
1920
+
{
1921
+
btf->hdr.str_len = new_len;
1922
+
}
1923
+
2129
1924
static int btf_commit_type(struct btf *btf, int data_sz)
2130
1925
{
2131
1926
int err;
2132
1927
2133
-
err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
1928
+
err = btf_add_type_idx_entry(btf, btf->hdr.type_len);
2134
1929
if (err)
2135
1930
return libbpf_err(err);
2136
1931
2137
-
btf->hdr->type_len += data_sz;
2138
-
btf->hdr->str_off += data_sz;
1932
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + data_sz);
2139
1933
btf->nr_types++;
2140
1934
return btf->start_id + btf->nr_types - 1;
2141
1935
}
···
2199
1963
__u32 *str_off;
2200
1964
int sz, err;
2201
1965
2202
-
sz = btf_type_size(src_type);
1966
+
sz = btf_type_size(p->src, src_type);
2203
1967
if (sz < 0)
2204
1968
return libbpf_err(sz);
2205
1969
2206
1970
/* deconstruct BTF, if necessary, and invalidate raw_data */
2207
-
if (btf_ensure_modifiable(p->dst))
2208
-
return libbpf_err(-ENOMEM);
1971
+
err = btf_ensure_modifiable(p->dst);
1972
+
if (err)
1973
+
return libbpf_err(err);
2209
1974
2210
1975
t = btf_add_type_mem(p->dst, sz);
2211
1976
if (!t)
···
2255
2018
src_start_id = src_btf->base_btf ? btf__type_cnt(src_btf->base_btf) : 1;
2256
2019
2257
2020
/* deconstruct BTF, if necessary, and invalidate raw_data */
2258
-
if (btf_ensure_modifiable(btf))
2259
-
return libbpf_err(-ENOMEM);
2021
+
err = btf_ensure_modifiable(btf);
2022
+
if (err)
2023
+
return libbpf_err(err);
2260
2024
2261
2025
/* remember original strings section size if we have to roll back
2262
2026
* partial strings section changes
2263
2027
*/
2264
-
old_strs_len = btf->hdr->str_len;
2028
+
old_strs_len = btf->hdr.str_len;
2265
2029
2266
-
data_sz = src_btf->hdr->type_len;
2030
+
data_sz = src_btf->hdr.type_len;
2267
2031
cnt = src_btf->nr_types;
2268
2032
2269
2033
/* pre-allocate enough memory for new types */
···
2289
2051
struct btf_field_iter it;
2290
2052
__u32 *type_id, *str_off;
2291
2053
2292
-
sz = btf_type_size(t);
2054
+
sz = btf_type_size(src_btf, t);
2293
2055
if (sz < 0) {
2294
2056
/* unlikely, has to be corrupted src_btf */
2295
2057
err = sz;
···
2341
2103
* update type count and various internal offsets and sizes to
2342
2104
* "commit" the changes and made them visible to the outside world.
2343
2105
*/
2344
-
btf->hdr->type_len += data_sz;
2345
-
btf->hdr->str_off += data_sz;
2106
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + data_sz);
2346
2107
btf->nr_types += cnt;
2347
2108
2348
2109
hashmap__free(p.str_off_map);
···
2352
2115
/* zero out preallocated memory as if it was just allocated with
2353
2116
* libbpf_add_mem()
2354
2117
*/
2355
-
memset(btf->types_data + btf->hdr->type_len, 0, data_sz);
2356
-
memset(btf->strs_data + old_strs_len, 0, btf->hdr->str_len - old_strs_len);
2118
+
memset(btf->types_data + btf->hdr.type_len, 0, data_sz);
2119
+
if (btf->strs_data)
2120
+
memset(btf->strs_data + old_strs_len, 0, btf->hdr.str_len - old_strs_len);
2357
2121
2358
2122
/* and now restore original strings section size; types data size
2359
2123
* wasn't modified, so doesn't need restoring, see big comment above
2360
2124
*/
2361
-
btf->hdr->str_len = old_strs_len;
2125
+
btf_hdr_update_str_len(btf, old_strs_len);
2362
2126
2363
2127
hashmap__free(p.str_off_map);
2364
2128
···
2379
2141
{
2380
2142
struct btf_type *t;
2381
2143
int sz, name_off;
2144
+
int err;
2382
2145
2383
2146
/* non-empty name */
2384
2147
if (str_is_empty(name))
···
2391
2152
return libbpf_err(-EINVAL);
2392
2153
2393
2154
/* deconstruct BTF, if necessary, and invalidate raw_data */
2394
-
if (btf_ensure_modifiable(btf))
2395
-
return libbpf_err(-ENOMEM);
2155
+
err = btf_ensure_modifiable(btf);
2156
+
if (err)
2157
+
return libbpf_err(err);
2396
2158
2397
2159
sz = sizeof(struct btf_type) + sizeof(int);
2398
2160
t = btf_add_type_mem(btf, sz);
···
2429
2189
{
2430
2190
struct btf_type *t;
2431
2191
int sz, name_off;
2192
+
int err;
2432
2193
2433
2194
/* non-empty name */
2434
2195
if (str_is_empty(name))
···
2440
2199
byte_sz != 16)
2441
2200
return libbpf_err(-EINVAL);
2442
2201
2443
-
if (btf_ensure_modifiable(btf))
2444
-
return libbpf_err(-ENOMEM);
2202
+
err = btf_ensure_modifiable(btf);
2203
+
if (err)
2204
+
return libbpf_err(err);
2445
2205
2446
2206
sz = sizeof(struct btf_type);
2447
2207
t = btf_add_type_mem(btf, sz);
···
2476
2234
{
2477
2235
struct btf_type *t;
2478
2236
int sz, name_off = 0;
2237
+
int err;
2479
2238
2480
2239
if (validate_type_id(ref_type_id))
2481
2240
return libbpf_err(-EINVAL);
2482
2241
2483
-
if (btf_ensure_modifiable(btf))
2484
-
return libbpf_err(-ENOMEM);
2242
+
err = btf_ensure_modifiable(btf);
2243
+
if (err)
2244
+
return libbpf_err(err);
2485
2245
2486
2246
sz = sizeof(struct btf_type);
2487
2247
t = btf_add_type_mem(btf, sz);
···
2528
2284
{
2529
2285
struct btf_type *t;
2530
2286
struct btf_array *a;
2287
+
int err;
2531
2288
int sz;
2532
2289
2533
2290
if (validate_type_id(index_type_id) || validate_type_id(elem_type_id))
2534
2291
return libbpf_err(-EINVAL);
2535
2292
2536
-
if (btf_ensure_modifiable(btf))
2537
-
return libbpf_err(-ENOMEM);
2293
+
err = btf_ensure_modifiable(btf);
2294
+
if (err)
2295
+
return libbpf_err(err);
2538
2296
2539
2297
sz = sizeof(struct btf_type) + sizeof(struct btf_array);
2540
2298
t = btf_add_type_mem(btf, sz);
···
2560
2314
{
2561
2315
struct btf_type *t;
2562
2316
int sz, name_off = 0;
2317
+
int err;
2563
2318
2564
-
if (btf_ensure_modifiable(btf))
2565
-
return libbpf_err(-ENOMEM);
2319
+
err = btf_ensure_modifiable(btf);
2320
+
if (err)
2321
+
return libbpf_err(err);
2566
2322
2567
2323
sz = sizeof(struct btf_type);
2568
2324
t = btf_add_type_mem(btf, sz);
···
2644
2396
struct btf_member *m;
2645
2397
bool is_bitfield;
2646
2398
int sz, name_off = 0;
2399
+
int err;
2647
2400
2648
2401
/* last type should be union/struct */
2649
2402
if (btf->nr_types == 0)
···
2665
2416
return libbpf_err(-EINVAL);
2666
2417
2667
2418
/* decompose and invalidate raw data */
2668
-
if (btf_ensure_modifiable(btf))
2669
-
return libbpf_err(-ENOMEM);
2419
+
err = btf_ensure_modifiable(btf);
2420
+
if (err)
2421
+
return libbpf_err(err);
2670
2422
2671
2423
sz = sizeof(struct btf_member);
2672
2424
m = btf_add_type_mem(btf, sz);
···
2689
2439
/* update parent type's vlen and kflag */
2690
2440
t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));
2691
2441
2692
-
btf->hdr->type_len += sz;
2693
-
btf->hdr->str_off += sz;
2442
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
2694
2443
return 0;
2695
2444
}
2696
2445
···
2698
2449
{
2699
2450
struct btf_type *t;
2700
2451
int sz, name_off = 0;
2452
+
int err;
2701
2453
2702
2454
/* byte_sz must be power of 2 */
2703
2455
if (!byte_sz || (byte_sz & (byte_sz - 1)) || byte_sz > 8)
2704
2456
return libbpf_err(-EINVAL);
2705
2457
2706
-
if (btf_ensure_modifiable(btf))
2707
-
return libbpf_err(-ENOMEM);
2458
+
err = btf_ensure_modifiable(btf);
2459
+
if (err)
2460
+
return libbpf_err(err);
2708
2461
2709
2462
sz = sizeof(struct btf_type);
2710
2463
t = btf_add_type_mem(btf, sz);
···
2762
2511
struct btf_type *t;
2763
2512
struct btf_enum *v;
2764
2513
int sz, name_off;
2514
+
int err;
2765
2515
2766
2516
/* last type should be BTF_KIND_ENUM */
2767
2517
if (btf->nr_types == 0)
···
2778
2526
return libbpf_err(-E2BIG);
2779
2527
2780
2528
/* decompose and invalidate raw data */
2781
-
if (btf_ensure_modifiable(btf))
2782
-
return libbpf_err(-ENOMEM);
2529
+
err = btf_ensure_modifiable(btf);
2530
+
if (err)
2531
+
return libbpf_err(err);
2783
2532
2784
2533
sz = sizeof(struct btf_enum);
2785
2534
v = btf_add_type_mem(btf, sz);
···
2802
2549
if (value < 0)
2803
2550
t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
2804
2551
2805
-
btf->hdr->type_len += sz;
2806
-
btf->hdr->str_off += sz;
2552
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
2807
2553
return 0;
2808
2554
}
2809
2555
···
2840
2588
struct btf_enum64 *v;
2841
2589
struct btf_type *t;
2842
2590
int sz, name_off;
2591
+
int err;
2843
2592
2844
2593
/* last type should be BTF_KIND_ENUM64 */
2845
2594
if (btf->nr_types == 0)
···
2854
2601
return libbpf_err(-EINVAL);
2855
2602
2856
2603
/* decompose and invalidate raw data */
2857
-
if (btf_ensure_modifiable(btf))
2858
-
return libbpf_err(-ENOMEM);
2604
+
err = btf_ensure_modifiable(btf);
2605
+
if (err)
2606
+
return libbpf_err(err);
2859
2607
2860
2608
sz = sizeof(struct btf_enum64);
2861
2609
v = btf_add_type_mem(btf, sz);
···
2875
2621
t = btf_last_type(btf);
2876
2622
btf_type_inc_vlen(t);
2877
2623
2878
-
btf->hdr->type_len += sz;
2879
-
btf->hdr->str_off += sz;
2624
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
2880
2625
return 0;
2881
2626
}
2882
2627
···
3044
2791
int btf__add_func_proto(struct btf *btf, int ret_type_id)
3045
2792
{
3046
2793
struct btf_type *t;
2794
+
int err;
3047
2795
int sz;
3048
2796
3049
2797
if (validate_type_id(ret_type_id))
3050
2798
return libbpf_err(-EINVAL);
3051
2799
3052
-
if (btf_ensure_modifiable(btf))
3053
-
return libbpf_err(-ENOMEM);
2800
+
err = btf_ensure_modifiable(btf);
2801
+
if (err)
2802
+
return libbpf_err(err);
3054
2803
3055
2804
sz = sizeof(struct btf_type);
3056
2805
t = btf_add_type_mem(btf, sz);
···
3082
2827
struct btf_type *t;
3083
2828
struct btf_param *p;
3084
2829
int sz, name_off = 0;
2830
+
int err;
3085
2831
3086
2832
if (validate_type_id(type_id))
3087
2833
return libbpf_err(-EINVAL);
···
3095
2839
return libbpf_err(-EINVAL);
3096
2840
3097
2841
/* decompose and invalidate raw data */
3098
-
if (btf_ensure_modifiable(btf))
3099
-
return libbpf_err(-ENOMEM);
2842
+
err = btf_ensure_modifiable(btf);
2843
+
if (err)
2844
+
return libbpf_err(err);
3100
2845
3101
2846
sz = sizeof(struct btf_param);
3102
2847
p = btf_add_type_mem(btf, sz);
···
3117
2860
t = btf_last_type(btf);
3118
2861
btf_type_inc_vlen(t);
3119
2862
3120
-
btf->hdr->type_len += sz;
3121
-
btf->hdr->str_off += sz;
2863
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
3122
2864
return 0;
3123
2865
}
3124
2866
···
3136
2880
struct btf_type *t;
3137
2881
struct btf_var *v;
3138
2882
int sz, name_off;
2883
+
int err;
3139
2884
3140
2885
/* non-empty name */
3141
2886
if (str_is_empty(name))
···
3148
2891
return libbpf_err(-EINVAL);
3149
2892
3150
2893
/* deconstruct BTF, if necessary, and invalidate raw_data */
3151
-
if (btf_ensure_modifiable(btf))
3152
-
return libbpf_err(-ENOMEM);
2894
+
err = btf_ensure_modifiable(btf);
2895
+
if (err)
2896
+
return libbpf_err(err);
3153
2897
3154
2898
sz = sizeof(struct btf_type) + sizeof(struct btf_var);
3155
2899
t = btf_add_type_mem(btf, sz);
···
3187
2929
{
3188
2930
struct btf_type *t;
3189
2931
int sz, name_off;
2932
+
int err;
3190
2933
3191
2934
/* non-empty name */
3192
2935
if (str_is_empty(name))
3193
2936
return libbpf_err(-EINVAL);
3194
2937
3195
-
if (btf_ensure_modifiable(btf))
3196
-
return libbpf_err(-ENOMEM);
2938
+
err = btf_ensure_modifiable(btf);
2939
+
if (err)
2940
+
return libbpf_err(err);
3197
2941
3198
2942
sz = sizeof(struct btf_type);
3199
2943
t = btf_add_type_mem(btf, sz);
···
3228
2968
{
3229
2969
struct btf_type *t;
3230
2970
struct btf_var_secinfo *v;
2971
+
int err;
3231
2972
int sz;
3232
2973
3233
2974
/* last type should be BTF_KIND_DATASEC */
···
3242
2981
return libbpf_err(-EINVAL);
3243
2982
3244
2983
/* decompose and invalidate raw data */
3245
-
if (btf_ensure_modifiable(btf))
3246
-
return libbpf_err(-ENOMEM);
2984
+
err = btf_ensure_modifiable(btf);
2985
+
if (err)
2986
+
return libbpf_err(err);
3247
2987
3248
2988
sz = sizeof(struct btf_var_secinfo);
3249
2989
v = btf_add_type_mem(btf, sz);
···
3259
2997
t = btf_last_type(btf);
3260
2998
btf_type_inc_vlen(t);
3261
2999
3262
-
btf->hdr->type_len += sz;
3263
-
btf->hdr->str_off += sz;
3000
+
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
3264
3001
return 0;
3265
3002
}
3266
3003
···
3268
3007
{
3269
3008
struct btf_type *t;
3270
3009
int sz, value_off;
3010
+
int err;
3271
3011
3272
3012
if (str_is_empty(value) || component_idx < -1)
3273
3013
return libbpf_err(-EINVAL);
···
3276
3014
if (validate_type_id(ref_type_id))
3277
3015
return libbpf_err(-EINVAL);
3278
3016
3279
-
if (btf_ensure_modifiable(btf))
3280
-
return libbpf_err(-ENOMEM);
3017
+
err = btf_ensure_modifiable(btf);
3018
+
if (err)
3019
+
return libbpf_err(err);
3281
3020
3282
3021
sz = sizeof(struct btf_type) + sizeof(struct btf_decl_tag);
3283
3022
t = btf_add_type_mem(btf, sz);
···
3902
3639
return libbpf_err(-EINVAL);
3903
3640
}
3904
3641
3905
-
if (btf_ensure_modifiable(btf)) {
3906
-
err = -ENOMEM;
3642
+
err = btf_ensure_modifiable(btf);
3643
+
if (err)
3907
3644
goto done;
3908
-
}
3909
3645
3910
3646
err = btf_dedup_prep(d);
3911
3647
if (err) {
···
4224
3962
4225
3963
/* replace BTF string data and hash with deduped ones */
4226
3964
strset__free(d->btf->strs_set);
4227
-
d->btf->hdr->str_len = strset__data_size(d->strs_set);
3965
+
btf_hdr_update_str_len(d->btf, strset__data_size(d->strs_set));
4228
3966
d->btf->strs_set = d->strs_set;
4229
3967
d->strs_set = NULL;
4230
3968
d->btf->strs_deduped = true;
···
5657
5395
continue;
5658
5396
5659
5397
t = btf__type_by_id(d->btf, id);
5660
-
len = btf_type_size(t);
5398
+
len = btf_type_size(d->btf, t);
5661
5399
if (len < 0)
5662
5400
return len;
5663
5401
···
5671
5409
/* shrink struct btf's internal types index and update btf_header */
5672
5410
d->btf->nr_types = next_type_id - d->btf->start_id;
5673
5411
d->btf->type_offs_cap = d->btf->nr_types;
5674
-
d->btf->hdr->type_len = p - d->btf->types_data;
5412
+
d->btf->hdr.type_len = p - d->btf->types_data;
5675
5413
new_offs = libbpf_reallocarray(d->btf->type_offs, d->btf->type_offs_cap,
5676
5414
sizeof(*new_offs));
5677
5415
if (d->btf->type_offs_cap && !new_offs)
5678
5416
return -ENOMEM;
5679
5417
d->btf->type_offs = new_offs;
5680
-
d->btf->hdr->str_off = d->btf->hdr->type_len;
5681
-
d->btf->raw_size = d->btf->hdr->hdr_len + d->btf->hdr->type_len + d->btf->hdr->str_len;
5418
+
if (d->btf->layout)
5419
+
d->btf->hdr.layout_off = d->btf->hdr.type_off + d->btf->hdr.type_len;
5420
+
d->btf->hdr.str_off = d->btf->hdr.type_off + d->btf->hdr.type_len + d->btf->hdr.layout_len;
5421
+
d->btf->raw_size = d->btf->hdr.hdr_len + d->btf->hdr.type_off + d->btf->hdr.type_len +
5422
+
d->btf->hdr.layout_len + d->btf->hdr.str_len;
5682
5423
return 0;
5683
5424
}
5684
5425
···
6139
5874
goto done;
6140
5875
}
6141
5876
dist.split_start_id = btf__type_cnt(old_base);
6142
-
dist.split_start_str = old_base->hdr->str_len;
5877
+
dist.split_start_str = old_base->hdr.str_len;
6143
5878
6144
5879
/* Pass over src split BTF; generate the list of base BTF type ids it
6145
5880
* references; these will constitute our distilled BTF set to be
···
6208
5943
6209
5944
const struct btf_header *btf_header(const struct btf *btf)
6210
5945
{
6211
-
return btf->hdr;
5946
+
return &btf->hdr;
6212
5947
}
6213
5948
6214
5949
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
6215
5950
{
6216
5951
btf->base_btf = (struct btf *)base_btf;
6217
5952
btf->start_id = btf__type_cnt(base_btf);
6218
-
btf->start_str_off = base_btf->hdr->str_len + base_btf->start_str_off;
5953
+
btf->start_str_off = base_btf->hdr.str_len + base_btf->start_str_off;
6219
5954
}
6220
5955
6221
5956
int btf__relocate(struct btf *btf, const struct btf *base_btf)
···
6282
6017
goto done;
6283
6018
}
6284
6019
6285
-
new_types = calloc(btf->hdr->type_len, 1);
6020
+
new_types = calloc(btf->hdr.type_len, 1);
6286
6021
if (!new_types) {
6287
6022
err = -ENOMEM;
6288
6023
goto done;
6289
6024
}
6290
6025
6291
-
if (btf_ensure_modifiable(btf)) {
6292
-
err = -ENOMEM;
6026
+
err = btf_ensure_modifiable(btf);
6027
+
if (err)
6293
6028
goto done;
6294
-
}
6295
6029
6296
6030
for (i = start_offs; i < id_map_cnt; i++) {
6297
6031
id = id_map[i];
···
6319
6055
6320
6056
id = order_map[i];
6321
6057
t = btf__type_by_id(btf, id);
6322
-
type_size = btf_type_size(t);
6058
+
type_size = btf_type_size(btf, t);
6323
6059
memcpy(nt, t, type_size);
6324
6060
6325
6061
/* fix up referenced IDs for BTF */
···
6345
6081
6346
6082
for (nt = new_types, i = 0; i < id_map_cnt - start_offs; i++) {
6347
6083
btf->type_offs[i] = nt - new_types;
6348
-
nt += btf_type_size(nt);
6084
+
nt += btf_type_size(btf, nt);
6349
6085
}
6350
6086
6351
6087
free(order_map);
+20
tools/lib/bpf/btf.h
+20
tools/lib/bpf/btf.h
···
109
109
*/
110
110
LIBBPF_API struct btf *btf__new_empty_split(struct btf *base_btf);
111
111
112
+
struct btf_new_opts {
113
+
size_t sz;
114
+
struct btf *base_btf; /* optional base BTF */
115
+
bool add_layout; /* add BTF layout information */
116
+
size_t:0;
117
+
};
118
+
#define btf_new_opts__last_field add_layout
119
+
120
+
/**
121
+
* @brief **btf__new_empty_opts()** creates an unpopulated BTF object with
122
+
* optional *base_btf* and BTF kind layout description if *add_layout*
123
+
* is set
124
+
* @return new BTF object instance which has to be eventually freed with
125
+
* **btf__free()**
126
+
*
127
+
* On error, NULL is returned and the thread-local `errno` variable is
128
+
* set to the error code.
129
+
*/
130
+
LIBBPF_API struct btf *btf__new_empty_opts(struct btf_new_opts *opts);
131
+
112
132
/**
113
133
* @brief **btf__distill_base()** creates new versions of the split BTF
114
134
* *src_btf* and its base BTF. The new base BTF will only contain the types
+29
tools/lib/bpf/features.c
+29
tools/lib/bpf/features.c
···
589
589
}
590
590
#endif
591
591
592
+
static int probe_kern_btf_layout(int token_fd)
593
+
{
594
+
static const char strs[] = "\0int";
595
+
__u32 types[] = {
596
+
/* int */
597
+
BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4),
598
+
};
599
+
struct btf_layout layout[] = {
600
+
{ 0, 0, 0 },
601
+
{ sizeof(__u32), 0, 0 },
602
+
};
603
+
struct btf_header hdr = {
604
+
.magic = BTF_MAGIC,
605
+
.version = BTF_VERSION,
606
+
.hdr_len = sizeof(struct btf_header),
607
+
.type_len = sizeof(types),
608
+
.str_off = sizeof(types) + sizeof(layout),
609
+
.str_len = sizeof(strs),
610
+
.layout_off = sizeof(types),
611
+
.layout_len = sizeof(layout),
612
+
};
613
+
614
+
return probe_fd(libbpf__load_raw_btf_hdr(&hdr, (char *)types, strs,
615
+
(char *)layout, token_fd));
616
+
}
617
+
592
618
typedef int (*feature_probe_fn)(int /* token_fd */);
593
619
594
620
static struct kern_feature_cache feature_cache;
···
695
669
},
696
670
[FEAT_UPROBE_SYSCALL] = {
697
671
"kernel supports uprobe syscall", probe_uprobe_syscall,
672
+
},
673
+
[FEAT_BTF_LAYOUT] = {
674
+
"kernel supports BTF layout", probe_kern_btf_layout,
698
675
},
699
676
};
700
677
+67
-21
tools/lib/bpf/libbpf.c
+67
-21
tools/lib/bpf/libbpf.c
···
3138
3138
bool has_type_tag = kernel_supports(obj, FEAT_BTF_TYPE_TAG);
3139
3139
bool has_enum64 = kernel_supports(obj, FEAT_BTF_ENUM64);
3140
3140
bool has_qmark_datasec = kernel_supports(obj, FEAT_BTF_QMARK_DATASEC);
3141
+
bool has_layout = kernel_supports(obj, FEAT_BTF_LAYOUT);
3141
3142
3142
3143
return !has_func || !has_datasec || !has_func_global || !has_float ||
3143
-
!has_decl_tag || !has_type_tag || !has_enum64 || !has_qmark_datasec;
3144
+
!has_decl_tag || !has_type_tag || !has_enum64 || !has_qmark_datasec ||
3145
+
!has_layout;
3144
3146
}
3145
3147
3146
-
static int bpf_object__sanitize_btf(struct bpf_object *obj, struct btf *btf)
3148
+
static struct btf *bpf_object__sanitize_btf(struct bpf_object *obj, struct btf *orig_btf)
3147
3149
{
3148
3150
bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
3149
3151
bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
···
3155
3153
bool has_type_tag = kernel_supports(obj, FEAT_BTF_TYPE_TAG);
3156
3154
bool has_enum64 = kernel_supports(obj, FEAT_BTF_ENUM64);
3157
3155
bool has_qmark_datasec = kernel_supports(obj, FEAT_BTF_QMARK_DATASEC);
3156
+
bool has_layout = kernel_supports(obj, FEAT_BTF_LAYOUT);
3158
3157
int enum64_placeholder_id = 0;
3158
+
const struct btf_header *hdr;
3159
+
struct btf *btf = NULL;
3160
+
const void *raw_data;
3159
3161
struct btf_type *t;
3160
3162
int i, j, vlen;
3163
+
__u32 sz;
3164
+
int err;
3165
+
3166
+
/* clone BTF to sanitize a copy and leave the original intact */
3167
+
raw_data = btf__raw_data(orig_btf, &sz);
3168
+
if (!raw_data)
3169
+
return ERR_PTR(-ENOMEM);
3170
+
/* btf_header() gives us endian-safe header info */
3171
+
hdr = btf_header(orig_btf);
3172
+
3173
+
if (!has_layout && hdr->hdr_len >= sizeof(struct btf_header) &&
3174
+
(hdr->layout_len != 0 || hdr->layout_off != 0)) {
3175
+
const struct btf_header *old_hdr = raw_data;
3176
+
struct btf_header *new_hdr;
3177
+
void *new_raw_data;
3178
+
__u32 new_str_off;
3179
+
3180
+
/*
3181
+
* Need to rewrite BTF to exclude layout information and
3182
+
* move string section to immediately after types.
3183
+
*/
3184
+
new_raw_data = malloc(sz);
3185
+
if (!new_raw_data)
3186
+
return ERR_PTR(-ENOMEM);
3187
+
3188
+
memcpy(new_raw_data, raw_data, sz);
3189
+
new_hdr = new_raw_data;
3190
+
new_hdr->layout_off = 0;
3191
+
new_hdr->layout_len = 0;
3192
+
new_str_off = hdr->type_off + hdr->type_len;
3193
+
/* Handle swapped endian case */
3194
+
if (old_hdr->magic != hdr->magic)
3195
+
new_hdr->str_off = bswap_32(new_str_off);
3196
+
else
3197
+
new_hdr->str_off = new_str_off;
3198
+
3199
+
memmove(new_raw_data + hdr->hdr_len + new_str_off,
3200
+
new_raw_data + hdr->hdr_len + hdr->str_off,
3201
+
hdr->str_len);
3202
+
sz = hdr->hdr_len + hdr->type_off + hdr->type_len + hdr->str_len;
3203
+
btf = btf__new(new_raw_data, sz);
3204
+
free(new_raw_data);
3205
+
} else {
3206
+
btf = btf__new(raw_data, sz);
3207
+
}
3208
+
err = libbpf_get_error(btf);
3209
+
if (err)
3210
+
return ERR_PTR(err);
3211
+
3212
+
/* enforce 8-byte pointers for BPF-targeted BTFs */
3213
+
btf__set_pointer_size(btf, 8);
3161
3214
3162
3215
for (i = 1; i < btf__type_cnt(btf); i++) {
3163
3216
t = (struct btf_type *)btf__type_by_id(btf, i);
···
3290
3233
3291
3234
if (enum64_placeholder_id == 0) {
3292
3235
enum64_placeholder_id = btf__add_int(btf, "enum64_placeholder", 1, 0);
3293
-
if (enum64_placeholder_id < 0)
3294
-
return enum64_placeholder_id;
3295
-
3236
+
if (enum64_placeholder_id < 0) {
3237
+
btf__free(btf);
3238
+
return ERR_PTR(enum64_placeholder_id);
3239
+
}
3296
3240
t = (struct btf_type *)btf__type_by_id(btf, i);
3297
3241
}
3298
3242
···
3307
3249
}
3308
3250
}
3309
3251
3310
-
return 0;
3252
+
return btf;
3311
3253
}
3312
3254
3313
3255
static bool libbpf_needs_btf(const struct bpf_object *obj)
···
3658
3600
3659
3601
sanitize = btf_needs_sanitization(obj);
3660
3602
if (sanitize) {
3661
-
const void *raw_data;
3662
-
__u32 sz;
3663
-
3664
-
/* clone BTF to sanitize a copy and leave the original intact */
3665
-
raw_data = btf__raw_data(obj->btf, &sz);
3666
-
kern_btf = btf__new(raw_data, sz);
3667
-
err = libbpf_get_error(kern_btf);
3668
-
if (err)
3669
-
return err;
3670
-
3671
-
/* enforce 8-byte pointers for BPF-targeted BTFs */
3672
-
btf__set_pointer_size(obj->btf, 8);
3673
-
err = bpf_object__sanitize_btf(obj, kern_btf);
3674
-
if (err)
3675
-
return err;
3603
+
kern_btf = bpf_object__sanitize_btf(obj, obj->btf);
3604
+
if (IS_ERR(kern_btf))
3605
+
return PTR_ERR(kern_btf);
3676
3606
}
3677
3607
3678
3608
if (obj->gen_loader) {
+2
tools/lib/bpf/libbpf.map
+2
tools/lib/bpf/libbpf.map
+6
tools/lib/bpf/libbpf_internal.h
+6
tools/lib/bpf/libbpf_internal.h
···
396
396
FEAT_LDIMM64_FULL_RANGE_OFF,
397
397
/* Kernel supports uprobe syscall */
398
398
FEAT_UPROBE_SYSCALL,
399
+
/* Kernel supports BTF layout information */
400
+
FEAT_BTF_LAYOUT,
399
401
__FEAT_CNT,
400
402
};
401
403
···
424
422
int libbpf__load_raw_btf(const char *raw_types, size_t types_len,
425
423
const char *str_sec, size_t str_len,
426
424
int token_fd);
425
+
int libbpf__load_raw_btf_hdr(const struct btf_header *hdr,
426
+
const char *raw_types, const char *str_sec,
427
+
const char *layout_sec, int token_fd);
428
+
427
429
int btf_load_into_kernel(struct btf *btf,
428
430
char *log_buf, size_t log_sz, __u32 log_level,
429
431
int token_fd);
+29
-19
tools/lib/bpf/libbpf_probes.c
+29
-19
tools/lib/bpf/libbpf_probes.c
···
218
218
return libbpf_err(ret);
219
219
}
220
220
221
+
int libbpf__load_raw_btf_hdr(const struct btf_header *hdr, const char *raw_types,
222
+
const char *str_sec, const char *layout_sec,
223
+
int token_fd)
224
+
{
225
+
LIBBPF_OPTS(bpf_btf_load_opts, opts,
226
+
.token_fd = token_fd,
227
+
.btf_flags = token_fd ? BPF_F_TOKEN_FD : 0,
228
+
);
229
+
int btf_fd, btf_len;
230
+
__u8 *raw_btf;
231
+
232
+
btf_len = hdr->hdr_len + hdr->type_off + hdr->type_len + hdr->str_len + hdr->layout_len;
233
+
raw_btf = malloc(btf_len);
234
+
if (!raw_btf)
235
+
return -ENOMEM;
236
+
237
+
memcpy(raw_btf, hdr, sizeof(*hdr));
238
+
memcpy(raw_btf + hdr->hdr_len + hdr->type_off, raw_types, hdr->type_len);
239
+
memcpy(raw_btf + hdr->hdr_len + hdr->str_off, str_sec, hdr->str_len);
240
+
if (layout_sec)
241
+
memcpy(raw_btf + hdr->hdr_len + hdr->layout_off, layout_sec, hdr->layout_len);
242
+
243
+
btf_fd = bpf_btf_load(raw_btf, btf_len, &opts);
244
+
245
+
free(raw_btf);
246
+
return btf_fd;
247
+
}
248
+
221
249
int libbpf__load_raw_btf(const char *raw_types, size_t types_len,
222
250
const char *str_sec, size_t str_len,
223
251
int token_fd)
···
258
230
.str_off = types_len,
259
231
.str_len = str_len,
260
232
};
261
-
LIBBPF_OPTS(bpf_btf_load_opts, opts,
262
-
.token_fd = token_fd,
263
-
.btf_flags = token_fd ? BPF_F_TOKEN_FD : 0,
264
-
);
265
-
int btf_fd, btf_len;
266
-
__u8 *raw_btf;
267
233
268
-
btf_len = hdr.hdr_len + hdr.type_len + hdr.str_len;
269
-
raw_btf = malloc(btf_len);
270
-
if (!raw_btf)
271
-
return -ENOMEM;
272
-
273
-
memcpy(raw_btf, &hdr, sizeof(hdr));
274
-
memcpy(raw_btf + hdr.hdr_len, raw_types, hdr.type_len);
275
-
memcpy(raw_btf + hdr.hdr_len + hdr.type_len, str_sec, hdr.str_len);
276
-
277
-
btf_fd = bpf_btf_load(raw_btf, btf_len, &opts);
278
-
279
-
free(raw_btf);
280
-
return btf_fd;
234
+
return libbpf__load_raw_btf_hdr(&hdr, raw_types, str_sec, NULL, token_fd);
281
235
}
282
236
283
237
static int load_local_storage_btf(void)
+226
tools/testing/selftests/bpf/prog_tests/btf_kind.c
+226
tools/testing/selftests/bpf/prog_tests/btf_kind.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2026, Oracle and/or its affiliates. */
3
+
4
+
#include <test_progs.h>
5
+
#include <bpf/btf.h>
6
+
#include <bpf/libbpf.h>
7
+
8
+
/* Verify kind encoding exists for each kind */
9
+
static void test_btf_kind_encoding(void)
10
+
{
11
+
LIBBPF_OPTS(btf_new_opts, opts);
12
+
const struct btf_header *hdr;
13
+
const void *raw_btf;
14
+
struct btf *btf;
15
+
__u32 raw_size;
16
+
17
+
opts.add_layout = true;
18
+
btf = btf__new_empty_opts(&opts);
19
+
if (!ASSERT_OK_PTR(btf, "btf_new"))
20
+
return;
21
+
22
+
raw_btf = btf__raw_data(btf, &raw_size);
23
+
if (!ASSERT_OK_PTR(raw_btf, "btf__raw_data"))
24
+
return;
25
+
26
+
hdr = raw_btf;
27
+
28
+
ASSERT_EQ(hdr->layout_off % 4, 0, "layout_aligned");
29
+
ASSERT_EQ(hdr->layout_len, sizeof(struct btf_layout) * NR_BTF_KINDS,
30
+
"layout_len");
31
+
ASSERT_EQ(hdr->str_off, hdr->layout_off + hdr->layout_len, "str_after_layout");
32
+
btf__free(btf);
33
+
34
+
opts.add_layout = false;
35
+
btf = btf__new_empty_opts(&opts);
36
+
if (!ASSERT_OK_PTR(btf, "btf_new"))
37
+
return;
38
+
39
+
raw_btf = btf__raw_data(btf, &raw_size);
40
+
if (!ASSERT_OK_PTR(raw_btf, "btf__raw_data"))
41
+
return;
42
+
43
+
hdr = raw_btf;
44
+
45
+
ASSERT_EQ(hdr->layout_off, 0, "no_layout_off");
46
+
ASSERT_EQ(hdr->layout_len, 0, "no_layout_len");
47
+
ASSERT_EQ(hdr->str_off, hdr->type_off + hdr->type_len, "strs_after_types");
48
+
btf__free(btf);
49
+
}
50
+
51
+
static int write_raw_btf(void *raw_btf, size_t raw_size, char *file)
52
+
{
53
+
int fd = mkstemp(file);
54
+
ssize_t n;
55
+
56
+
if (!ASSERT_OK_FD(fd, "open_raw_btf"))
57
+
return -1;
58
+
n = write(fd, raw_btf, raw_size);
59
+
close(fd);
60
+
if (!ASSERT_EQ(n, (ssize_t)raw_size, "write_raw_btf"))
61
+
return -1;
62
+
return 0;
63
+
}
64
+
65
+
/*
66
+
* Fabricate an unrecognized kind at BTF_KIND_MAX + 1, and after adding
67
+
* the appropriate struct/typedefs to the BTF such that it recognizes
68
+
* this kind, ensure that parsing of BTF containing the unrecognized kind
69
+
* can succeed.
70
+
*/
71
+
void test_btf_kind_decoding(void)
72
+
{
73
+
char btf_kind_file1[] = "/tmp/test_btf_kind.XXXXXX";
74
+
char btf_kind_file2[] = "/tmp/test_btf_kind.XXXXXX";
75
+
char btf_kind_file3[] = "/tmp/test_btf_kind.XXXXXX";
76
+
struct btf *btf = NULL, *new_btf = NULL;
77
+
__s32 int_id, unrec_id, id, id2;
78
+
LIBBPF_OPTS(btf_new_opts, opts);
79
+
struct btf_layout *l;
80
+
struct btf_header *hdr;
81
+
const void *raw_btf;
82
+
struct btf_type *t;
83
+
void *new_raw_btf;
84
+
void *str_data;
85
+
__u32 raw_size;
86
+
87
+
opts.add_layout = true;
88
+
btf = btf__new_empty_opts(&opts);
89
+
if (!ASSERT_OK_PTR(btf, "btf_new"))
90
+
return;
91
+
92
+
int_id = btf__add_int(btf, "test_char", 1, BTF_INT_CHAR);
93
+
if (!ASSERT_GT(int_id, 0, "add_int_id"))
94
+
return;
95
+
96
+
/*
97
+
* Create our type with unrecognized kind by adding a typedef kind
98
+
* we will overwrite it with our unrecognized kind value.
99
+
*/
100
+
unrec_id = btf__add_typedef(btf, "unrec_kind", int_id);
101
+
if (!ASSERT_GT(unrec_id, 0, "add_unrec_id"))
102
+
return;
103
+
104
+
/*
105
+
* Add an id after it that we will look up to verify we can parse
106
+
* beyond unrecognized kinds.
107
+
*/
108
+
id = btf__add_typedef(btf, "test_lookup", int_id);
109
+
if (!ASSERT_GT(id, 0, "add_test_lookup_id"))
110
+
return;
111
+
id2 = btf__add_typedef(btf, "test_lookup2", int_id);
112
+
if (!ASSERT_GT(id2, 0, "add_test_lookup_id2"))
113
+
return;
114
+
115
+
raw_btf = (void *)btf__raw_data(btf, &raw_size);
116
+
if (!ASSERT_OK_PTR(raw_btf, "btf__raw_data"))
117
+
return;
118
+
119
+
new_raw_btf = calloc(1, raw_size + sizeof(*l));
120
+
if (!ASSERT_OK_PTR(new_raw_btf, "calloc_raw_btf"))
121
+
return;
122
+
memcpy(new_raw_btf, raw_btf, raw_size);
123
+
124
+
hdr = new_raw_btf;
125
+
126
+
/* Move strings to make space for one new layout description */
127
+
raw_size += sizeof(*l);
128
+
str_data = new_raw_btf + hdr->hdr_len + hdr->str_off;
129
+
memmove(str_data + sizeof(*l), str_data, hdr->str_len);
130
+
hdr->str_off += sizeof(*l);
131
+
132
+
/* Add new layout description */
133
+
hdr->layout_len += sizeof(*l);
134
+
l = new_raw_btf + hdr->hdr_len + hdr->layout_off;
135
+
l[NR_BTF_KINDS].info_sz = 0;
136
+
l[NR_BTF_KINDS].elem_sz = 0;
137
+
l[NR_BTF_KINDS].flags = 0;
138
+
139
+
/* Now modify typedef added above to be an unrecognized kind. */
140
+
t = (void *)hdr + hdr->hdr_len + hdr->type_off + sizeof(struct btf_type) +
141
+
sizeof(__u32);
142
+
t->info = (NR_BTF_KINDS << 24);
143
+
144
+
/* Write BTF to a raw file, ready for parsing. */
145
+
if (write_raw_btf(new_raw_btf, raw_size, btf_kind_file1))
146
+
goto out;
147
+
148
+
/*
149
+
* Verify parsing succeeds, and that we can read type info past
150
+
* the unrecognized kind.
151
+
*/
152
+
new_btf = btf__parse_raw(btf_kind_file1);
153
+
if (ASSERT_OK_PTR(new_btf, "btf__parse_raw")) {
154
+
ASSERT_EQ(btf__find_by_name(new_btf, "unrec_kind"), unrec_id,
155
+
"unrec_kind_found");
156
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup",
157
+
BTF_KIND_TYPEDEF), id,
158
+
"verify_id_lookup");
159
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup2",
160
+
BTF_KIND_TYPEDEF), id2,
161
+
"verify_id2_lookup");
162
+
}
163
+
btf__free(new_btf);
164
+
new_btf = NULL;
165
+
166
+
/*
167
+
* Next, change info_sz to equal sizeof(struct btf_type); this means the
168
+
* "test_lookup" kind will be reinterpreted as a singular info element
169
+
* following the unrecognized kind.
170
+
*/
171
+
l[NR_BTF_KINDS].info_sz = sizeof(struct btf_type);
172
+
if (write_raw_btf(new_raw_btf, raw_size, btf_kind_file2))
173
+
goto out;
174
+
175
+
new_btf = btf__parse_raw(btf_kind_file2);
176
+
if (ASSERT_OK_PTR(new_btf, "btf__parse_raw")) {
177
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup",
178
+
BTF_KIND_TYPEDEF), -ENOENT,
179
+
"verify_id_not_found");
180
+
/* id of "test_lookup2" will be id2 -1 as we have removed one type */
181
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup2",
182
+
BTF_KIND_TYPEDEF), id2 - 1,
183
+
"verify_id_lookup2");
184
+
185
+
}
186
+
btf__free(new_btf);
187
+
new_btf = NULL;
188
+
189
+
/*
190
+
* Change elem_sz to equal sizeof(struct btf_type) and set vlen
191
+
* associated with unrecognized type to 1; this allows us to verify
192
+
* vlen-specified BTF can still be parsed.
193
+
*/
194
+
l[NR_BTF_KINDS].info_sz = 0;
195
+
l[NR_BTF_KINDS].elem_sz = sizeof(struct btf_type);
196
+
t->info |= 1;
197
+
if (write_raw_btf(new_raw_btf, raw_size, btf_kind_file3))
198
+
goto out;
199
+
200
+
new_btf = btf__parse_raw(btf_kind_file3);
201
+
if (ASSERT_OK_PTR(new_btf, "btf__parse_raw")) {
202
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup",
203
+
BTF_KIND_TYPEDEF), -ENOENT,
204
+
"verify_id_not_found");
205
+
/* id of "test_lookup2" will be id2 -1 as we have removed one type */
206
+
ASSERT_EQ(btf__find_by_name_kind(new_btf, "test_lookup2",
207
+
BTF_KIND_TYPEDEF), id2 - 1,
208
+
"verify_id_lookup2");
209
+
210
+
}
211
+
out:
212
+
btf__free(new_btf);
213
+
free(new_raw_btf);
214
+
unlink(btf_kind_file1);
215
+
unlink(btf_kind_file2);
216
+
unlink(btf_kind_file3);
217
+
btf__free(btf);
218
+
}
219
+
220
+
void test_btf_kind(void)
221
+
{
222
+
if (test__start_subtest("btf_kind_encoding"))
223
+
test_btf_kind_encoding();
224
+
if (test__start_subtest("btf_kind_decoding"))
225
+
test_btf_kind_decoding();
226
+
}