Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge branch 'net-page_pool-add-netlink-based-introspection'
Jakub Kicinski says:
====================
net: page_pool: add netlink-based introspection
We recently started to deploy newer kernels / drivers at Meta,
making significant use of page pools for the first time.
We immediately run into page pool leaks both real and false positive
warnings. As Eric pointed out/predicted there's no guarantee that
applications will read / close their sockets so a page pool page
may be stuck in a socket (but not leaked) forever. This happens
a lot in our fleet. Most of these are obviously due to application
bugs but we should not be printing kernel warnings due to minor
application resource leaks.
Conversely the page pool memory may get leaked at runtime, and
we have no way to detect / track that, unless someone reconfigures
the NIC and destroys the page pools which leaked the pages.
The solution presented here is to expose the memory use of page
pools via netlink. This allows for continuous monitoring of memory
used by page pools, regardless if they were destroyed or not.
Sample in patch 15 can print the memory use and recycling
efficiency:
$ ./page-pool
eth0[2] page pools: 10 (zombies: 0)
refs: 41984 bytes: 171966464 (refs: 0 bytes: 0)
recycling: 90.3% (alloc: 656:397681 recycle: 89652:270201)
v4:
- use dev_net(netdev)->loopback_dev
- extend inflight doc
v3: https://lore.kernel.org/all/20231122034420.1158898-1-kuba@kernel.org/
- ID is still here, can't decide if it matters
- rename destroyed -> detach-time, good enough?
- fix build for netsec
v2: https://lore.kernel.org/r/20231121000048.789613-1-kuba@kernel.org
- hopefully fix build with PAGE_POOL=n
v1: https://lore.kernel.org/all/20231024160220.3973311-1-kuba@kernel.org/
- The main change compared to the RFC is that the API now exposes
outstanding references and byte counts even for "live" page pools.
The warning is no longer printed if page pool is accessible via netlink.
RFC: https://lore.kernel.org/all/20230816234303.3786178-1-kuba@kernel.org/
====================
Link: https://lore.kernel.org/r/20231126230740.2148636-1-kuba@kernel.org
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
+1574
-33
+172
Documentation/netlink/specs/netdev.yaml
+172
Documentation/netlink/specs/netdev.yaml
···
86
86
See Documentation/networking/xdp-rx-metadata.rst for more details.
87
87
type: u64
88
88
enum: xdp-rx-metadata
89
+
-
90
+
name: page-pool
91
+
attributes:
92
+
-
93
+
name: id
94
+
doc: Unique ID of a Page Pool instance.
95
+
type: uint
96
+
checks:
97
+
min: 1
98
+
max: u32-max
99
+
-
100
+
name: ifindex
101
+
doc: |
102
+
ifindex of the netdev to which the pool belongs.
103
+
May be reported as 0 if the page pool was allocated for a netdev
104
+
which got destroyed already (page pools may outlast their netdevs
105
+
because they wait for all memory to be returned).
106
+
type: u32
107
+
checks:
108
+
min: 1
109
+
max: s32-max
110
+
-
111
+
name: napi-id
112
+
doc: Id of NAPI using this Page Pool instance.
113
+
type: uint
114
+
checks:
115
+
min: 1
116
+
max: u32-max
117
+
-
118
+
name: inflight
119
+
type: uint
120
+
doc: |
121
+
Number of outstanding references to this page pool (allocated
122
+
but yet to be freed pages). Allocated pages may be held in
123
+
socket receive queues, driver receive ring, page pool recycling
124
+
ring, the page pool cache, etc.
125
+
-
126
+
name: inflight-mem
127
+
type: uint
128
+
doc: |
129
+
Amount of memory held by inflight pages.
130
+
-
131
+
name: detach-time
132
+
type: uint
133
+
doc: |
134
+
Seconds in CLOCK_BOOTTIME of when Page Pool was detached by
135
+
the driver. Once detached Page Pool can no longer be used to
136
+
allocate memory.
137
+
Page Pools wait for all the memory allocated from them to be freed
138
+
before truly disappearing. "Detached" Page Pools cannot be
139
+
"re-attached", they are just waiting to disappear.
140
+
Attribute is absent if Page Pool has not been detached, and
141
+
can still be used to allocate new memory.
142
+
-
143
+
name: page-pool-info
144
+
subset-of: page-pool
145
+
attributes:
146
+
-
147
+
name: id
148
+
-
149
+
name: ifindex
150
+
-
151
+
name: page-pool-stats
152
+
doc: |
153
+
Page pool statistics, see docs for struct page_pool_stats
154
+
for information about individual statistics.
155
+
attributes:
156
+
-
157
+
name: info
158
+
doc: Page pool identifying information.
159
+
type: nest
160
+
nested-attributes: page-pool-info
161
+
-
162
+
name: alloc-fast
163
+
type: uint
164
+
value: 8 # reserve some attr ids in case we need more metadata later
165
+
-
166
+
name: alloc-slow
167
+
type: uint
168
+
-
169
+
name: alloc-slow-high-order
170
+
type: uint
171
+
-
172
+
name: alloc-empty
173
+
type: uint
174
+
-
175
+
name: alloc-refill
176
+
type: uint
177
+
-
178
+
name: alloc-waive
179
+
type: uint
180
+
-
181
+
name: recycle-cached
182
+
type: uint
183
+
-
184
+
name: recycle-cache-full
185
+
type: uint
186
+
-
187
+
name: recycle-ring
188
+
type: uint
189
+
-
190
+
name: recycle-ring-full
191
+
type: uint
192
+
-
193
+
name: recycle-released-refcnt
194
+
type: uint
89
195
90
196
operations:
91
197
list:
···
226
120
doc: Notification about device configuration being changed.
227
121
notify: dev-get
228
122
mcgrp: mgmt
123
+
-
124
+
name: page-pool-get
125
+
doc: |
126
+
Get / dump information about Page Pools.
127
+
(Only Page Pools associated with a net_device can be listed.)
128
+
attribute-set: page-pool
129
+
do:
130
+
request:
131
+
attributes:
132
+
- id
133
+
reply: &pp-reply
134
+
attributes:
135
+
- id
136
+
- ifindex
137
+
- napi-id
138
+
- inflight
139
+
- inflight-mem
140
+
- detach-time
141
+
dump:
142
+
reply: *pp-reply
143
+
config-cond: page-pool
144
+
-
145
+
name: page-pool-add-ntf
146
+
doc: Notification about page pool appearing.
147
+
notify: page-pool-get
148
+
mcgrp: page-pool
149
+
config-cond: page-pool
150
+
-
151
+
name: page-pool-del-ntf
152
+
doc: Notification about page pool disappearing.
153
+
notify: page-pool-get
154
+
mcgrp: page-pool
155
+
config-cond: page-pool
156
+
-
157
+
name: page-pool-change-ntf
158
+
doc: Notification about page pool configuration being changed.
159
+
notify: page-pool-get
160
+
mcgrp: page-pool
161
+
config-cond: page-pool
162
+
-
163
+
name: page-pool-stats-get
164
+
doc: Get page pool statistics.
165
+
attribute-set: page-pool-stats
166
+
do:
167
+
request:
168
+
attributes:
169
+
- info
170
+
reply: &pp-stats-reply
171
+
attributes:
172
+
- info
173
+
- alloc-fast
174
+
- alloc-slow
175
+
- alloc-slow-high-order
176
+
- alloc-empty
177
+
- alloc-refill
178
+
- alloc-waive
179
+
- recycle-cached
180
+
- recycle-cache-full
181
+
- recycle-ring
182
+
- recycle-ring-full
183
+
- recycle-released-refcnt
184
+
dump:
185
+
reply: *pp-stats-reply
186
+
config-cond: page-pool-stats
229
187
230
188
mcast-groups:
231
189
list:
232
190
-
233
191
name: mgmt
192
+
-
193
+
name: page-pool
+8
-2
Documentation/networking/page_pool.rst
+8
-2
Documentation/networking/page_pool.rst
···
41
41
| Fast cache | | ptr-ring cache |
42
42
+-----------------+ +------------------+
43
43
44
+
Monitoring
45
+
==========
46
+
Information about page pools on the system can be accessed via the netdev
47
+
genetlink family (see Documentation/netlink/specs/netdev.yaml).
48
+
44
49
API interface
45
50
=============
46
51
The number of pools created **must** match the number of hardware queues
···
112
107
It takes a pointer to a ``struct page_pool`` and a pointer to a struct
113
108
page_pool_stats allocated by the caller.
114
109
115
-
The API will fill in the provided struct page_pool_stats with
116
-
statistics about the page_pool.
110
+
Older drivers expose page pool statistics via ethtool or debugfs.
111
+
The same statistics are accessible via the netlink netdev family
112
+
in a driver-independent fashion.
117
113
118
114
.. kernel-doc:: include/net/page_pool/types.h
119
115
:identifiers: struct page_pool_recycle_stats
+1
drivers/net/ethernet/broadcom/bnxt/bnxt.c
+1
drivers/net/ethernet/broadcom/bnxt/bnxt.c
+1
drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+1
drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+1
drivers/net/ethernet/microsoft/mana/mana_en.c
+1
drivers/net/ethernet/microsoft/mana/mana_en.c
+2
drivers/net/ethernet/socionext/netsec.c
+2
drivers/net/ethernet/socionext/netsec.c
+20
include/linux/list.h
+20
include/linux/list.h
···
1119
1119
old->first = NULL;
1120
1120
}
1121
1121
1122
+
/**
1123
+
* hlist_splice_init() - move all entries from one list to another
1124
+
* @from: hlist_head from which entries will be moved
1125
+
* @last: last entry on the @from list
1126
+
* @to: hlist_head to which entries will be moved
1127
+
*
1128
+
* @to can be empty, @from must contain at least @last.
1129
+
*/
1130
+
static inline void hlist_splice_init(struct hlist_head *from,
1131
+
struct hlist_node *last,
1132
+
struct hlist_head *to)
1133
+
{
1134
+
if (to->first)
1135
+
to->first->pprev = &last->next;
1136
+
last->next = to->first;
1137
+
to->first = from->first;
1138
+
from->first->pprev = &to->first;
1139
+
from->first = NULL;
1140
+
}
1141
+
1122
1142
#define hlist_entry(ptr, type, member) container_of(ptr,type,member)
1123
1143
1124
1144
#define hlist_for_each(pos, head) \
+4
include/linux/netdevice.h
+4
include/linux/netdevice.h
···
2447
2447
#if IS_ENABLED(CONFIG_DPLL)
2448
2448
struct dpll_pin *dpll_pin;
2449
2449
#endif
2450
+
#if IS_ENABLED(CONFIG_PAGE_POOL)
2451
+
/** @page_pools: page pools created for this netdevice */
2452
+
struct hlist_head page_pools;
2453
+
#endif
2450
2454
};
2451
2455
#define to_net_dev(d) container_of(d, struct net_device, dev)
2452
2456
+2
include/linux/poison.h
+2
include/linux/poison.h
···
83
83
84
84
/********** net/core/skbuff.c **********/
85
85
#define SKB_LIST_POISON_NEXT ((void *)(0x800 + POISON_POINTER_DELTA))
86
+
/********** net/ **********/
87
+
#define NET_PTR_POISON ((void *)(0x801 + POISON_POINTER_DELTA))
86
88
87
89
/********** kernel/bpf/ **********/
88
90
#define BPF_PTR_POISON ((void *)(0xeB9FUL + POISON_POINTER_DELTA))
+2
-6
include/net/page_pool/helpers.h
+2
-6
include/net/page_pool/helpers.h
···
55
55
#include <net/page_pool/types.h>
56
56
57
57
#ifdef CONFIG_PAGE_POOL_STATS
58
+
/* Deprecated driver-facing API, use netlink instead */
58
59
int page_pool_ethtool_stats_get_count(void);
59
60
u8 *page_pool_ethtool_stats_get_strings(u8 *data);
60
61
u64 *page_pool_ethtool_stats_get(u64 *data, void *stats);
61
62
62
-
/*
63
-
* Drivers that wish to harvest page pool stats and report them to users
64
-
* (perhaps via ethtool, debugfs, or another mechanism) can allocate a
65
-
* struct page_pool_stats call page_pool_get_stats to get stats for the specified pool.
66
-
*/
67
-
bool page_pool_get_stats(struct page_pool *pool,
63
+
bool page_pool_get_stats(const struct page_pool *pool,
68
64
struct page_pool_stats *stats);
69
65
#else
70
66
static inline int page_pool_ethtool_stats_get_count(void)
+10
include/net/page_pool/types.h
+10
include/net/page_pool/types.h
···
5
5
6
6
#include <linux/dma-direction.h>
7
7
#include <linux/ptr_ring.h>
8
+
#include <linux/types.h>
8
9
9
10
#define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA
10
11
* map/unmap
···
49
48
* @pool_size: size of the ptr_ring
50
49
* @nid: NUMA node id to allocate from pages from
51
50
* @dev: device, for DMA pre-mapping purposes
51
+
* @netdev: netdev this pool will serve (leave as NULL if none or multiple)
52
52
* @napi: NAPI which is the sole consumer of pages, otherwise NULL
53
53
* @dma_dir: DMA mapping direction
54
54
* @max_len: max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV
···
68
66
unsigned int offset;
69
67
);
70
68
struct_group_tagged(page_pool_params_slow, slow,
69
+
struct net_device *netdev;
71
70
/* private: used by test code only */
72
71
void (*init_callback)(struct page *page, void *arg);
73
72
void *init_arg;
···
190
187
191
188
/* Slow/Control-path information follows */
192
189
struct page_pool_params_slow slow;
190
+
/* User-facing fields, protected by page_pools_lock */
191
+
struct {
192
+
struct hlist_node list;
193
+
u64 detach_time;
194
+
u32 napi_id;
195
+
u32 id;
196
+
} user;
193
197
};
194
198
195
199
struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
+36
include/uapi/linux/netdev.h
+36
include/uapi/linux/netdev.h
···
65
65
};
66
66
67
67
enum {
68
+
NETDEV_A_PAGE_POOL_ID = 1,
69
+
NETDEV_A_PAGE_POOL_IFINDEX,
70
+
NETDEV_A_PAGE_POOL_NAPI_ID,
71
+
NETDEV_A_PAGE_POOL_INFLIGHT,
72
+
NETDEV_A_PAGE_POOL_INFLIGHT_MEM,
73
+
NETDEV_A_PAGE_POOL_DETACH_TIME,
74
+
75
+
__NETDEV_A_PAGE_POOL_MAX,
76
+
NETDEV_A_PAGE_POOL_MAX = (__NETDEV_A_PAGE_POOL_MAX - 1)
77
+
};
78
+
79
+
enum {
80
+
NETDEV_A_PAGE_POOL_STATS_INFO = 1,
81
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST = 8,
82
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW,
83
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER,
84
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY,
85
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL,
86
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE,
87
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED,
88
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL,
89
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING,
90
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL,
91
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT,
92
+
93
+
__NETDEV_A_PAGE_POOL_STATS_MAX,
94
+
NETDEV_A_PAGE_POOL_STATS_MAX = (__NETDEV_A_PAGE_POOL_STATS_MAX - 1)
95
+
};
96
+
97
+
enum {
68
98
NETDEV_CMD_DEV_GET = 1,
69
99
NETDEV_CMD_DEV_ADD_NTF,
70
100
NETDEV_CMD_DEV_DEL_NTF,
71
101
NETDEV_CMD_DEV_CHANGE_NTF,
102
+
NETDEV_CMD_PAGE_POOL_GET,
103
+
NETDEV_CMD_PAGE_POOL_ADD_NTF,
104
+
NETDEV_CMD_PAGE_POOL_DEL_NTF,
105
+
NETDEV_CMD_PAGE_POOL_CHANGE_NTF,
106
+
NETDEV_CMD_PAGE_POOL_STATS_GET,
72
107
73
108
__NETDEV_CMD_MAX,
74
109
NETDEV_CMD_MAX = (__NETDEV_CMD_MAX - 1)
75
110
};
76
111
77
112
#define NETDEV_MCGRP_MGMT "mgmt"
113
+
#define NETDEV_MCGRP_PAGE_POOL "page-pool"
78
114
79
115
#endif /* _UAPI_LINUX_NETDEV_H */
+1
-1
net/core/Makefile
+1
-1
net/core/Makefile
···
18
18
obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) += dev_addr_lists_test.o
19
19
20
20
obj-y += net-sysfs.o
21
-
obj-$(CONFIG_PAGE_POOL) += page_pool.o
21
+
obj-$(CONFIG_PAGE_POOL) += page_pool.o page_pool_user.o
22
22
obj-$(CONFIG_PROC_FS) += net-procfs.o
23
23
obj-$(CONFIG_NET_PKTGEN) += pktgen.o
24
24
obj-$(CONFIG_NETPOLL) += netpoll.o
+60
net/core/netdev-genl-gen.c
+60
net/core/netdev-genl-gen.c
···
10
10
11
11
#include <uapi/linux/netdev.h>
12
12
13
+
/* Integer value ranges */
14
+
static const struct netlink_range_validation netdev_a_page_pool_id_range = {
15
+
.min = 1ULL,
16
+
.max = 4294967295ULL,
17
+
};
18
+
19
+
static const struct netlink_range_validation netdev_a_page_pool_ifindex_range = {
20
+
.min = 1ULL,
21
+
.max = 2147483647ULL,
22
+
};
23
+
24
+
/* Common nested types */
25
+
const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFINDEX + 1] = {
26
+
[NETDEV_A_PAGE_POOL_ID] = NLA_POLICY_FULL_RANGE(NLA_UINT, &netdev_a_page_pool_id_range),
27
+
[NETDEV_A_PAGE_POOL_IFINDEX] = NLA_POLICY_FULL_RANGE(NLA_U32, &netdev_a_page_pool_ifindex_range),
28
+
};
29
+
13
30
/* NETDEV_CMD_DEV_GET - do */
14
31
static const struct nla_policy netdev_dev_get_nl_policy[NETDEV_A_DEV_IFINDEX + 1] = {
15
32
[NETDEV_A_DEV_IFINDEX] = NLA_POLICY_MIN(NLA_U32, 1),
16
33
};
34
+
35
+
/* NETDEV_CMD_PAGE_POOL_GET - do */
36
+
#ifdef CONFIG_PAGE_POOL
37
+
static const struct nla_policy netdev_page_pool_get_nl_policy[NETDEV_A_PAGE_POOL_ID + 1] = {
38
+
[NETDEV_A_PAGE_POOL_ID] = NLA_POLICY_FULL_RANGE(NLA_UINT, &netdev_a_page_pool_id_range),
39
+
};
40
+
#endif /* CONFIG_PAGE_POOL */
41
+
42
+
/* NETDEV_CMD_PAGE_POOL_STATS_GET - do */
43
+
#ifdef CONFIG_PAGE_POOL_STATS
44
+
static const struct nla_policy netdev_page_pool_stats_get_nl_policy[NETDEV_A_PAGE_POOL_STATS_INFO + 1] = {
45
+
[NETDEV_A_PAGE_POOL_STATS_INFO] = NLA_POLICY_NESTED(netdev_page_pool_info_nl_policy),
46
+
};
47
+
#endif /* CONFIG_PAGE_POOL_STATS */
17
48
18
49
/* Ops table for netdev */
19
50
static const struct genl_split_ops netdev_nl_ops[] = {
···
60
29
.dumpit = netdev_nl_dev_get_dumpit,
61
30
.flags = GENL_CMD_CAP_DUMP,
62
31
},
32
+
#ifdef CONFIG_PAGE_POOL
33
+
{
34
+
.cmd = NETDEV_CMD_PAGE_POOL_GET,
35
+
.doit = netdev_nl_page_pool_get_doit,
36
+
.policy = netdev_page_pool_get_nl_policy,
37
+
.maxattr = NETDEV_A_PAGE_POOL_ID,
38
+
.flags = GENL_CMD_CAP_DO,
39
+
},
40
+
{
41
+
.cmd = NETDEV_CMD_PAGE_POOL_GET,
42
+
.dumpit = netdev_nl_page_pool_get_dumpit,
43
+
.flags = GENL_CMD_CAP_DUMP,
44
+
},
45
+
#endif /* CONFIG_PAGE_POOL */
46
+
#ifdef CONFIG_PAGE_POOL_STATS
47
+
{
48
+
.cmd = NETDEV_CMD_PAGE_POOL_STATS_GET,
49
+
.doit = netdev_nl_page_pool_stats_get_doit,
50
+
.policy = netdev_page_pool_stats_get_nl_policy,
51
+
.maxattr = NETDEV_A_PAGE_POOL_STATS_INFO,
52
+
.flags = GENL_CMD_CAP_DO,
53
+
},
54
+
{
55
+
.cmd = NETDEV_CMD_PAGE_POOL_STATS_GET,
56
+
.dumpit = netdev_nl_page_pool_stats_get_dumpit,
57
+
.flags = GENL_CMD_CAP_DUMP,
58
+
},
59
+
#endif /* CONFIG_PAGE_POOL_STATS */
63
60
};
64
61
65
62
static const struct genl_multicast_group netdev_nl_mcgrps[] = {
66
63
[NETDEV_NLGRP_MGMT] = { "mgmt", },
64
+
[NETDEV_NLGRP_PAGE_POOL] = { "page-pool", },
67
65
};
68
66
69
67
struct genl_family netdev_nl_family __ro_after_init = {
+11
net/core/netdev-genl-gen.h
+11
net/core/netdev-genl-gen.h
···
11
11
12
12
#include <uapi/linux/netdev.h>
13
13
14
+
/* Common nested types */
15
+
extern const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFINDEX + 1];
16
+
14
17
int netdev_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info);
15
18
int netdev_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
19
+
int netdev_nl_page_pool_get_doit(struct sk_buff *skb, struct genl_info *info);
20
+
int netdev_nl_page_pool_get_dumpit(struct sk_buff *skb,
21
+
struct netlink_callback *cb);
22
+
int netdev_nl_page_pool_stats_get_doit(struct sk_buff *skb,
23
+
struct genl_info *info);
24
+
int netdev_nl_page_pool_stats_get_dumpit(struct sk_buff *skb,
25
+
struct netlink_callback *cb);
16
26
17
27
enum {
18
28
NETDEV_NLGRP_MGMT,
29
+
NETDEV_NLGRP_PAGE_POOL,
19
30
};
20
31
21
32
extern struct genl_family netdev_nl_family;
+47
-22
net/core/page_pool.c
+47
-22
net/core/page_pool.c
···
23
23
24
24
#include <trace/events/page_pool.h>
25
25
26
+
#include "page_pool_priv.h"
27
+
26
28
#define DEFER_TIME (msecs_to_jiffies(1000))
27
29
#define DEFER_WARN_INTERVAL (60 * HZ)
28
30
···
71
69
* is passed to this API which is filled in. The caller can then report
72
70
* those stats to the user (perhaps via ethtool, debugfs, etc.).
73
71
*/
74
-
bool page_pool_get_stats(struct page_pool *pool,
72
+
bool page_pool_get_stats(const struct page_pool *pool,
75
73
struct page_pool_stats *stats)
76
74
{
77
75
int cpu = 0;
···
240
238
return 0;
241
239
}
242
240
241
+
static void page_pool_uninit(struct page_pool *pool)
242
+
{
243
+
ptr_ring_cleanup(&pool->ring, NULL);
244
+
245
+
if (pool->p.flags & PP_FLAG_DMA_MAP)
246
+
put_device(pool->p.dev);
247
+
248
+
#ifdef CONFIG_PAGE_POOL_STATS
249
+
free_percpu(pool->recycle_stats);
250
+
#endif
251
+
}
252
+
243
253
/**
244
254
* page_pool_create() - create a page pool.
245
255
* @params: parameters, see struct page_pool_params
···
266
252
return ERR_PTR(-ENOMEM);
267
253
268
254
err = page_pool_init(pool, params);
269
-
if (err < 0) {
270
-
pr_warn("%s() gave up with errno %d\n", __func__, err);
271
-
kfree(pool);
272
-
return ERR_PTR(err);
273
-
}
255
+
if (err < 0)
256
+
goto err_free;
257
+
258
+
err = page_pool_list(pool);
259
+
if (err)
260
+
goto err_uninit;
274
261
275
262
return pool;
263
+
264
+
err_uninit:
265
+
page_pool_uninit(pool);
266
+
err_free:
267
+
pr_warn("%s() gave up with errno %d\n", __func__, err);
268
+
kfree(pool);
269
+
return ERR_PTR(err);
276
270
}
277
271
EXPORT_SYMBOL(page_pool_create);
278
272
···
529
507
*/
530
508
#define _distance(a, b) (s32)((a) - (b))
531
509
532
-
static s32 page_pool_inflight(struct page_pool *pool)
510
+
s32 page_pool_inflight(const struct page_pool *pool, bool strict)
533
511
{
534
512
u32 release_cnt = atomic_read(&pool->pages_state_release_cnt);
535
513
u32 hold_cnt = READ_ONCE(pool->pages_state_hold_cnt);
···
537
515
538
516
inflight = _distance(hold_cnt, release_cnt);
539
517
540
-
trace_page_pool_release(pool, inflight, hold_cnt, release_cnt);
541
-
WARN(inflight < 0, "Negative(%d) inflight packet-pages", inflight);
518
+
if (strict) {
519
+
trace_page_pool_release(pool, inflight, hold_cnt, release_cnt);
520
+
WARN(inflight < 0, "Negative(%d) inflight packet-pages",
521
+
inflight);
522
+
} else {
523
+
inflight = max(0, inflight);
524
+
}
542
525
543
526
return inflight;
544
527
}
···
848
821
if (pool->disconnect)
849
822
pool->disconnect(pool);
850
823
851
-
ptr_ring_cleanup(&pool->ring, NULL);
852
-
853
-
if (pool->p.flags & PP_FLAG_DMA_MAP)
854
-
put_device(pool->p.dev);
855
-
856
-
#ifdef CONFIG_PAGE_POOL_STATS
857
-
free_percpu(pool->recycle_stats);
858
-
#endif
824
+
page_pool_unlist(pool);
825
+
page_pool_uninit(pool);
859
826
kfree(pool);
860
827
}
861
828
···
886
865
int inflight;
887
866
888
867
page_pool_scrub(pool);
889
-
inflight = page_pool_inflight(pool);
868
+
inflight = page_pool_inflight(pool, true);
890
869
if (!inflight)
891
870
__page_pool_destroy(pool);
892
871
···
897
876
{
898
877
struct delayed_work *dwq = to_delayed_work(wq);
899
878
struct page_pool *pool = container_of(dwq, typeof(*pool), release_dw);
879
+
void *netdev;
900
880
int inflight;
901
881
902
882
inflight = page_pool_release(pool);
903
883
if (!inflight)
904
884
return;
905
885
906
-
/* Periodic warning */
907
-
if (time_after_eq(jiffies, pool->defer_warn)) {
886
+
/* Periodic warning for page pools the user can't see */
887
+
netdev = READ_ONCE(pool->slow.netdev);
888
+
if (time_after_eq(jiffies, pool->defer_warn) &&
889
+
(!netdev || netdev == NET_PTR_POISON)) {
908
890
int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
909
891
910
-
pr_warn("%s() stalled pool shutdown %d inflight %d sec\n",
911
-
__func__, inflight, sec);
892
+
pr_warn("%s() stalled pool shutdown: id %u, %d inflight %d sec\n",
893
+
__func__, pool->user.id, inflight, sec);
912
894
pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
913
895
}
914
896
···
956
932
if (!page_pool_release(pool))
957
933
return;
958
934
935
+
page_pool_detached(pool);
959
936
pool->defer_start = jiffies;
960
937
pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
961
938
+12
net/core/page_pool_priv.h
+12
net/core/page_pool_priv.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0 */
2
+
3
+
#ifndef __PAGE_POOL_PRIV_H
4
+
#define __PAGE_POOL_PRIV_H
5
+
6
+
s32 page_pool_inflight(const struct page_pool *pool, bool strict);
7
+
8
+
int page_pool_list(struct page_pool *pool);
9
+
void page_pool_detached(struct page_pool *pool);
10
+
void page_pool_unlist(struct page_pool *pool);
11
+
12
+
#endif
+408
net/core/page_pool_user.c
+408
net/core/page_pool_user.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
#include <linux/mutex.h>
4
+
#include <linux/netdevice.h>
5
+
#include <linux/xarray.h>
6
+
#include <net/net_debug.h>
7
+
#include <net/page_pool/types.h>
8
+
#include <net/page_pool/helpers.h>
9
+
#include <net/sock.h>
10
+
11
+
#include "page_pool_priv.h"
12
+
#include "netdev-genl-gen.h"
13
+
14
+
static DEFINE_XARRAY_FLAGS(page_pools, XA_FLAGS_ALLOC1);
15
+
/* Protects: page_pools, netdevice->page_pools, pool->slow.netdev, pool->user.
16
+
* Ordering: inside rtnl_lock
17
+
*/
18
+
static DEFINE_MUTEX(page_pools_lock);
19
+
20
+
/* Page pools are only reachable from user space (via netlink) if they are
21
+
* linked to a netdev at creation time. Following page pool "visibility"
22
+
* states are possible:
23
+
* - normal
24
+
* - user.list: linked to real netdev, netdev: real netdev
25
+
* - orphaned - real netdev has disappeared
26
+
* - user.list: linked to lo, netdev: lo
27
+
* - invisible - either (a) created without netdev linking, (b) unlisted due
28
+
* to error, or (c) the entire namespace which owned this pool disappeared
29
+
* - user.list: unhashed, netdev: unknown
30
+
*/
31
+
32
+
typedef int (*pp_nl_fill_cb)(struct sk_buff *rsp, const struct page_pool *pool,
33
+
const struct genl_info *info);
34
+
35
+
static int
36
+
netdev_nl_page_pool_get_do(struct genl_info *info, u32 id, pp_nl_fill_cb fill)
37
+
{
38
+
struct page_pool *pool;
39
+
struct sk_buff *rsp;
40
+
int err;
41
+
42
+
mutex_lock(&page_pools_lock);
43
+
pool = xa_load(&page_pools, id);
44
+
if (!pool || hlist_unhashed(&pool->user.list) ||
45
+
!net_eq(dev_net(pool->slow.netdev), genl_info_net(info))) {
46
+
err = -ENOENT;
47
+
goto err_unlock;
48
+
}
49
+
50
+
rsp = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL);
51
+
if (!rsp) {
52
+
err = -ENOMEM;
53
+
goto err_unlock;
54
+
}
55
+
56
+
err = fill(rsp, pool, info);
57
+
if (err)
58
+
goto err_free_msg;
59
+
60
+
mutex_unlock(&page_pools_lock);
61
+
62
+
return genlmsg_reply(rsp, info);
63
+
64
+
err_free_msg:
65
+
nlmsg_free(rsp);
66
+
err_unlock:
67
+
mutex_unlock(&page_pools_lock);
68
+
return err;
69
+
}
70
+
71
+
struct page_pool_dump_cb {
72
+
unsigned long ifindex;
73
+
u32 pp_id;
74
+
};
75
+
76
+
static int
77
+
netdev_nl_page_pool_get_dump(struct sk_buff *skb, struct netlink_callback *cb,
78
+
pp_nl_fill_cb fill)
79
+
{
80
+
struct page_pool_dump_cb *state = (void *)cb->ctx;
81
+
const struct genl_info *info = genl_info_dump(cb);
82
+
struct net *net = sock_net(skb->sk);
83
+
struct net_device *netdev;
84
+
struct page_pool *pool;
85
+
int err = 0;
86
+
87
+
rtnl_lock();
88
+
mutex_lock(&page_pools_lock);
89
+
for_each_netdev_dump(net, netdev, state->ifindex) {
90
+
hlist_for_each_entry(pool, &netdev->page_pools, user.list) {
91
+
if (state->pp_id && state->pp_id < pool->user.id)
92
+
continue;
93
+
94
+
state->pp_id = pool->user.id;
95
+
err = fill(skb, pool, info);
96
+
if (err)
97
+
break;
98
+
}
99
+
100
+
state->pp_id = 0;
101
+
}
102
+
mutex_unlock(&page_pools_lock);
103
+
rtnl_unlock();
104
+
105
+
if (skb->len && err == -EMSGSIZE)
106
+
return skb->len;
107
+
return err;
108
+
}
109
+
110
+
static int
111
+
page_pool_nl_stats_fill(struct sk_buff *rsp, const struct page_pool *pool,
112
+
const struct genl_info *info)
113
+
{
114
+
#ifdef CONFIG_PAGE_POOL_STATS
115
+
struct page_pool_stats stats = {};
116
+
struct nlattr *nest;
117
+
void *hdr;
118
+
119
+
if (!page_pool_get_stats(pool, &stats))
120
+
return 0;
121
+
122
+
hdr = genlmsg_iput(rsp, info);
123
+
if (!hdr)
124
+
return -EMSGSIZE;
125
+
126
+
nest = nla_nest_start(rsp, NETDEV_A_PAGE_POOL_STATS_INFO);
127
+
128
+
if (nla_put_uint(rsp, NETDEV_A_PAGE_POOL_ID, pool->user.id) ||
129
+
(pool->slow.netdev->ifindex != LOOPBACK_IFINDEX &&
130
+
nla_put_u32(rsp, NETDEV_A_PAGE_POOL_IFINDEX,
131
+
pool->slow.netdev->ifindex)))
132
+
goto err_cancel_nest;
133
+
134
+
nla_nest_end(rsp, nest);
135
+
136
+
if (nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST,
137
+
stats.alloc_stats.fast) ||
138
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW,
139
+
stats.alloc_stats.slow) ||
140
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER,
141
+
stats.alloc_stats.slow_high_order) ||
142
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY,
143
+
stats.alloc_stats.empty) ||
144
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL,
145
+
stats.alloc_stats.refill) ||
146
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE,
147
+
stats.alloc_stats.waive) ||
148
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED,
149
+
stats.recycle_stats.cached) ||
150
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL,
151
+
stats.recycle_stats.cache_full) ||
152
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING,
153
+
stats.recycle_stats.ring) ||
154
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL,
155
+
stats.recycle_stats.ring_full) ||
156
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT,
157
+
stats.recycle_stats.released_refcnt))
158
+
goto err_cancel_msg;
159
+
160
+
genlmsg_end(rsp, hdr);
161
+
162
+
return 0;
163
+
err_cancel_nest:
164
+
nla_nest_cancel(rsp, nest);
165
+
err_cancel_msg:
166
+
genlmsg_cancel(rsp, hdr);
167
+
return -EMSGSIZE;
168
+
#else
169
+
GENL_SET_ERR_MSG(info, "kernel built without CONFIG_PAGE_POOL_STATS");
170
+
return -EOPNOTSUPP;
171
+
#endif
172
+
}
173
+
174
+
int netdev_nl_page_pool_stats_get_doit(struct sk_buff *skb,
175
+
struct genl_info *info)
176
+
{
177
+
struct nlattr *tb[ARRAY_SIZE(netdev_page_pool_info_nl_policy)];
178
+
struct nlattr *nest;
179
+
int err;
180
+
u32 id;
181
+
182
+
if (GENL_REQ_ATTR_CHECK(info, NETDEV_A_PAGE_POOL_STATS_INFO))
183
+
return -EINVAL;
184
+
185
+
nest = info->attrs[NETDEV_A_PAGE_POOL_STATS_INFO];
186
+
err = nla_parse_nested(tb, ARRAY_SIZE(tb) - 1, nest,
187
+
netdev_page_pool_info_nl_policy,
188
+
info->extack);
189
+
if (err)
190
+
return err;
191
+
192
+
if (NL_REQ_ATTR_CHECK(info->extack, nest, tb, NETDEV_A_PAGE_POOL_ID))
193
+
return -EINVAL;
194
+
if (tb[NETDEV_A_PAGE_POOL_IFINDEX]) {
195
+
NL_SET_ERR_MSG_ATTR(info->extack,
196
+
tb[NETDEV_A_PAGE_POOL_IFINDEX],
197
+
"selecting by ifindex not supported");
198
+
return -EINVAL;
199
+
}
200
+
201
+
id = nla_get_uint(tb[NETDEV_A_PAGE_POOL_ID]);
202
+
203
+
return netdev_nl_page_pool_get_do(info, id, page_pool_nl_stats_fill);
204
+
}
205
+
206
+
int netdev_nl_page_pool_stats_get_dumpit(struct sk_buff *skb,
207
+
struct netlink_callback *cb)
208
+
{
209
+
return netdev_nl_page_pool_get_dump(skb, cb, page_pool_nl_stats_fill);
210
+
}
211
+
212
+
static int
213
+
page_pool_nl_fill(struct sk_buff *rsp, const struct page_pool *pool,
214
+
const struct genl_info *info)
215
+
{
216
+
size_t inflight, refsz;
217
+
void *hdr;
218
+
219
+
hdr = genlmsg_iput(rsp, info);
220
+
if (!hdr)
221
+
return -EMSGSIZE;
222
+
223
+
if (nla_put_uint(rsp, NETDEV_A_PAGE_POOL_ID, pool->user.id))
224
+
goto err_cancel;
225
+
226
+
if (pool->slow.netdev->ifindex != LOOPBACK_IFINDEX &&
227
+
nla_put_u32(rsp, NETDEV_A_PAGE_POOL_IFINDEX,
228
+
pool->slow.netdev->ifindex))
229
+
goto err_cancel;
230
+
if (pool->user.napi_id &&
231
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_NAPI_ID, pool->user.napi_id))
232
+
goto err_cancel;
233
+
234
+
inflight = page_pool_inflight(pool, false);
235
+
refsz = PAGE_SIZE << pool->p.order;
236
+
if (nla_put_uint(rsp, NETDEV_A_PAGE_POOL_INFLIGHT, inflight) ||
237
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_INFLIGHT_MEM,
238
+
inflight * refsz))
239
+
goto err_cancel;
240
+
if (pool->user.detach_time &&
241
+
nla_put_uint(rsp, NETDEV_A_PAGE_POOL_DETACH_TIME,
242
+
pool->user.detach_time))
243
+
goto err_cancel;
244
+
245
+
genlmsg_end(rsp, hdr);
246
+
247
+
return 0;
248
+
err_cancel:
249
+
genlmsg_cancel(rsp, hdr);
250
+
return -EMSGSIZE;
251
+
}
252
+
253
+
static void netdev_nl_page_pool_event(const struct page_pool *pool, u32 cmd)
254
+
{
255
+
struct genl_info info;
256
+
struct sk_buff *ntf;
257
+
struct net *net;
258
+
259
+
lockdep_assert_held(&page_pools_lock);
260
+
261
+
/* 'invisible' page pools don't matter */
262
+
if (hlist_unhashed(&pool->user.list))
263
+
return;
264
+
net = dev_net(pool->slow.netdev);
265
+
266
+
if (!genl_has_listeners(&netdev_nl_family, net, NETDEV_NLGRP_PAGE_POOL))
267
+
return;
268
+
269
+
genl_info_init_ntf(&info, &netdev_nl_family, cmd);
270
+
271
+
ntf = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL);
272
+
if (!ntf)
273
+
return;
274
+
275
+
if (page_pool_nl_fill(ntf, pool, &info)) {
276
+
nlmsg_free(ntf);
277
+
return;
278
+
}
279
+
280
+
genlmsg_multicast_netns(&netdev_nl_family, net, ntf,
281
+
0, NETDEV_NLGRP_PAGE_POOL, GFP_KERNEL);
282
+
}
283
+
284
+
int netdev_nl_page_pool_get_doit(struct sk_buff *skb, struct genl_info *info)
285
+
{
286
+
u32 id;
287
+
288
+
if (GENL_REQ_ATTR_CHECK(info, NETDEV_A_PAGE_POOL_ID))
289
+
return -EINVAL;
290
+
291
+
id = nla_get_uint(info->attrs[NETDEV_A_PAGE_POOL_ID]);
292
+
293
+
return netdev_nl_page_pool_get_do(info, id, page_pool_nl_fill);
294
+
}
295
+
296
+
int netdev_nl_page_pool_get_dumpit(struct sk_buff *skb,
297
+
struct netlink_callback *cb)
298
+
{
299
+
return netdev_nl_page_pool_get_dump(skb, cb, page_pool_nl_fill);
300
+
}
301
+
302
+
int page_pool_list(struct page_pool *pool)
303
+
{
304
+
static u32 id_alloc_next;
305
+
int err;
306
+
307
+
mutex_lock(&page_pools_lock);
308
+
err = xa_alloc_cyclic(&page_pools, &pool->user.id, pool, xa_limit_32b,
309
+
&id_alloc_next, GFP_KERNEL);
310
+
if (err < 0)
311
+
goto err_unlock;
312
+
313
+
if (pool->slow.netdev) {
314
+
hlist_add_head(&pool->user.list,
315
+
&pool->slow.netdev->page_pools);
316
+
pool->user.napi_id = pool->p.napi ? pool->p.napi->napi_id : 0;
317
+
318
+
netdev_nl_page_pool_event(pool, NETDEV_CMD_PAGE_POOL_ADD_NTF);
319
+
}
320
+
321
+
mutex_unlock(&page_pools_lock);
322
+
return 0;
323
+
324
+
err_unlock:
325
+
mutex_unlock(&page_pools_lock);
326
+
return err;
327
+
}
328
+
329
+
void page_pool_detached(struct page_pool *pool)
330
+
{
331
+
mutex_lock(&page_pools_lock);
332
+
pool->user.detach_time = ktime_get_boottime_seconds();
333
+
netdev_nl_page_pool_event(pool, NETDEV_CMD_PAGE_POOL_CHANGE_NTF);
334
+
mutex_unlock(&page_pools_lock);
335
+
}
336
+
337
+
void page_pool_unlist(struct page_pool *pool)
338
+
{
339
+
mutex_lock(&page_pools_lock);
340
+
netdev_nl_page_pool_event(pool, NETDEV_CMD_PAGE_POOL_DEL_NTF);
341
+
xa_erase(&page_pools, pool->user.id);
342
+
hlist_del(&pool->user.list);
343
+
mutex_unlock(&page_pools_lock);
344
+
}
345
+
346
+
static void page_pool_unreg_netdev_wipe(struct net_device *netdev)
347
+
{
348
+
struct page_pool *pool;
349
+
struct hlist_node *n;
350
+
351
+
mutex_lock(&page_pools_lock);
352
+
hlist_for_each_entry_safe(pool, n, &netdev->page_pools, user.list) {
353
+
hlist_del_init(&pool->user.list);
354
+
pool->slow.netdev = NET_PTR_POISON;
355
+
}
356
+
mutex_unlock(&page_pools_lock);
357
+
}
358
+
359
+
static void page_pool_unreg_netdev(struct net_device *netdev)
360
+
{
361
+
struct page_pool *pool, *last;
362
+
struct net_device *lo;
363
+
364
+
lo = dev_net(netdev)->loopback_dev;
365
+
366
+
mutex_lock(&page_pools_lock);
367
+
last = NULL;
368
+
hlist_for_each_entry(pool, &netdev->page_pools, user.list) {
369
+
pool->slow.netdev = lo;
370
+
netdev_nl_page_pool_event(pool,
371
+
NETDEV_CMD_PAGE_POOL_CHANGE_NTF);
372
+
last = pool;
373
+
}
374
+
if (last)
375
+
hlist_splice_init(&netdev->page_pools, &last->user.list,
376
+
&lo->page_pools);
377
+
mutex_unlock(&page_pools_lock);
378
+
}
379
+
380
+
static int
381
+
page_pool_netdevice_event(struct notifier_block *nb,
382
+
unsigned long event, void *ptr)
383
+
{
384
+
struct net_device *netdev = netdev_notifier_info_to_dev(ptr);
385
+
386
+
if (event != NETDEV_UNREGISTER)
387
+
return NOTIFY_DONE;
388
+
389
+
if (hlist_empty(&netdev->page_pools))
390
+
return NOTIFY_OK;
391
+
392
+
if (netdev->ifindex != LOOPBACK_IFINDEX)
393
+
page_pool_unreg_netdev(netdev);
394
+
else
395
+
page_pool_unreg_netdev_wipe(netdev);
396
+
return NOTIFY_OK;
397
+
}
398
+
399
+
static struct notifier_block page_pool_netdevice_nb = {
400
+
.notifier_call = page_pool_netdevice_event,
401
+
};
402
+
403
+
static int __init page_pool_user_init(void)
404
+
{
405
+
return register_netdevice_notifier(&page_pool_netdevice_nb);
406
+
}
407
+
408
+
subsys_initcall(page_pool_user_init);
+36
tools/include/uapi/linux/netdev.h
+36
tools/include/uapi/linux/netdev.h
···
65
65
};
66
66
67
67
enum {
68
+
NETDEV_A_PAGE_POOL_ID = 1,
69
+
NETDEV_A_PAGE_POOL_IFINDEX,
70
+
NETDEV_A_PAGE_POOL_NAPI_ID,
71
+
NETDEV_A_PAGE_POOL_INFLIGHT,
72
+
NETDEV_A_PAGE_POOL_INFLIGHT_MEM,
73
+
NETDEV_A_PAGE_POOL_DETACH_TIME,
74
+
75
+
__NETDEV_A_PAGE_POOL_MAX,
76
+
NETDEV_A_PAGE_POOL_MAX = (__NETDEV_A_PAGE_POOL_MAX - 1)
77
+
};
78
+
79
+
enum {
80
+
NETDEV_A_PAGE_POOL_STATS_INFO = 1,
81
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST = 8,
82
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW,
83
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER,
84
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY,
85
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL,
86
+
NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE,
87
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED,
88
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL,
89
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING,
90
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL,
91
+
NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT,
92
+
93
+
__NETDEV_A_PAGE_POOL_STATS_MAX,
94
+
NETDEV_A_PAGE_POOL_STATS_MAX = (__NETDEV_A_PAGE_POOL_STATS_MAX - 1)
95
+
};
96
+
97
+
enum {
68
98
NETDEV_CMD_DEV_GET = 1,
69
99
NETDEV_CMD_DEV_ADD_NTF,
70
100
NETDEV_CMD_DEV_DEL_NTF,
71
101
NETDEV_CMD_DEV_CHANGE_NTF,
102
+
NETDEV_CMD_PAGE_POOL_GET,
103
+
NETDEV_CMD_PAGE_POOL_ADD_NTF,
104
+
NETDEV_CMD_PAGE_POOL_DEL_NTF,
105
+
NETDEV_CMD_PAGE_POOL_CHANGE_NTF,
106
+
NETDEV_CMD_PAGE_POOL_STATS_GET,
72
107
73
108
__NETDEV_CMD_MAX,
74
109
NETDEV_CMD_MAX = (__NETDEV_CMD_MAX - 1)
75
110
};
76
111
77
112
#define NETDEV_MCGRP_MGMT "mgmt"
113
+
#define NETDEV_MCGRP_PAGE_POOL "page-pool"
78
114
79
115
#endif /* _UAPI_LINUX_NETDEV_H */
+419
tools/net/ynl/generated/netdev-user.c
+419
tools/net/ynl/generated/netdev-user.c
···
18
18
[NETDEV_CMD_DEV_ADD_NTF] = "dev-add-ntf",
19
19
[NETDEV_CMD_DEV_DEL_NTF] = "dev-del-ntf",
20
20
[NETDEV_CMD_DEV_CHANGE_NTF] = "dev-change-ntf",
21
+
[NETDEV_CMD_PAGE_POOL_GET] = "page-pool-get",
22
+
[NETDEV_CMD_PAGE_POOL_ADD_NTF] = "page-pool-add-ntf",
23
+
[NETDEV_CMD_PAGE_POOL_DEL_NTF] = "page-pool-del-ntf",
24
+
[NETDEV_CMD_PAGE_POOL_CHANGE_NTF] = "page-pool-change-ntf",
25
+
[NETDEV_CMD_PAGE_POOL_STATS_GET] = "page-pool-stats-get",
21
26
};
22
27
23
28
const char *netdev_op_str(int op)
···
64
59
}
65
60
66
61
/* Policies */
62
+
struct ynl_policy_attr netdev_page_pool_info_policy[NETDEV_A_PAGE_POOL_MAX + 1] = {
63
+
[NETDEV_A_PAGE_POOL_ID] = { .name = "id", .type = YNL_PT_UINT, },
64
+
[NETDEV_A_PAGE_POOL_IFINDEX] = { .name = "ifindex", .type = YNL_PT_U32, },
65
+
};
66
+
67
+
struct ynl_policy_nest netdev_page_pool_info_nest = {
68
+
.max_attr = NETDEV_A_PAGE_POOL_MAX,
69
+
.table = netdev_page_pool_info_policy,
70
+
};
71
+
67
72
struct ynl_policy_attr netdev_dev_policy[NETDEV_A_DEV_MAX + 1] = {
68
73
[NETDEV_A_DEV_IFINDEX] = { .name = "ifindex", .type = YNL_PT_U32, },
69
74
[NETDEV_A_DEV_PAD] = { .name = "pad", .type = YNL_PT_IGNORE, },
···
87
72
.table = netdev_dev_policy,
88
73
};
89
74
75
+
struct ynl_policy_attr netdev_page_pool_policy[NETDEV_A_PAGE_POOL_MAX + 1] = {
76
+
[NETDEV_A_PAGE_POOL_ID] = { .name = "id", .type = YNL_PT_UINT, },
77
+
[NETDEV_A_PAGE_POOL_IFINDEX] = { .name = "ifindex", .type = YNL_PT_U32, },
78
+
[NETDEV_A_PAGE_POOL_NAPI_ID] = { .name = "napi-id", .type = YNL_PT_UINT, },
79
+
[NETDEV_A_PAGE_POOL_INFLIGHT] = { .name = "inflight", .type = YNL_PT_UINT, },
80
+
[NETDEV_A_PAGE_POOL_INFLIGHT_MEM] = { .name = "inflight-mem", .type = YNL_PT_UINT, },
81
+
[NETDEV_A_PAGE_POOL_DETACH_TIME] = { .name = "detach-time", .type = YNL_PT_UINT, },
82
+
};
83
+
84
+
struct ynl_policy_nest netdev_page_pool_nest = {
85
+
.max_attr = NETDEV_A_PAGE_POOL_MAX,
86
+
.table = netdev_page_pool_policy,
87
+
};
88
+
89
+
struct ynl_policy_attr netdev_page_pool_stats_policy[NETDEV_A_PAGE_POOL_STATS_MAX + 1] = {
90
+
[NETDEV_A_PAGE_POOL_STATS_INFO] = { .name = "info", .type = YNL_PT_NEST, .nest = &netdev_page_pool_info_nest, },
91
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST] = { .name = "alloc-fast", .type = YNL_PT_UINT, },
92
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW] = { .name = "alloc-slow", .type = YNL_PT_UINT, },
93
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER] = { .name = "alloc-slow-high-order", .type = YNL_PT_UINT, },
94
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY] = { .name = "alloc-empty", .type = YNL_PT_UINT, },
95
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL] = { .name = "alloc-refill", .type = YNL_PT_UINT, },
96
+
[NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE] = { .name = "alloc-waive", .type = YNL_PT_UINT, },
97
+
[NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED] = { .name = "recycle-cached", .type = YNL_PT_UINT, },
98
+
[NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL] = { .name = "recycle-cache-full", .type = YNL_PT_UINT, },
99
+
[NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING] = { .name = "recycle-ring", .type = YNL_PT_UINT, },
100
+
[NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL] = { .name = "recycle-ring-full", .type = YNL_PT_UINT, },
101
+
[NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT] = { .name = "recycle-released-refcnt", .type = YNL_PT_UINT, },
102
+
};
103
+
104
+
struct ynl_policy_nest netdev_page_pool_stats_nest = {
105
+
.max_attr = NETDEV_A_PAGE_POOL_STATS_MAX,
106
+
.table = netdev_page_pool_stats_policy,
107
+
};
108
+
90
109
/* Common nested types */
110
+
void netdev_page_pool_info_free(struct netdev_page_pool_info *obj)
111
+
{
112
+
}
113
+
114
+
int netdev_page_pool_info_put(struct nlmsghdr *nlh, unsigned int attr_type,
115
+
struct netdev_page_pool_info *obj)
116
+
{
117
+
struct nlattr *nest;
118
+
119
+
nest = mnl_attr_nest_start(nlh, attr_type);
120
+
if (obj->_present.id)
121
+
mnl_attr_put_uint(nlh, NETDEV_A_PAGE_POOL_ID, obj->id);
122
+
if (obj->_present.ifindex)
123
+
mnl_attr_put_u32(nlh, NETDEV_A_PAGE_POOL_IFINDEX, obj->ifindex);
124
+
mnl_attr_nest_end(nlh, nest);
125
+
126
+
return 0;
127
+
}
128
+
129
+
int netdev_page_pool_info_parse(struct ynl_parse_arg *yarg,
130
+
const struct nlattr *nested)
131
+
{
132
+
struct netdev_page_pool_info *dst = yarg->data;
133
+
const struct nlattr *attr;
134
+
135
+
mnl_attr_for_each_nested(attr, nested) {
136
+
unsigned int type = mnl_attr_get_type(attr);
137
+
138
+
if (type == NETDEV_A_PAGE_POOL_ID) {
139
+
if (ynl_attr_validate(yarg, attr))
140
+
return MNL_CB_ERROR;
141
+
dst->_present.id = 1;
142
+
dst->id = mnl_attr_get_uint(attr);
143
+
} else if (type == NETDEV_A_PAGE_POOL_IFINDEX) {
144
+
if (ynl_attr_validate(yarg, attr))
145
+
return MNL_CB_ERROR;
146
+
dst->_present.ifindex = 1;
147
+
dst->ifindex = mnl_attr_get_u32(attr);
148
+
}
149
+
}
150
+
151
+
return 0;
152
+
}
153
+
91
154
/* ============== NETDEV_CMD_DEV_GET ============== */
92
155
/* NETDEV_CMD_DEV_GET - do */
93
156
void netdev_dev_get_req_free(struct netdev_dev_get_req *req)
···
290
197
free(rsp);
291
198
}
292
199
200
+
/* ============== NETDEV_CMD_PAGE_POOL_GET ============== */
201
+
/* NETDEV_CMD_PAGE_POOL_GET - do */
202
+
void netdev_page_pool_get_req_free(struct netdev_page_pool_get_req *req)
203
+
{
204
+
free(req);
205
+
}
206
+
207
+
void netdev_page_pool_get_rsp_free(struct netdev_page_pool_get_rsp *rsp)
208
+
{
209
+
free(rsp);
210
+
}
211
+
212
+
int netdev_page_pool_get_rsp_parse(const struct nlmsghdr *nlh, void *data)
213
+
{
214
+
struct netdev_page_pool_get_rsp *dst;
215
+
struct ynl_parse_arg *yarg = data;
216
+
const struct nlattr *attr;
217
+
218
+
dst = yarg->data;
219
+
220
+
mnl_attr_for_each(attr, nlh, sizeof(struct genlmsghdr)) {
221
+
unsigned int type = mnl_attr_get_type(attr);
222
+
223
+
if (type == NETDEV_A_PAGE_POOL_ID) {
224
+
if (ynl_attr_validate(yarg, attr))
225
+
return MNL_CB_ERROR;
226
+
dst->_present.id = 1;
227
+
dst->id = mnl_attr_get_uint(attr);
228
+
} else if (type == NETDEV_A_PAGE_POOL_IFINDEX) {
229
+
if (ynl_attr_validate(yarg, attr))
230
+
return MNL_CB_ERROR;
231
+
dst->_present.ifindex = 1;
232
+
dst->ifindex = mnl_attr_get_u32(attr);
233
+
} else if (type == NETDEV_A_PAGE_POOL_NAPI_ID) {
234
+
if (ynl_attr_validate(yarg, attr))
235
+
return MNL_CB_ERROR;
236
+
dst->_present.napi_id = 1;
237
+
dst->napi_id = mnl_attr_get_uint(attr);
238
+
} else if (type == NETDEV_A_PAGE_POOL_INFLIGHT) {
239
+
if (ynl_attr_validate(yarg, attr))
240
+
return MNL_CB_ERROR;
241
+
dst->_present.inflight = 1;
242
+
dst->inflight = mnl_attr_get_uint(attr);
243
+
} else if (type == NETDEV_A_PAGE_POOL_INFLIGHT_MEM) {
244
+
if (ynl_attr_validate(yarg, attr))
245
+
return MNL_CB_ERROR;
246
+
dst->_present.inflight_mem = 1;
247
+
dst->inflight_mem = mnl_attr_get_uint(attr);
248
+
} else if (type == NETDEV_A_PAGE_POOL_DETACH_TIME) {
249
+
if (ynl_attr_validate(yarg, attr))
250
+
return MNL_CB_ERROR;
251
+
dst->_present.detach_time = 1;
252
+
dst->detach_time = mnl_attr_get_uint(attr);
253
+
}
254
+
}
255
+
256
+
return MNL_CB_OK;
257
+
}
258
+
259
+
struct netdev_page_pool_get_rsp *
260
+
netdev_page_pool_get(struct ynl_sock *ys, struct netdev_page_pool_get_req *req)
261
+
{
262
+
struct ynl_req_state yrs = { .yarg = { .ys = ys, }, };
263
+
struct netdev_page_pool_get_rsp *rsp;
264
+
struct nlmsghdr *nlh;
265
+
int err;
266
+
267
+
nlh = ynl_gemsg_start_req(ys, ys->family_id, NETDEV_CMD_PAGE_POOL_GET, 1);
268
+
ys->req_policy = &netdev_page_pool_nest;
269
+
yrs.yarg.rsp_policy = &netdev_page_pool_nest;
270
+
271
+
if (req->_present.id)
272
+
mnl_attr_put_uint(nlh, NETDEV_A_PAGE_POOL_ID, req->id);
273
+
274
+
rsp = calloc(1, sizeof(*rsp));
275
+
yrs.yarg.data = rsp;
276
+
yrs.cb = netdev_page_pool_get_rsp_parse;
277
+
yrs.rsp_cmd = NETDEV_CMD_PAGE_POOL_GET;
278
+
279
+
err = ynl_exec(ys, nlh, &yrs);
280
+
if (err < 0)
281
+
goto err_free;
282
+
283
+
return rsp;
284
+
285
+
err_free:
286
+
netdev_page_pool_get_rsp_free(rsp);
287
+
return NULL;
288
+
}
289
+
290
+
/* NETDEV_CMD_PAGE_POOL_GET - dump */
291
+
void netdev_page_pool_get_list_free(struct netdev_page_pool_get_list *rsp)
292
+
{
293
+
struct netdev_page_pool_get_list *next = rsp;
294
+
295
+
while ((void *)next != YNL_LIST_END) {
296
+
rsp = next;
297
+
next = rsp->next;
298
+
299
+
free(rsp);
300
+
}
301
+
}
302
+
303
+
struct netdev_page_pool_get_list *
304
+
netdev_page_pool_get_dump(struct ynl_sock *ys)
305
+
{
306
+
struct ynl_dump_state yds = {};
307
+
struct nlmsghdr *nlh;
308
+
int err;
309
+
310
+
yds.ys = ys;
311
+
yds.alloc_sz = sizeof(struct netdev_page_pool_get_list);
312
+
yds.cb = netdev_page_pool_get_rsp_parse;
313
+
yds.rsp_cmd = NETDEV_CMD_PAGE_POOL_GET;
314
+
yds.rsp_policy = &netdev_page_pool_nest;
315
+
316
+
nlh = ynl_gemsg_start_dump(ys, ys->family_id, NETDEV_CMD_PAGE_POOL_GET, 1);
317
+
318
+
err = ynl_exec_dump(ys, nlh, &yds);
319
+
if (err < 0)
320
+
goto free_list;
321
+
322
+
return yds.first;
323
+
324
+
free_list:
325
+
netdev_page_pool_get_list_free(yds.first);
326
+
return NULL;
327
+
}
328
+
329
+
/* NETDEV_CMD_PAGE_POOL_GET - notify */
330
+
void netdev_page_pool_get_ntf_free(struct netdev_page_pool_get_ntf *rsp)
331
+
{
332
+
free(rsp);
333
+
}
334
+
335
+
/* ============== NETDEV_CMD_PAGE_POOL_STATS_GET ============== */
336
+
/* NETDEV_CMD_PAGE_POOL_STATS_GET - do */
337
+
void
338
+
netdev_page_pool_stats_get_req_free(struct netdev_page_pool_stats_get_req *req)
339
+
{
340
+
netdev_page_pool_info_free(&req->info);
341
+
free(req);
342
+
}
343
+
344
+
void
345
+
netdev_page_pool_stats_get_rsp_free(struct netdev_page_pool_stats_get_rsp *rsp)
346
+
{
347
+
netdev_page_pool_info_free(&rsp->info);
348
+
free(rsp);
349
+
}
350
+
351
+
int netdev_page_pool_stats_get_rsp_parse(const struct nlmsghdr *nlh,
352
+
void *data)
353
+
{
354
+
struct netdev_page_pool_stats_get_rsp *dst;
355
+
struct ynl_parse_arg *yarg = data;
356
+
const struct nlattr *attr;
357
+
struct ynl_parse_arg parg;
358
+
359
+
dst = yarg->data;
360
+
parg.ys = yarg->ys;
361
+
362
+
mnl_attr_for_each(attr, nlh, sizeof(struct genlmsghdr)) {
363
+
unsigned int type = mnl_attr_get_type(attr);
364
+
365
+
if (type == NETDEV_A_PAGE_POOL_STATS_INFO) {
366
+
if (ynl_attr_validate(yarg, attr))
367
+
return MNL_CB_ERROR;
368
+
dst->_present.info = 1;
369
+
370
+
parg.rsp_policy = &netdev_page_pool_info_nest;
371
+
parg.data = &dst->info;
372
+
if (netdev_page_pool_info_parse(&parg, attr))
373
+
return MNL_CB_ERROR;
374
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST) {
375
+
if (ynl_attr_validate(yarg, attr))
376
+
return MNL_CB_ERROR;
377
+
dst->_present.alloc_fast = 1;
378
+
dst->alloc_fast = mnl_attr_get_uint(attr);
379
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW) {
380
+
if (ynl_attr_validate(yarg, attr))
381
+
return MNL_CB_ERROR;
382
+
dst->_present.alloc_slow = 1;
383
+
dst->alloc_slow = mnl_attr_get_uint(attr);
384
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER) {
385
+
if (ynl_attr_validate(yarg, attr))
386
+
return MNL_CB_ERROR;
387
+
dst->_present.alloc_slow_high_order = 1;
388
+
dst->alloc_slow_high_order = mnl_attr_get_uint(attr);
389
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY) {
390
+
if (ynl_attr_validate(yarg, attr))
391
+
return MNL_CB_ERROR;
392
+
dst->_present.alloc_empty = 1;
393
+
dst->alloc_empty = mnl_attr_get_uint(attr);
394
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL) {
395
+
if (ynl_attr_validate(yarg, attr))
396
+
return MNL_CB_ERROR;
397
+
dst->_present.alloc_refill = 1;
398
+
dst->alloc_refill = mnl_attr_get_uint(attr);
399
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE) {
400
+
if (ynl_attr_validate(yarg, attr))
401
+
return MNL_CB_ERROR;
402
+
dst->_present.alloc_waive = 1;
403
+
dst->alloc_waive = mnl_attr_get_uint(attr);
404
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED) {
405
+
if (ynl_attr_validate(yarg, attr))
406
+
return MNL_CB_ERROR;
407
+
dst->_present.recycle_cached = 1;
408
+
dst->recycle_cached = mnl_attr_get_uint(attr);
409
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL) {
410
+
if (ynl_attr_validate(yarg, attr))
411
+
return MNL_CB_ERROR;
412
+
dst->_present.recycle_cache_full = 1;
413
+
dst->recycle_cache_full = mnl_attr_get_uint(attr);
414
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING) {
415
+
if (ynl_attr_validate(yarg, attr))
416
+
return MNL_CB_ERROR;
417
+
dst->_present.recycle_ring = 1;
418
+
dst->recycle_ring = mnl_attr_get_uint(attr);
419
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL) {
420
+
if (ynl_attr_validate(yarg, attr))
421
+
return MNL_CB_ERROR;
422
+
dst->_present.recycle_ring_full = 1;
423
+
dst->recycle_ring_full = mnl_attr_get_uint(attr);
424
+
} else if (type == NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT) {
425
+
if (ynl_attr_validate(yarg, attr))
426
+
return MNL_CB_ERROR;
427
+
dst->_present.recycle_released_refcnt = 1;
428
+
dst->recycle_released_refcnt = mnl_attr_get_uint(attr);
429
+
}
430
+
}
431
+
432
+
return MNL_CB_OK;
433
+
}
434
+
435
+
struct netdev_page_pool_stats_get_rsp *
436
+
netdev_page_pool_stats_get(struct ynl_sock *ys,
437
+
struct netdev_page_pool_stats_get_req *req)
438
+
{
439
+
struct ynl_req_state yrs = { .yarg = { .ys = ys, }, };
440
+
struct netdev_page_pool_stats_get_rsp *rsp;
441
+
struct nlmsghdr *nlh;
442
+
int err;
443
+
444
+
nlh = ynl_gemsg_start_req(ys, ys->family_id, NETDEV_CMD_PAGE_POOL_STATS_GET, 1);
445
+
ys->req_policy = &netdev_page_pool_stats_nest;
446
+
yrs.yarg.rsp_policy = &netdev_page_pool_stats_nest;
447
+
448
+
if (req->_present.info)
449
+
netdev_page_pool_info_put(nlh, NETDEV_A_PAGE_POOL_STATS_INFO, &req->info);
450
+
451
+
rsp = calloc(1, sizeof(*rsp));
452
+
yrs.yarg.data = rsp;
453
+
yrs.cb = netdev_page_pool_stats_get_rsp_parse;
454
+
yrs.rsp_cmd = NETDEV_CMD_PAGE_POOL_STATS_GET;
455
+
456
+
err = ynl_exec(ys, nlh, &yrs);
457
+
if (err < 0)
458
+
goto err_free;
459
+
460
+
return rsp;
461
+
462
+
err_free:
463
+
netdev_page_pool_stats_get_rsp_free(rsp);
464
+
return NULL;
465
+
}
466
+
467
+
/* NETDEV_CMD_PAGE_POOL_STATS_GET - dump */
468
+
void
469
+
netdev_page_pool_stats_get_list_free(struct netdev_page_pool_stats_get_list *rsp)
470
+
{
471
+
struct netdev_page_pool_stats_get_list *next = rsp;
472
+
473
+
while ((void *)next != YNL_LIST_END) {
474
+
rsp = next;
475
+
next = rsp->next;
476
+
477
+
netdev_page_pool_info_free(&rsp->obj.info);
478
+
free(rsp);
479
+
}
480
+
}
481
+
482
+
struct netdev_page_pool_stats_get_list *
483
+
netdev_page_pool_stats_get_dump(struct ynl_sock *ys)
484
+
{
485
+
struct ynl_dump_state yds = {};
486
+
struct nlmsghdr *nlh;
487
+
int err;
488
+
489
+
yds.ys = ys;
490
+
yds.alloc_sz = sizeof(struct netdev_page_pool_stats_get_list);
491
+
yds.cb = netdev_page_pool_stats_get_rsp_parse;
492
+
yds.rsp_cmd = NETDEV_CMD_PAGE_POOL_STATS_GET;
493
+
yds.rsp_policy = &netdev_page_pool_stats_nest;
494
+
495
+
nlh = ynl_gemsg_start_dump(ys, ys->family_id, NETDEV_CMD_PAGE_POOL_STATS_GET, 1);
496
+
497
+
err = ynl_exec_dump(ys, nlh, &yds);
498
+
if (err < 0)
499
+
goto free_list;
500
+
501
+
return yds.first;
502
+
503
+
free_list:
504
+
netdev_page_pool_stats_get_list_free(yds.first);
505
+
return NULL;
506
+
}
507
+
293
508
static const struct ynl_ntf_info netdev_ntf_info[] = {
294
509
[NETDEV_CMD_DEV_ADD_NTF] = {
295
510
.alloc_sz = sizeof(struct netdev_dev_get_ntf),
···
616
215
.cb = netdev_dev_get_rsp_parse,
617
216
.policy = &netdev_dev_nest,
618
217
.free = (void *)netdev_dev_get_ntf_free,
218
+
},
219
+
[NETDEV_CMD_PAGE_POOL_ADD_NTF] = {
220
+
.alloc_sz = sizeof(struct netdev_page_pool_get_ntf),
221
+
.cb = netdev_page_pool_get_rsp_parse,
222
+
.policy = &netdev_page_pool_nest,
223
+
.free = (void *)netdev_page_pool_get_ntf_free,
224
+
},
225
+
[NETDEV_CMD_PAGE_POOL_DEL_NTF] = {
226
+
.alloc_sz = sizeof(struct netdev_page_pool_get_ntf),
227
+
.cb = netdev_page_pool_get_rsp_parse,
228
+
.policy = &netdev_page_pool_nest,
229
+
.free = (void *)netdev_page_pool_get_ntf_free,
230
+
},
231
+
[NETDEV_CMD_PAGE_POOL_CHANGE_NTF] = {
232
+
.alloc_sz = sizeof(struct netdev_page_pool_get_ntf),
233
+
.cb = netdev_page_pool_get_rsp_parse,
234
+
.policy = &netdev_page_pool_nest,
235
+
.free = (void *)netdev_page_pool_get_ntf_free,
619
236
},
620
237
};
621
238
+171
tools/net/ynl/generated/netdev-user.h
+171
tools/net/ynl/generated/netdev-user.h
···
21
21
const char *netdev_xdp_rx_metadata_str(enum netdev_xdp_rx_metadata value);
22
22
23
23
/* Common nested types */
24
+
struct netdev_page_pool_info {
25
+
struct {
26
+
__u32 id:1;
27
+
__u32 ifindex:1;
28
+
} _present;
29
+
30
+
__u64 id;
31
+
__u32 ifindex;
32
+
};
33
+
24
34
/* ============== NETDEV_CMD_DEV_GET ============== */
25
35
/* NETDEV_CMD_DEV_GET - do */
26
36
struct netdev_dev_get_req {
···
96
86
};
97
87
98
88
void netdev_dev_get_ntf_free(struct netdev_dev_get_ntf *rsp);
89
+
90
+
/* ============== NETDEV_CMD_PAGE_POOL_GET ============== */
91
+
/* NETDEV_CMD_PAGE_POOL_GET - do */
92
+
struct netdev_page_pool_get_req {
93
+
struct {
94
+
__u32 id:1;
95
+
} _present;
96
+
97
+
__u64 id;
98
+
};
99
+
100
+
static inline struct netdev_page_pool_get_req *
101
+
netdev_page_pool_get_req_alloc(void)
102
+
{
103
+
return calloc(1, sizeof(struct netdev_page_pool_get_req));
104
+
}
105
+
void netdev_page_pool_get_req_free(struct netdev_page_pool_get_req *req);
106
+
107
+
static inline void
108
+
netdev_page_pool_get_req_set_id(struct netdev_page_pool_get_req *req, __u64 id)
109
+
{
110
+
req->_present.id = 1;
111
+
req->id = id;
112
+
}
113
+
114
+
struct netdev_page_pool_get_rsp {
115
+
struct {
116
+
__u32 id:1;
117
+
__u32 ifindex:1;
118
+
__u32 napi_id:1;
119
+
__u32 inflight:1;
120
+
__u32 inflight_mem:1;
121
+
__u32 detach_time:1;
122
+
} _present;
123
+
124
+
__u64 id;
125
+
__u32 ifindex;
126
+
__u64 napi_id;
127
+
__u64 inflight;
128
+
__u64 inflight_mem;
129
+
__u64 detach_time;
130
+
};
131
+
132
+
void netdev_page_pool_get_rsp_free(struct netdev_page_pool_get_rsp *rsp);
133
+
134
+
/*
135
+
* Get / dump information about Page Pools.
136
+
(Only Page Pools associated with a net_device can be listed.)
137
+
138
+
*/
139
+
struct netdev_page_pool_get_rsp *
140
+
netdev_page_pool_get(struct ynl_sock *ys, struct netdev_page_pool_get_req *req);
141
+
142
+
/* NETDEV_CMD_PAGE_POOL_GET - dump */
143
+
struct netdev_page_pool_get_list {
144
+
struct netdev_page_pool_get_list *next;
145
+
struct netdev_page_pool_get_rsp obj __attribute__((aligned(8)));
146
+
};
147
+
148
+
void netdev_page_pool_get_list_free(struct netdev_page_pool_get_list *rsp);
149
+
150
+
struct netdev_page_pool_get_list *
151
+
netdev_page_pool_get_dump(struct ynl_sock *ys);
152
+
153
+
/* NETDEV_CMD_PAGE_POOL_GET - notify */
154
+
struct netdev_page_pool_get_ntf {
155
+
__u16 family;
156
+
__u8 cmd;
157
+
struct ynl_ntf_base_type *next;
158
+
void (*free)(struct netdev_page_pool_get_ntf *ntf);
159
+
struct netdev_page_pool_get_rsp obj __attribute__((aligned(8)));
160
+
};
161
+
162
+
void netdev_page_pool_get_ntf_free(struct netdev_page_pool_get_ntf *rsp);
163
+
164
+
/* ============== NETDEV_CMD_PAGE_POOL_STATS_GET ============== */
165
+
/* NETDEV_CMD_PAGE_POOL_STATS_GET - do */
166
+
struct netdev_page_pool_stats_get_req {
167
+
struct {
168
+
__u32 info:1;
169
+
} _present;
170
+
171
+
struct netdev_page_pool_info info;
172
+
};
173
+
174
+
static inline struct netdev_page_pool_stats_get_req *
175
+
netdev_page_pool_stats_get_req_alloc(void)
176
+
{
177
+
return calloc(1, sizeof(struct netdev_page_pool_stats_get_req));
178
+
}
179
+
void
180
+
netdev_page_pool_stats_get_req_free(struct netdev_page_pool_stats_get_req *req);
181
+
182
+
static inline void
183
+
netdev_page_pool_stats_get_req_set_info_id(struct netdev_page_pool_stats_get_req *req,
184
+
__u64 id)
185
+
{
186
+
req->_present.info = 1;
187
+
req->info._present.id = 1;
188
+
req->info.id = id;
189
+
}
190
+
static inline void
191
+
netdev_page_pool_stats_get_req_set_info_ifindex(struct netdev_page_pool_stats_get_req *req,
192
+
__u32 ifindex)
193
+
{
194
+
req->_present.info = 1;
195
+
req->info._present.ifindex = 1;
196
+
req->info.ifindex = ifindex;
197
+
}
198
+
199
+
struct netdev_page_pool_stats_get_rsp {
200
+
struct {
201
+
__u32 info:1;
202
+
__u32 alloc_fast:1;
203
+
__u32 alloc_slow:1;
204
+
__u32 alloc_slow_high_order:1;
205
+
__u32 alloc_empty:1;
206
+
__u32 alloc_refill:1;
207
+
__u32 alloc_waive:1;
208
+
__u32 recycle_cached:1;
209
+
__u32 recycle_cache_full:1;
210
+
__u32 recycle_ring:1;
211
+
__u32 recycle_ring_full:1;
212
+
__u32 recycle_released_refcnt:1;
213
+
} _present;
214
+
215
+
struct netdev_page_pool_info info;
216
+
__u64 alloc_fast;
217
+
__u64 alloc_slow;
218
+
__u64 alloc_slow_high_order;
219
+
__u64 alloc_empty;
220
+
__u64 alloc_refill;
221
+
__u64 alloc_waive;
222
+
__u64 recycle_cached;
223
+
__u64 recycle_cache_full;
224
+
__u64 recycle_ring;
225
+
__u64 recycle_ring_full;
226
+
__u64 recycle_released_refcnt;
227
+
};
228
+
229
+
void
230
+
netdev_page_pool_stats_get_rsp_free(struct netdev_page_pool_stats_get_rsp *rsp);
231
+
232
+
/*
233
+
* Get page pool statistics.
234
+
*/
235
+
struct netdev_page_pool_stats_get_rsp *
236
+
netdev_page_pool_stats_get(struct ynl_sock *ys,
237
+
struct netdev_page_pool_stats_get_req *req);
238
+
239
+
/* NETDEV_CMD_PAGE_POOL_STATS_GET - dump */
240
+
struct netdev_page_pool_stats_get_list {
241
+
struct netdev_page_pool_stats_get_list *next;
242
+
struct netdev_page_pool_stats_get_rsp obj __attribute__((aligned(8)));
243
+
};
244
+
245
+
void
246
+
netdev_page_pool_stats_get_list_free(struct netdev_page_pool_stats_get_list *rsp);
247
+
248
+
struct netdev_page_pool_stats_get_list *
249
+
netdev_page_pool_stats_get_dump(struct ynl_sock *ys);
99
250
100
251
#endif /* _LINUX_NETDEV_GEN_H */
+1
-1
tools/net/ynl/lib/ynl.h
+1
-1
tools/net/ynl/lib/ynl.h
···
239
239
#ifndef MNL_HAS_AUTO_SCALARS
240
240
static inline uint64_t mnl_attr_get_uint(const struct nlattr *attr)
241
241
{
242
-
if (mnl_attr_get_len(attr) == 4)
242
+
if (mnl_attr_get_payload_len(attr) == 4)
243
243
return mnl_attr_get_u32(attr);
244
244
return mnl_attr_get_u64(attr);
245
245
}
+1
-1
tools/net/ynl/samples/Makefile
+1
-1
tools/net/ynl/samples/Makefile
+147
tools/net/ynl/samples/page-pool.c
+147
tools/net/ynl/samples/page-pool.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#define _GNU_SOURCE
3
+
4
+
#include <stdio.h>
5
+
#include <string.h>
6
+
7
+
#include <ynl.h>
8
+
9
+
#include <net/if.h>
10
+
11
+
#include "netdev-user.h"
12
+
13
+
struct stat {
14
+
unsigned int ifc;
15
+
16
+
struct {
17
+
unsigned int cnt;
18
+
size_t refs, bytes;
19
+
} live[2];
20
+
21
+
size_t alloc_slow, alloc_fast, recycle_ring, recycle_cache;
22
+
};
23
+
24
+
struct stats_array {
25
+
unsigned int i, max;
26
+
struct stat *s;
27
+
};
28
+
29
+
static struct stat *find_ifc(struct stats_array *a, unsigned int ifindex)
30
+
{
31
+
unsigned int i;
32
+
33
+
for (i = 0; i < a->i; i++) {
34
+
if (a->s[i].ifc == ifindex)
35
+
return &a->s[i];
36
+
}
37
+
38
+
a->i++;
39
+
if (a->i == a->max) {
40
+
a->max *= 2;
41
+
a->s = reallocarray(a->s, a->max, sizeof(*a->s));
42
+
}
43
+
a->s[i].ifc = ifindex;
44
+
return &a->s[i];
45
+
}
46
+
47
+
static void count(struct stat *s, unsigned int l,
48
+
struct netdev_page_pool_get_rsp *pp)
49
+
{
50
+
s->live[l].cnt++;
51
+
if (pp->_present.inflight)
52
+
s->live[l].refs += pp->inflight;
53
+
if (pp->_present.inflight_mem)
54
+
s->live[l].bytes += pp->inflight_mem;
55
+
}
56
+
57
+
int main(int argc, char **argv)
58
+
{
59
+
struct netdev_page_pool_stats_get_list *pp_stats;
60
+
struct netdev_page_pool_get_list *pools;
61
+
struct stats_array a = {};
62
+
struct ynl_error yerr;
63
+
struct ynl_sock *ys;
64
+
65
+
ys = ynl_sock_create(&ynl_netdev_family, &yerr);
66
+
if (!ys) {
67
+
fprintf(stderr, "YNL: %s\n", yerr.msg);
68
+
return 1;
69
+
}
70
+
71
+
a.max = 128;
72
+
a.s = calloc(a.max, sizeof(*a.s));
73
+
if (!a.s)
74
+
goto err_close;
75
+
76
+
pools = netdev_page_pool_get_dump(ys);
77
+
if (!pools)
78
+
goto err_free;
79
+
80
+
ynl_dump_foreach(pools, pp) {
81
+
struct stat *s = find_ifc(&a, pp->ifindex);
82
+
83
+
count(s, 1, pp);
84
+
if (pp->_present.destroyed)
85
+
count(s, 0, pp);
86
+
}
87
+
netdev_page_pool_get_list_free(pools);
88
+
89
+
pp_stats = netdev_page_pool_stats_get_dump(ys);
90
+
if (!pp_stats)
91
+
goto err_free;
92
+
93
+
ynl_dump_foreach(pp_stats, pp) {
94
+
struct stat *s = find_ifc(&a, pp->info.ifindex);
95
+
96
+
if (pp->_present.alloc_fast)
97
+
s->alloc_fast += pp->alloc_fast;
98
+
if (pp->_present.alloc_slow)
99
+
s->alloc_slow += pp->alloc_slow;
100
+
if (pp->_present.recycle_ring)
101
+
s->recycle_ring += pp->recycle_ring;
102
+
if (pp->_present.recycle_cached)
103
+
s->recycle_cache += pp->recycle_cached;
104
+
}
105
+
netdev_page_pool_stats_get_list_free(pp_stats);
106
+
107
+
for (unsigned int i = 0; i < a.i; i++) {
108
+
char ifname[IF_NAMESIZE];
109
+
struct stat *s = &a.s[i];
110
+
const char *name;
111
+
double recycle;
112
+
113
+
if (!s->ifc) {
114
+
name = "<orphan>\t";
115
+
} else {
116
+
name = if_indextoname(s->ifc, ifname);
117
+
if (name)
118
+
printf("%8s", name);
119
+
printf("[%d]\t", s->ifc);
120
+
}
121
+
122
+
printf("page pools: %u (zombies: %u)\n",
123
+
s->live[1].cnt, s->live[0].cnt);
124
+
printf("\t\trefs: %zu bytes: %zu (refs: %zu bytes: %zu)\n",
125
+
s->live[1].refs, s->live[1].bytes,
126
+
s->live[0].refs, s->live[0].bytes);
127
+
128
+
/* We don't know how many pages are sitting in cache and ring
129
+
* so we will under-count the recycling rate a bit.
130
+
*/
131
+
recycle = (double)(s->recycle_ring + s->recycle_cache) /
132
+
(s->alloc_fast + s->alloc_slow) * 100;
133
+
printf("\t\trecycling: %.1lf%% (alloc: %zu:%zu recycle: %zu:%zu)\n",
134
+
recycle, s->alloc_slow, s->alloc_fast,
135
+
s->recycle_ring, s->recycle_cache);
136
+
}
137
+
138
+
ynl_sock_destroy(ys);
139
+
return 0;
140
+
141
+
err_free:
142
+
free(a.s);
143
+
err_close:
144
+
fprintf(stderr, "YNL: %s\n", ys->err.msg);
145
+
ynl_sock_destroy(ys);
146
+
return 2;
147
+
}