Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'bpf-introduce-bpf_f_cpu-and-bpf_f_all_cpus-flags-for-percpu-maps'

Leon Hwang says:

====================
bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps

This patch set introduces the BPF_F_CPU and BPF_F_ALL_CPUS flags for
percpu maps, as the requirement of BPF_F_ALL_CPUS flag for percpu_array
maps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].

The goal of BPF_F_ALL_CPUS flag is to reduce data caching overhead in light
skeletons by allowing a single value to be reused to update values across all
CPUs. This avoids the M:N problem where M cached values are used to update a
map on N CPUs kernel.

The BPF_F_CPU flag is accompanied by *flags*-embedded cpu info, which
specifies the target CPU for the operation:

* For lookup operations: the flag field alongside cpu info enable querying
a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable
updating value for specified CPU.

Links:
[1] https://lore.kernel.org/bpf/20250526162146.24429-1-leon.hwang@linux.dev/

Changes:
v12 -> v13:
* No changes, rebased on latest tree.

v11 -> v12:
* Dropped the v11 changes.
* Stabilized the lru_percpu_hash map test by keeping an extra spare entry,
which can be used temporarily during updates to avoid unintended LRU
evictions.

v10 -> v11:
* Support the combination of BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS for
update operations.
* Fix unstable lru_percpu_hash map test using the combination of
BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS to avoid LRU eviction
(reported by Alexei).

v9 -> v10:
* Add tests to verify array and hash maps do not support BPF_F_CPU and
BPF_F_ALL_CPUS flags.
* Address comment from Andrii:
* Copy map value using copy_map_value_long for percpu_cgroup_storage
maps in a separate patch.

v8 -> v9:
* Change value type from u64 to u32 in selftests.
* Address comments from Andrii:
* Keep value_size unaligned and update everywhere for consistency when
cpu flags are specified.
* Update value by getting pointer for percpu hash and percpu
cgroup_storage maps.

v7 -> v8:
* Address comments from Andrii:
* Check BPF_F_LOCK when update percpu_array, percpu_hash and
lru_percpu_hash maps.
* Refactor flags check in __htab_map_lookup_and_delete_batch().
* Keep value_size unaligned and copy value using copy_map_value() in
__htab_map_lookup_and_delete_batch() when BPF_F_CPU is specified.
* Update warn message in libbpf's validate_map_op().
* Update comment of libbpf's bpf_map__lookup_elem().

v6 -> v7:
* Get correct value size for percpu_hash and lru_percpu_hash in
update_batch API.
* Set 'count' as 'max_entries' in test cases for lookup_batch API.
* Address comment from Alexei:
* Move cpu flags check into bpf_map_check_op_flags().

v5 -> v6:
* Move bpf_map_check_op_flags() from 'bpf.h' to 'syscall.c'.
* Address comments from Alexei:
* Drop the refactoring code of data copying logic for percpu maps.
* Drop bpf_map_check_op_flags() wrappers.

v4 -> v5:
* Address comments from Andrii:
* Refactor data copying logic for all percpu maps.
* Drop this_cpu_ptr() micro-optimization.
* Drop cpu check in libbpf's validate_map_op().
* Enhance bpf_map_check_op_flags() using *allowed flags* instead of
'extra_flags_mask'.

v3 -> v4:
* Address comments from Andrii:
* Remove unnecessary map_type check in bpf_map_value_size().
* Reduce code churn.
* Remove unnecessary do_delete check in
__htab_map_lookup_and_delete_batch().
* Introduce bpf_percpu_copy_to_user() and bpf_percpu_copy_from_user().
* Rename check_map_flags() to bpf_map_check_op_flags() with
extra_flags_mask.
* Add human-readable pr_warn() explanations in validate_map_op().
* Use flags in bpf_map__delete_elem() and
bpf_map__lookup_and_delete_elem().
* Drop "for alignment reasons".
v3 link: https://lore.kernel.org/bpf/20250821160817.70285-1-leon.hwang@linux.dev/

v2 -> v3:
* Address comments from Alexei:
* Use BPF_F_ALL_CPUS instead of BPF_ALL_CPUS magic.
* Introduce these two cpu flags for all percpu maps.
* Address comments from Jiri:
* Reduce some unnecessary u32 cast.
* Refactor more generic map flags check function.
* A code style issue.
v2 link: https://lore.kernel.org/bpf/20250805163017.17015-1-leon.hwang@linux.dev/

v1 -> v2:
* Address comments from Andrii:
* Embed cpu info as high 32 bits of *flags* totally.
* Use ERANGE instead of E2BIG.
* Few format issues.
====================

Link: https://patch.msgid.link/20260107022022.12843-1-leon.hwang@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

+560 -85
+2 -2
include/linux/bpf-cgroup.h
··· 172 172 void bpf_cgroup_storage_unlink(struct bpf_cgroup_storage *storage); 173 173 int bpf_cgroup_storage_assign(struct bpf_prog_aux *aux, struct bpf_map *map); 174 174 175 - int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value); 175 + int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value, u64 flags); 176 176 int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, 177 177 void *value, u64 flags); 178 178 ··· 470 470 static inline void bpf_cgroup_storage_free( 471 471 struct bpf_cgroup_storage *storage) {} 472 472 static inline int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, 473 - void *value) { 473 + void *value, u64 flags) { 474 474 return 0; 475 475 } 476 476 static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
+32 -3
include/linux/bpf.h
··· 2847 2847 struct bpf_func_state *caller, 2848 2848 struct bpf_func_state *callee); 2849 2849 2850 - int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value); 2851 - int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value); 2850 + int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 flags); 2851 + int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 flags); 2852 2852 int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value, 2853 2853 u64 flags); 2854 2854 int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, ··· 3915 3915 } 3916 3916 #endif 3917 3917 3918 + static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type) 3919 + { 3920 + switch (map_type) { 3921 + case BPF_MAP_TYPE_PERCPU_ARRAY: 3922 + case BPF_MAP_TYPE_PERCPU_HASH: 3923 + case BPF_MAP_TYPE_LRU_PERCPU_HASH: 3924 + case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE: 3925 + return true; 3926 + default: 3927 + return false; 3928 + } 3929 + } 3930 + 3918 3931 static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allowed_flags) 3919 3932 { 3920 - if (flags & ~allowed_flags) 3933 + u32 cpu; 3934 + 3935 + if ((u32)flags & ~allowed_flags) 3921 3936 return -EINVAL; 3922 3937 3923 3938 if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK)) 3924 3939 return -EINVAL; 3940 + 3941 + if (!(flags & BPF_F_CPU) && flags >> 32) 3942 + return -EINVAL; 3943 + 3944 + if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) { 3945 + if (!bpf_map_supports_cpu_flags(map->map_type)) 3946 + return -EINVAL; 3947 + if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS)) 3948 + return -EINVAL; 3949 + 3950 + cpu = flags >> 32; 3951 + if ((flags & BPF_F_CPU) && cpu >= num_possible_cpus()) 3952 + return -ERANGE; 3953 + } 3925 3954 3926 3955 return 0; 3927 3956 }
+2
include/uapi/linux/bpf.h
··· 1384 1384 BPF_NOEXIST = 1, /* create new element if it didn't exist */ 1385 1385 BPF_EXIST = 2, /* update existing element */ 1386 1386 BPF_F_LOCK = 4, /* spin_lock-ed map_lookup/map_update */ 1387 + BPF_F_CPU = 8, /* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ 1388 + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ 1387 1389 }; 1388 1390 1389 1391 /* flags for BPF_MAP_CREATE command */
+24 -7
kernel/bpf/arraymap.c
··· 307 307 return per_cpu_ptr(array->pptrs[index & array->index_mask], cpu); 308 308 } 309 309 310 - int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) 310 + int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 map_flags) 311 311 { 312 312 struct bpf_array *array = container_of(map, struct bpf_array, map); 313 313 u32 index = *(u32 *)key; ··· 325 325 size = array->elem_size; 326 326 rcu_read_lock(); 327 327 pptr = array->pptrs[index & array->index_mask]; 328 + if (map_flags & BPF_F_CPU) { 329 + cpu = map_flags >> 32; 330 + copy_map_value(map, value, per_cpu_ptr(pptr, cpu)); 331 + check_and_init_map_value(map, value); 332 + goto unlock; 333 + } 328 334 for_each_possible_cpu(cpu) { 329 335 copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu)); 330 336 check_and_init_map_value(map, value + off); 331 337 off += size; 332 338 } 339 + unlock: 333 340 rcu_read_unlock(); 334 341 return 0; 335 342 } ··· 405 398 struct bpf_array *array = container_of(map, struct bpf_array, map); 406 399 u32 index = *(u32 *)key; 407 400 void __percpu *pptr; 408 - int cpu, off = 0; 401 + void *ptr, *val; 409 402 u32 size; 403 + int cpu; 410 404 411 - if (unlikely(map_flags > BPF_EXIST)) 405 + if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS)) 412 406 /* unknown flags */ 413 407 return -EINVAL; 414 408 ··· 430 422 size = array->elem_size; 431 423 rcu_read_lock(); 432 424 pptr = array->pptrs[index & array->index_mask]; 433 - for_each_possible_cpu(cpu) { 434 - copy_map_value_long(map, per_cpu_ptr(pptr, cpu), value + off); 435 - bpf_obj_free_fields(array->map.record, per_cpu_ptr(pptr, cpu)); 436 - off += size; 425 + if (map_flags & BPF_F_CPU) { 426 + cpu = map_flags >> 32; 427 + ptr = per_cpu_ptr(pptr, cpu); 428 + copy_map_value(map, ptr, value); 429 + bpf_obj_free_fields(array->map.record, ptr); 430 + goto unlock; 437 431 } 432 + for_each_possible_cpu(cpu) { 433 + ptr = per_cpu_ptr(pptr, cpu); 434 + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; 435 + copy_map_value(map, ptr, val); 436 + bpf_obj_free_fields(array->map.record, ptr); 437 + } 438 + unlock: 438 439 rcu_read_unlock(); 439 440 return 0; 440 441 }
+64 -30
kernel/bpf/hashtab.c
··· 932 932 } 933 933 934 934 static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, 935 - void *value, bool onallcpus) 935 + void *value, bool onallcpus, u64 map_flags) 936 936 { 937 937 void *ptr; 938 938 ··· 943 943 bpf_obj_free_fields(htab->map.record, ptr); 944 944 } else { 945 945 u32 size = round_up(htab->map.value_size, 8); 946 - int off = 0, cpu; 946 + void *val; 947 + int cpu; 948 + 949 + if (map_flags & BPF_F_CPU) { 950 + cpu = map_flags >> 32; 951 + ptr = per_cpu_ptr(pptr, cpu); 952 + copy_map_value(&htab->map, ptr, value); 953 + bpf_obj_free_fields(htab->map.record, ptr); 954 + return; 955 + } 947 956 948 957 for_each_possible_cpu(cpu) { 949 958 ptr = per_cpu_ptr(pptr, cpu); 950 - copy_map_value_long(&htab->map, ptr, value + off); 959 + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; 960 + copy_map_value(&htab->map, ptr, val); 951 961 bpf_obj_free_fields(htab->map.record, ptr); 952 - off += size; 953 962 } 954 963 } 955 964 } 956 965 957 966 static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr, 958 - void *value, bool onallcpus) 967 + void *value, bool onallcpus, u64 map_flags) 959 968 { 960 969 /* When not setting the initial value on all cpus, zero-fill element 961 970 * values for other cpus. Otherwise, bpf program has no way to ensure ··· 982 973 zero_map_value(&htab->map, per_cpu_ptr(pptr, cpu)); 983 974 } 984 975 } else { 985 - pcpu_copy_value(htab, pptr, value, onallcpus); 976 + pcpu_copy_value(htab, pptr, value, onallcpus, map_flags); 986 977 } 987 978 } 988 979 ··· 994 985 static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, 995 986 void *value, u32 key_size, u32 hash, 996 987 bool percpu, bool onallcpus, 997 - struct htab_elem *old_elem) 988 + struct htab_elem *old_elem, u64 map_flags) 998 989 { 999 990 u32 size = htab->map.value_size; 1000 991 bool prealloc = htab_is_prealloc(htab); ··· 1052 1043 pptr = *(void __percpu **)ptr; 1053 1044 } 1054 1045 1055 - pcpu_init_value(htab, pptr, value, onallcpus); 1046 + pcpu_init_value(htab, pptr, value, onallcpus, map_flags); 1056 1047 1057 1048 if (!prealloc) 1058 1049 htab_elem_set_ptr(l_new, key_size, pptr); ··· 1156 1147 } 1157 1148 1158 1149 l_new = alloc_htab_elem(htab, key, value, key_size, hash, false, false, 1159 - l_old); 1150 + l_old, map_flags); 1160 1151 if (IS_ERR(l_new)) { 1161 1152 /* all pre-allocated elements are in use or memory exhausted */ 1162 1153 ret = PTR_ERR(l_new); ··· 1258 1249 return ret; 1259 1250 } 1260 1251 1252 + static int htab_map_check_update_flags(bool onallcpus, u64 map_flags) 1253 + { 1254 + if (unlikely(!onallcpus && map_flags > BPF_EXIST)) 1255 + return -EINVAL; 1256 + if (unlikely(onallcpus && ((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))) 1257 + return -EINVAL; 1258 + return 0; 1259 + } 1260 + 1261 1261 static long htab_map_update_elem_in_place(struct bpf_map *map, void *key, 1262 1262 void *value, u64 map_flags, 1263 1263 bool percpu, bool onallcpus) ··· 1280 1262 u32 key_size, hash; 1281 1263 int ret; 1282 1264 1283 - if (unlikely(map_flags > BPF_EXIST)) 1284 - /* unknown flags */ 1285 - return -EINVAL; 1265 + ret = htab_map_check_update_flags(onallcpus, map_flags); 1266 + if (unlikely(ret)) 1267 + return ret; 1286 1268 1287 1269 WARN_ON_ONCE(!bpf_rcu_lock_held()); 1288 1270 ··· 1307 1289 /* Update value in-place */ 1308 1290 if (percpu) { 1309 1291 pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size), 1310 - value, onallcpus); 1292 + value, onallcpus, map_flags); 1311 1293 } else { 1312 1294 void **inner_map_pptr = htab_elem_value(l_old, key_size); 1313 1295 ··· 1316 1298 } 1317 1299 } else { 1318 1300 l_new = alloc_htab_elem(htab, key, value, key_size, 1319 - hash, percpu, onallcpus, NULL); 1301 + hash, percpu, onallcpus, NULL, map_flags); 1320 1302 if (IS_ERR(l_new)) { 1321 1303 ret = PTR_ERR(l_new); 1322 1304 goto err; ··· 1342 1324 u32 key_size, hash; 1343 1325 int ret; 1344 1326 1345 - if (unlikely(map_flags > BPF_EXIST)) 1346 - /* unknown flags */ 1347 - return -EINVAL; 1327 + ret = htab_map_check_update_flags(onallcpus, map_flags); 1328 + if (unlikely(ret)) 1329 + return ret; 1348 1330 1349 1331 WARN_ON_ONCE(!bpf_rcu_lock_held()); 1350 1332 ··· 1381 1363 1382 1364 /* per-cpu hash map can update value in-place */ 1383 1365 pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size), 1384 - value, onallcpus); 1366 + value, onallcpus, map_flags); 1385 1367 } else { 1386 1368 pcpu_init_value(htab, htab_elem_get_ptr(l_new, key_size), 1387 - value, onallcpus); 1369 + value, onallcpus, map_flags); 1388 1370 hlist_nulls_add_head_rcu(&l_new->hash_node, head); 1389 1371 l_new = NULL; 1390 1372 } ··· 1696 1678 void __user *ukeys = u64_to_user_ptr(attr->batch.keys); 1697 1679 void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch); 1698 1680 u32 batch, max_count, size, bucket_size, map_id; 1681 + u64 elem_map_flags, map_flags, allowed_flags; 1699 1682 u32 bucket_cnt, total, key_size, value_size; 1700 1683 struct htab_elem *node_to_free = NULL; 1701 - u64 elem_map_flags, map_flags; 1702 1684 struct hlist_nulls_head *head; 1703 1685 struct hlist_nulls_node *n; 1704 1686 unsigned long flags = 0; ··· 1708 1690 int ret = 0; 1709 1691 1710 1692 elem_map_flags = attr->batch.elem_flags; 1711 - if ((elem_map_flags & ~BPF_F_LOCK) || 1712 - ((elem_map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))) 1713 - return -EINVAL; 1693 + allowed_flags = BPF_F_LOCK; 1694 + if (!do_delete && is_percpu) 1695 + allowed_flags |= BPF_F_CPU; 1696 + ret = bpf_map_check_op_flags(map, elem_map_flags, allowed_flags); 1697 + if (ret) 1698 + return ret; 1714 1699 1715 1700 map_flags = attr->batch.flags; 1716 1701 if (map_flags) ··· 1736 1715 key_size = htab->map.key_size; 1737 1716 value_size = htab->map.value_size; 1738 1717 size = round_up(value_size, 8); 1739 - if (is_percpu) 1718 + if (is_percpu && !(elem_map_flags & BPF_F_CPU)) 1740 1719 value_size = size * num_possible_cpus(); 1741 1720 total = 0; 1742 1721 /* while experimenting with hash tables with sizes ranging from 10 to ··· 1819 1798 void __percpu *pptr; 1820 1799 1821 1800 pptr = htab_elem_get_ptr(l, map->key_size); 1822 - for_each_possible_cpu(cpu) { 1823 - copy_map_value_long(&htab->map, dst_val + off, per_cpu_ptr(pptr, cpu)); 1824 - check_and_init_map_value(&htab->map, dst_val + off); 1825 - off += size; 1801 + if (elem_map_flags & BPF_F_CPU) { 1802 + cpu = elem_map_flags >> 32; 1803 + copy_map_value(&htab->map, dst_val, per_cpu_ptr(pptr, cpu)); 1804 + check_and_init_map_value(&htab->map, dst_val); 1805 + } else { 1806 + for_each_possible_cpu(cpu) { 1807 + copy_map_value_long(&htab->map, dst_val + off, 1808 + per_cpu_ptr(pptr, cpu)); 1809 + check_and_init_map_value(&htab->map, dst_val + off); 1810 + off += size; 1811 + } 1826 1812 } 1827 1813 } else { 1828 1814 value = htab_elem_value(l, key_size); ··· 2385 2357 return NULL; 2386 2358 } 2387 2359 2388 - int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value) 2360 + int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 map_flags) 2389 2361 { 2390 2362 struct htab_elem *l; 2391 2363 void __percpu *pptr; ··· 2402 2374 l = __htab_map_lookup_elem(map, key); 2403 2375 if (!l) 2404 2376 goto out; 2377 + ret = 0; 2405 2378 /* We do not mark LRU map element here in order to not mess up 2406 2379 * eviction heuristics when user space does a map walk. 2407 2380 */ 2408 2381 pptr = htab_elem_get_ptr(l, map->key_size); 2382 + if (map_flags & BPF_F_CPU) { 2383 + cpu = map_flags >> 32; 2384 + copy_map_value(map, value, per_cpu_ptr(pptr, cpu)); 2385 + check_and_init_map_value(map, value); 2386 + goto out; 2387 + } 2409 2388 for_each_possible_cpu(cpu) { 2410 2389 copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu)); 2411 2390 check_and_init_map_value(map, value + off); 2412 2391 off += size; 2413 2392 } 2414 - ret = 0; 2415 2393 out: 2416 2394 rcu_read_unlock(); 2417 2395 return ret;
+19 -8
kernel/bpf/local_storage.c
··· 180 180 } 181 181 182 182 int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, 183 - void *value) 183 + void *value, u64 map_flags) 184 184 { 185 185 struct bpf_cgroup_storage_map *map = map_to_storage(_map); 186 186 struct bpf_cgroup_storage *storage; ··· 198 198 * access 'value_size' of them, so copying rounded areas 199 199 * will not leak any kernel data 200 200 */ 201 + if (map_flags & BPF_F_CPU) { 202 + cpu = map_flags >> 32; 203 + copy_map_value(_map, value, per_cpu_ptr(storage->percpu_buf, cpu)); 204 + goto unlock; 205 + } 201 206 size = round_up(_map->value_size, 8); 202 207 for_each_possible_cpu(cpu) { 203 - bpf_long_memcpy(value + off, 204 - per_cpu_ptr(storage->percpu_buf, cpu), size); 208 + copy_map_value_long(_map, value + off, per_cpu_ptr(storage->percpu_buf, cpu)); 205 209 off += size; 206 210 } 211 + unlock: 207 212 rcu_read_unlock(); 208 213 return 0; 209 214 } ··· 218 213 { 219 214 struct bpf_cgroup_storage_map *map = map_to_storage(_map); 220 215 struct bpf_cgroup_storage *storage; 221 - int cpu, off = 0; 216 + void *val; 222 217 u32 size; 218 + int cpu; 223 219 224 - if (map_flags != BPF_ANY && map_flags != BPF_EXIST) 220 + if ((u32)map_flags & ~(BPF_ANY | BPF_EXIST | BPF_F_CPU | BPF_F_ALL_CPUS)) 225 221 return -EINVAL; 226 222 227 223 rcu_read_lock(); ··· 238 232 * returned or zeros which were zero-filled by percpu_alloc, 239 233 * so no kernel data leaks possible 240 234 */ 235 + if (map_flags & BPF_F_CPU) { 236 + cpu = map_flags >> 32; 237 + copy_map_value(_map, per_cpu_ptr(storage->percpu_buf, cpu), value); 238 + goto unlock; 239 + } 241 240 size = round_up(_map->value_size, 8); 242 241 for_each_possible_cpu(cpu) { 243 - bpf_long_memcpy(per_cpu_ptr(storage->percpu_buf, cpu), 244 - value + off, size); 245 - off += size; 242 + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; 243 + copy_map_value(_map, per_cpu_ptr(storage->percpu_buf, cpu), val); 246 244 } 245 + unlock: 247 246 rcu_read_unlock(); 248 247 return 0; 249 248 }
+19 -16
kernel/bpf/syscall.c
··· 133 133 return atomic64_read(&map->writecnt) != 0; 134 134 } 135 135 136 - static u32 bpf_map_value_size(const struct bpf_map *map) 136 + static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags) 137 137 { 138 - if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || 139 - map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || 140 - map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY || 141 - map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) 138 + if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) 139 + return map->value_size; 140 + else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || 141 + map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || 142 + map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY || 143 + map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) 142 144 return round_up(map->value_size, 8) * num_possible_cpus(); 143 145 else if (IS_FD_MAP(map)) 144 146 return sizeof(u32); ··· 316 314 bpf_disable_instrumentation(); 317 315 if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || 318 316 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { 319 - err = bpf_percpu_hash_copy(map, key, value); 317 + err = bpf_percpu_hash_copy(map, key, value, flags); 320 318 } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { 321 - err = bpf_percpu_array_copy(map, key, value); 319 + err = bpf_percpu_array_copy(map, key, value, flags); 322 320 } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { 323 - err = bpf_percpu_cgroup_storage_copy(map, key, value); 321 + err = bpf_percpu_cgroup_storage_copy(map, key, value, flags); 324 322 } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) { 325 323 err = bpf_stackmap_extract(map, key, value, false); 326 324 } else if (IS_FD_ARRAY(map) || IS_FD_PROG_ARRAY(map)) { ··· 1731 1729 if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ)) 1732 1730 return -EPERM; 1733 1731 1734 - err = bpf_map_check_op_flags(map, attr->flags, BPF_F_LOCK); 1732 + err = bpf_map_check_op_flags(map, attr->flags, BPF_F_LOCK | BPF_F_CPU); 1735 1733 if (err) 1736 1734 return err; 1737 1735 ··· 1739 1737 if (IS_ERR(key)) 1740 1738 return PTR_ERR(key); 1741 1739 1742 - value_size = bpf_map_value_size(map); 1740 + value_size = bpf_map_value_size(map, attr->flags); 1743 1741 1744 1742 err = -ENOMEM; 1745 1743 value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); ··· 1806 1804 goto err_put; 1807 1805 } 1808 1806 1809 - value_size = bpf_map_value_size(map); 1807 + value_size = bpf_map_value_size(map, attr->flags); 1810 1808 value = kvmemdup_bpfptr(uvalue, value_size); 1811 1809 if (IS_ERR(value)) { 1812 1810 err = PTR_ERR(value); ··· 2002 2000 void *key, *value; 2003 2001 int err = 0; 2004 2002 2005 - err = bpf_map_check_op_flags(map, attr->batch.elem_flags, BPF_F_LOCK); 2003 + err = bpf_map_check_op_flags(map, attr->batch.elem_flags, 2004 + BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS); 2006 2005 if (err) 2007 2006 return err; 2008 2007 2009 - value_size = bpf_map_value_size(map); 2008 + value_size = bpf_map_value_size(map, attr->batch.elem_flags); 2010 2009 2011 2010 max_count = attr->batch.count; 2012 2011 if (!max_count) ··· 2062 2059 u32 value_size, cp, max_count; 2063 2060 int err; 2064 2061 2065 - err = bpf_map_check_op_flags(map, attr->batch.elem_flags, BPF_F_LOCK); 2062 + err = bpf_map_check_op_flags(map, attr->batch.elem_flags, BPF_F_LOCK | BPF_F_CPU); 2066 2063 if (err) 2067 2064 return err; 2068 2065 2069 - value_size = bpf_map_value_size(map); 2066 + value_size = bpf_map_value_size(map, attr->batch.elem_flags); 2070 2067 2071 2068 max_count = attr->batch.count; 2072 2069 if (!max_count) ··· 2188 2185 goto err_put; 2189 2186 } 2190 2187 2191 - value_size = bpf_map_value_size(map); 2188 + value_size = bpf_map_value_size(map, 0); 2192 2189 2193 2190 err = -ENOMEM; 2194 2191 value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN);
+2
tools/include/uapi/linux/bpf.h
··· 1384 1384 BPF_NOEXIST = 1, /* create new element if it didn't exist */ 1385 1385 BPF_EXIST = 2, /* update existing element */ 1386 1386 BPF_F_LOCK = 4, /* spin_lock-ed map_lookup/map_update */ 1387 + BPF_F_CPU = 8, /* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ 1388 + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ 1387 1389 }; 1388 1390 1389 1391 /* flags for BPF_MAP_CREATE command */
+8
tools/lib/bpf/bpf.h
··· 289 289 * Update spin_lock-ed map elements. This must be 290 290 * specified if the map value contains a spinlock. 291 291 * 292 + * **BPF_F_CPU** 293 + * As for percpu maps, update value on the specified CPU. And the cpu 294 + * info is embedded into the high 32 bits of **opts->elem_flags**. 295 + * 296 + * **BPF_F_ALL_CPUS** 297 + * As for percpu maps, update value across all CPUs. This flag cannot 298 + * be used with BPF_F_CPU at the same time. 299 + * 292 300 * @param fd BPF map file descriptor 293 301 * @param keys pointer to an array of *count* keys 294 302 * @param values pointer to an array of *count* values
+20 -6
tools/lib/bpf/libbpf.c
··· 10919 10919 } 10920 10920 10921 10921 static int validate_map_op(const struct bpf_map *map, size_t key_sz, 10922 - size_t value_sz, bool check_value_sz) 10922 + size_t value_sz, bool check_value_sz, __u64 flags) 10923 10923 { 10924 10924 if (!map_is_created(map)) /* map is not yet created */ 10925 10925 return -ENOENT; ··· 10946 10946 int num_cpu = libbpf_num_possible_cpus(); 10947 10947 size_t elem_sz = roundup(map->def.value_size, 8); 10948 10948 10949 + if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) { 10950 + if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS)) { 10951 + pr_warn("map '%s': BPF_F_CPU and BPF_F_ALL_CPUS are mutually exclusive\n", 10952 + map->name); 10953 + return -EINVAL; 10954 + } 10955 + if (map->def.value_size != value_sz) { 10956 + pr_warn("map '%s': unexpected value size %zu provided for either BPF_F_CPU or BPF_F_ALL_CPUS, expected %u\n", 10957 + map->name, value_sz, map->def.value_size); 10958 + return -EINVAL; 10959 + } 10960 + break; 10961 + } 10962 + 10949 10963 if (value_sz != num_cpu * elem_sz) { 10950 10964 pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n", 10951 10965 map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz); ··· 10984 10970 { 10985 10971 int err; 10986 10972 10987 - err = validate_map_op(map, key_sz, value_sz, true); 10973 + err = validate_map_op(map, key_sz, value_sz, true, flags); 10988 10974 if (err) 10989 10975 return libbpf_err(err); 10990 10976 ··· 10997 10983 { 10998 10984 int err; 10999 10985 11000 - err = validate_map_op(map, key_sz, value_sz, true); 10986 + err = validate_map_op(map, key_sz, value_sz, true, flags); 11001 10987 if (err) 11002 10988 return libbpf_err(err); 11003 10989 ··· 11009 10995 { 11010 10996 int err; 11011 10997 11012 - err = validate_map_op(map, key_sz, 0, false /* check_value_sz */); 10998 + err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, flags); 11013 10999 if (err) 11014 11000 return libbpf_err(err); 11015 11001 ··· 11022 11008 { 11023 11009 int err; 11024 11010 11025 - err = validate_map_op(map, key_sz, value_sz, true); 11011 + err = validate_map_op(map, key_sz, value_sz, true, flags); 11026 11012 if (err) 11027 11013 return libbpf_err(err); 11028 11014 ··· 11034 11020 { 11035 11021 int err; 11036 11022 11037 - err = validate_map_op(map, key_sz, 0, false /* check_value_sz */); 11023 + err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0); 11038 11024 if (err) 11039 11025 return libbpf_err(err); 11040 11026
+8 -13
tools/lib/bpf/libbpf.h
··· 1216 1216 * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** 1217 1217 * @param value pointer to memory in which looked up value will be stored 1218 1218 * @param value_sz size in byte of value data memory; it has to match BPF map 1219 - * definition's **value_size**. For per-CPU BPF maps value size has to be 1220 - * a product of BPF map value size and number of possible CPUs in the system 1221 - * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for 1222 - * per-CPU values value size has to be aligned up to closest 8 bytes for 1223 - * alignment reasons, so expected size is: `round_up(value_size, 8) 1224 - * * libbpf_num_possible_cpus()`. 1219 + * definition's **value_size**. For per-CPU BPF maps, value size can be 1220 + * `value_size` if either **BPF_F_CPU** or **BPF_F_ALL_CPUS** is specified 1221 + * in **flags**, otherwise a product of BPF map value size and number of 1222 + * possible CPUs in the system (could be fetched with 1223 + * **libbpf_num_possible_cpus()**). Note also that for per-CPU values value 1224 + * size has to be aligned up to closest 8 bytes, so expected size is: 1225 + * `round_up(value_size, 8) * libbpf_num_possible_cpus()`. 1225 1226 * @param flags extra flags passed to kernel for this operation 1226 1227 * @return 0, on success; negative error, otherwise 1227 1228 * ··· 1240 1239 * @param key pointer to memory containing bytes of the key 1241 1240 * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** 1242 1241 * @param value pointer to memory containing bytes of the value 1243 - * @param value_sz size in byte of value data memory; it has to match BPF map 1244 - * definition's **value_size**. For per-CPU BPF maps value size has to be 1245 - * a product of BPF map value size and number of possible CPUs in the system 1246 - * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for 1247 - * per-CPU values value size has to be aligned up to closest 8 bytes for 1248 - * alignment reasons, so expected size is: `round_up(value_size, 8) 1249 - * * libbpf_num_possible_cpus()`. 1242 + * @param value_sz refer to **bpf_map__lookup_elem**'s description.' 1250 1243 * @param flags extra flags passed to kernel for this operation 1251 1244 * @return 0, on success; negative error, otherwise 1252 1245 *
+328
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c
··· 1 1 // SPDX-License-Identifier: GPL-2.0 2 2 #include <test_progs.h> 3 + #include "cgroup_helpers.h" 3 4 #include "percpu_alloc_array.skel.h" 4 5 #include "percpu_alloc_cgrp_local_storage.skel.h" 5 6 #include "percpu_alloc_fail.skel.h" ··· 116 115 RUN_TESTS(percpu_alloc_fail); 117 116 } 118 117 118 + static void test_percpu_map_op_cpu_flag(struct bpf_map *map, void *keys, size_t key_sz, u32 entries, 119 + int nr_cpus, bool test_batch) 120 + { 121 + size_t value_sz = sizeof(u32), value_sz_cpus, value_sz_total; 122 + u32 *values = NULL, *values_percpu = NULL; 123 + const u32 value = 0xDEADC0DE; 124 + int i, j, cpu, map_fd, err; 125 + u64 batch = 0, flags; 126 + void *values_row; 127 + u32 count, v; 128 + LIBBPF_OPTS(bpf_map_batch_opts, batch_opts); 129 + 130 + value_sz_cpus = value_sz * nr_cpus; 131 + values = calloc(entries, value_sz_cpus); 132 + if (!ASSERT_OK_PTR(values, "calloc values")) 133 + return; 134 + 135 + values_percpu = calloc(entries, roundup(value_sz, 8) * nr_cpus); 136 + if (!ASSERT_OK_PTR(values_percpu, "calloc values_percpu")) { 137 + free(values); 138 + return; 139 + } 140 + 141 + value_sz_total = value_sz_cpus * entries; 142 + memset(values, 0, value_sz_total); 143 + 144 + map_fd = bpf_map__fd(map); 145 + flags = BPF_F_CPU | BPF_F_ALL_CPUS; 146 + err = bpf_map_lookup_elem_flags(map_fd, keys, values, flags); 147 + if (!ASSERT_ERR(err, "bpf_map_lookup_elem_flags cpu|all_cpus")) 148 + goto out; 149 + 150 + err = bpf_map_update_elem(map_fd, keys, values, flags); 151 + if (!ASSERT_ERR(err, "bpf_map_update_elem cpu|all_cpus")) 152 + goto out; 153 + 154 + flags = BPF_F_ALL_CPUS; 155 + err = bpf_map_lookup_elem_flags(map_fd, keys, values, flags); 156 + if (!ASSERT_ERR(err, "bpf_map_lookup_elem_flags all_cpus")) 157 + goto out; 158 + 159 + flags = BPF_F_LOCK | BPF_F_CPU; 160 + err = bpf_map_lookup_elem_flags(map_fd, keys, values, flags); 161 + if (!ASSERT_ERR(err, "bpf_map_lookup_elem_flags BPF_F_LOCK")) 162 + goto out; 163 + 164 + flags = BPF_F_LOCK | BPF_F_ALL_CPUS; 165 + err = bpf_map_update_elem(map_fd, keys, values, flags); 166 + if (!ASSERT_ERR(err, "bpf_map_update_elem BPF_F_LOCK")) 167 + goto out; 168 + 169 + flags = (u64)nr_cpus << 32 | BPF_F_CPU; 170 + err = bpf_map_update_elem(map_fd, keys, values, flags); 171 + if (!ASSERT_EQ(err, -ERANGE, "bpf_map_update_elem -ERANGE")) 172 + goto out; 173 + 174 + err = bpf_map__update_elem(map, keys, key_sz, values, value_sz, flags); 175 + if (!ASSERT_EQ(err, -ERANGE, "bpf_map__update_elem -ERANGE")) 176 + goto out; 177 + 178 + err = bpf_map_lookup_elem_flags(map_fd, keys, values, flags); 179 + if (!ASSERT_EQ(err, -ERANGE, "bpf_map_lookup_elem_flags -ERANGE")) 180 + goto out; 181 + 182 + err = bpf_map__lookup_elem(map, keys, key_sz, values, value_sz, flags); 183 + if (!ASSERT_EQ(err, -ERANGE, "bpf_map__lookup_elem -ERANGE")) 184 + goto out; 185 + 186 + for (cpu = 0; cpu < nr_cpus; cpu++) { 187 + /* clear value on all cpus */ 188 + values[0] = 0; 189 + flags = BPF_F_ALL_CPUS; 190 + for (i = 0; i < entries; i++) { 191 + err = bpf_map__update_elem(map, keys + i * key_sz, key_sz, values, 192 + value_sz, flags); 193 + if (!ASSERT_OK(err, "bpf_map__update_elem all_cpus")) 194 + goto out; 195 + } 196 + 197 + /* update value on specified cpu */ 198 + for (i = 0; i < entries; i++) { 199 + values[0] = value; 200 + flags = (u64)cpu << 32 | BPF_F_CPU; 201 + err = bpf_map__update_elem(map, keys + i * key_sz, key_sz, values, 202 + value_sz, flags); 203 + if (!ASSERT_OK(err, "bpf_map__update_elem specified cpu")) 204 + goto out; 205 + 206 + /* lookup then check value on CPUs */ 207 + for (j = 0; j < nr_cpus; j++) { 208 + flags = (u64)j << 32 | BPF_F_CPU; 209 + err = bpf_map__lookup_elem(map, keys + i * key_sz, key_sz, values, 210 + value_sz, flags); 211 + if (!ASSERT_OK(err, "bpf_map__lookup_elem specified cpu")) 212 + goto out; 213 + if (!ASSERT_EQ(values[0], j != cpu ? 0 : value, 214 + "bpf_map__lookup_elem value on specified cpu")) 215 + goto out; 216 + } 217 + } 218 + } 219 + 220 + if (!test_batch) 221 + goto out; 222 + 223 + count = entries; 224 + batch_opts.elem_flags = (u64)nr_cpus << 32 | BPF_F_CPU; 225 + err = bpf_map_update_batch(map_fd, keys, values, &count, &batch_opts); 226 + if (!ASSERT_EQ(err, -ERANGE, "bpf_map_update_batch -ERANGE")) 227 + goto out; 228 + 229 + for (cpu = 0; cpu < nr_cpus; cpu++) { 230 + memset(values, 0, value_sz_total); 231 + 232 + /* clear values across all CPUs */ 233 + count = entries; 234 + batch_opts.elem_flags = BPF_F_ALL_CPUS; 235 + err = bpf_map_update_batch(map_fd, keys, values, &count, &batch_opts); 236 + if (!ASSERT_OK(err, "bpf_map_update_batch all_cpus")) 237 + goto out; 238 + 239 + /* update values on specified CPU */ 240 + for (i = 0; i < entries; i++) 241 + values[i] = value; 242 + 243 + count = entries; 244 + batch_opts.elem_flags = (u64)cpu << 32 | BPF_F_CPU; 245 + err = bpf_map_update_batch(map_fd, keys, values, &count, &batch_opts); 246 + if (!ASSERT_OK(err, "bpf_map_update_batch specified cpu")) 247 + goto out; 248 + 249 + /* lookup values on specified CPU */ 250 + batch = 0; 251 + count = entries; 252 + memset(values, 0, entries * value_sz); 253 + err = bpf_map_lookup_batch(map_fd, NULL, &batch, keys, values, &count, &batch_opts); 254 + if (!ASSERT_TRUE(!err || err == -ENOENT, "bpf_map_lookup_batch specified cpu")) 255 + goto out; 256 + 257 + for (i = 0; i < entries; i++) 258 + if (!ASSERT_EQ(values[i], value, 259 + "bpf_map_lookup_batch value on specified cpu")) 260 + goto out; 261 + 262 + /* lookup values from all CPUs */ 263 + batch = 0; 264 + count = entries; 265 + batch_opts.elem_flags = 0; 266 + memset(values_percpu, 0, roundup(value_sz, 8) * nr_cpus * entries); 267 + err = bpf_map_lookup_batch(map_fd, NULL, &batch, keys, values_percpu, &count, 268 + &batch_opts); 269 + if (!ASSERT_TRUE(!err || err == -ENOENT, "bpf_map_lookup_batch all_cpus")) 270 + goto out; 271 + 272 + for (i = 0; i < entries; i++) { 273 + values_row = (void *) values_percpu + 274 + roundup(value_sz, 8) * i * nr_cpus; 275 + for (j = 0; j < nr_cpus; j++) { 276 + v = *(u32 *) (values_row + roundup(value_sz, 8) * j); 277 + if (!ASSERT_EQ(v, j != cpu ? 0 : value, 278 + "bpf_map_lookup_batch value all_cpus")) 279 + goto out; 280 + } 281 + } 282 + } 283 + 284 + out: 285 + free(values_percpu); 286 + free(values); 287 + } 288 + 289 + 290 + static void test_percpu_map_cpu_flag(enum bpf_map_type map_type) 291 + { 292 + struct percpu_alloc_array *skel; 293 + size_t key_sz = sizeof(int); 294 + int *keys, nr_cpus, i, err; 295 + struct bpf_map *map; 296 + u32 max_entries; 297 + 298 + nr_cpus = libbpf_num_possible_cpus(); 299 + if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus")) 300 + return; 301 + 302 + max_entries = nr_cpus + 1; 303 + keys = calloc(max_entries, key_sz); 304 + if (!ASSERT_OK_PTR(keys, "calloc keys")) 305 + return; 306 + 307 + for (i = 0; i < max_entries; i++) 308 + keys[i] = i; 309 + 310 + skel = percpu_alloc_array__open(); 311 + if (!ASSERT_OK_PTR(skel, "percpu_alloc_array__open")) { 312 + free(keys); 313 + return; 314 + } 315 + 316 + map = skel->maps.percpu; 317 + bpf_map__set_type(map, map_type); 318 + bpf_map__set_max_entries(map, max_entries); 319 + 320 + err = percpu_alloc_array__load(skel); 321 + if (!ASSERT_OK(err, "test_percpu_alloc__load")) 322 + goto out; 323 + 324 + test_percpu_map_op_cpu_flag(map, keys, key_sz, max_entries - 1, nr_cpus, true); 325 + out: 326 + percpu_alloc_array__destroy(skel); 327 + free(keys); 328 + } 329 + 330 + static void test_percpu_array_cpu_flag(void) 331 + { 332 + test_percpu_map_cpu_flag(BPF_MAP_TYPE_PERCPU_ARRAY); 333 + } 334 + 335 + static void test_percpu_hash_cpu_flag(void) 336 + { 337 + test_percpu_map_cpu_flag(BPF_MAP_TYPE_PERCPU_HASH); 338 + } 339 + 340 + static void test_lru_percpu_hash_cpu_flag(void) 341 + { 342 + test_percpu_map_cpu_flag(BPF_MAP_TYPE_LRU_PERCPU_HASH); 343 + } 344 + 345 + static void test_percpu_cgroup_storage_cpu_flag(void) 346 + { 347 + struct percpu_alloc_array *skel = NULL; 348 + struct bpf_cgroup_storage_key key; 349 + int cgroup, prog_fd, nr_cpus, err; 350 + struct bpf_map *map; 351 + 352 + nr_cpus = libbpf_num_possible_cpus(); 353 + if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus")) 354 + return; 355 + 356 + err = setup_cgroup_environment(); 357 + if (!ASSERT_OK(err, "setup_cgroup_environment")) 358 + return; 359 + 360 + cgroup = create_and_get_cgroup("/cg_percpu"); 361 + if (!ASSERT_GE(cgroup, 0, "create_and_get_cgroup")) { 362 + cleanup_cgroup_environment(); 363 + return; 364 + } 365 + 366 + err = join_cgroup("/cg_percpu"); 367 + if (!ASSERT_OK(err, "join_cgroup")) 368 + goto out; 369 + 370 + skel = percpu_alloc_array__open_and_load(); 371 + if (!ASSERT_OK_PTR(skel, "percpu_alloc_array__open_and_load")) 372 + goto out; 373 + 374 + prog_fd = bpf_program__fd(skel->progs.cgroup_egress); 375 + err = bpf_prog_attach(prog_fd, cgroup, BPF_CGROUP_INET_EGRESS, 0); 376 + if (!ASSERT_OK(err, "bpf_prog_attach")) 377 + goto out; 378 + 379 + map = skel->maps.percpu_cgroup_storage; 380 + err = bpf_map_get_next_key(bpf_map__fd(map), NULL, &key); 381 + if (!ASSERT_OK(err, "bpf_map_get_next_key")) 382 + goto out; 383 + 384 + test_percpu_map_op_cpu_flag(map, &key, sizeof(key), 1, nr_cpus, false); 385 + out: 386 + bpf_prog_detach2(-1, cgroup, BPF_CGROUP_INET_EGRESS); 387 + close(cgroup); 388 + cleanup_cgroup_environment(); 389 + percpu_alloc_array__destroy(skel); 390 + } 391 + 392 + static void test_map_op_cpu_flag(enum bpf_map_type map_type) 393 + { 394 + u32 max_entries = 1, count = max_entries; 395 + u64 flags, batch = 0, val = 0; 396 + int err, map_fd, key = 0; 397 + LIBBPF_OPTS(bpf_map_batch_opts, batch_opts); 398 + 399 + map_fd = bpf_map_create(map_type, "test_cpu_flag", sizeof(int), sizeof(u64), max_entries, 400 + NULL); 401 + if (!ASSERT_GE(map_fd, 0, "bpf_map_create")) 402 + return; 403 + 404 + flags = BPF_F_ALL_CPUS; 405 + err = bpf_map_update_elem(map_fd, &key, &val, flags); 406 + ASSERT_ERR(err, "bpf_map_update_elem all_cpus"); 407 + 408 + batch_opts.elem_flags = BPF_F_ALL_CPUS; 409 + err = bpf_map_update_batch(map_fd, &key, &val, &count, &batch_opts); 410 + ASSERT_ERR(err, "bpf_map_update_batch all_cpus"); 411 + 412 + flags = BPF_F_CPU; 413 + err = bpf_map_lookup_elem_flags(map_fd, &key, &val, flags); 414 + ASSERT_ERR(err, "bpf_map_lookup_elem_flags cpu"); 415 + 416 + batch_opts.elem_flags = BPF_F_CPU; 417 + err = bpf_map_lookup_batch(map_fd, NULL, &batch, &key, &val, &count, &batch_opts); 418 + ASSERT_ERR(err, "bpf_map_lookup_batch cpu"); 419 + 420 + close(map_fd); 421 + } 422 + 423 + static void test_array_cpu_flag(void) 424 + { 425 + test_map_op_cpu_flag(BPF_MAP_TYPE_ARRAY); 426 + } 427 + 428 + static void test_hash_cpu_flag(void) 429 + { 430 + test_map_op_cpu_flag(BPF_MAP_TYPE_HASH); 431 + } 432 + 119 433 void test_percpu_alloc(void) 120 434 { 121 435 if (test__start_subtest("array")) ··· 441 125 test_cgrp_local_storage(); 442 126 if (test__start_subtest("failure_tests")) 443 127 test_failure(); 128 + if (test__start_subtest("cpu_flag_percpu_array")) 129 + test_percpu_array_cpu_flag(); 130 + if (test__start_subtest("cpu_flag_percpu_hash")) 131 + test_percpu_hash_cpu_flag(); 132 + if (test__start_subtest("cpu_flag_lru_percpu_hash")) 133 + test_lru_percpu_hash_cpu_flag(); 134 + if (test__start_subtest("cpu_flag_percpu_cgroup_storage")) 135 + test_percpu_cgroup_storage_cpu_flag(); 136 + if (test__start_subtest("cpu_flag_array")) 137 + test_array_cpu_flag(); 138 + if (test__start_subtest("cpu_flag_hash")) 139 + test_hash_cpu_flag(); 444 140 }
+32
tools/testing/selftests/bpf/progs/percpu_alloc_array.c
··· 187 187 return 0; 188 188 } 189 189 190 + struct { 191 + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); 192 + __uint(max_entries, 2); 193 + __type(key, int); 194 + __type(value, u32); 195 + } percpu SEC(".maps"); 196 + 197 + SEC("?fentry/bpf_fentry_test1") 198 + int BPF_PROG(test_percpu_array, int x) 199 + { 200 + u64 value = 0xDEADC0DE; 201 + int key = 0; 202 + 203 + bpf_map_update_elem(&percpu, &key, &value, BPF_ANY); 204 + return 0; 205 + } 206 + 207 + struct { 208 + __uint(type, BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE); 209 + __type(key, struct bpf_cgroup_storage_key); 210 + __type(value, u32); 211 + } percpu_cgroup_storage SEC(".maps"); 212 + 213 + SEC("cgroup_skb/egress") 214 + int cgroup_egress(struct __sk_buff *skb) 215 + { 216 + u32 *val = bpf_get_local_storage(&percpu_cgroup_storage, 0); 217 + 218 + *val = 1; 219 + return 1; 220 + } 221 + 190 222 char _license[] SEC("license") = "GPL";