Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

bpf: Introduce lock-free bpf_async_update_prog_callback()

Introduce bpf_async_update_prog_callback(): lock-free update of cb->prog
and cb->callback_fn. This function allows updating prog and callback_fn
fields of the struct bpf_async_cb without holding lock.
For now use it under the lock from __bpf_async_set_callback(), in the
next patches that lock will be removed.

Lock-free algorithm:
* Acquire a guard reference on prog to prevent it from being freed
during the retry loop.
* Retry loop:
1. Each iteration acquires a new prog reference and stores it
in cb->prog via xchg. The previous prog is released.
2. The loop condition checks if both cb->prog and cb->callback_fn
match what we just wrote. If either differs, a concurrent writer
overwrote our value, and we must retry.
3. When we retry, our previously-stored prog was already released by
the concurrent writer or will be released by us after
overwriting.
* Release guard reference.

Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/r/20260120-timer_nolock-v6-3-670ffdd787b4@meta.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

authored by

Mykyta Yatsenko and committed by
Alexei Starovoitov
8bb1e32b 57d31e72

+37 -30
+37 -30
kernel/bpf/helpers.c
··· 1354 1354 .arg3_type = ARG_ANYTHING, 1355 1355 }; 1356 1356 1357 + static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callback_fn, 1358 + struct bpf_prog *prog) 1359 + { 1360 + struct bpf_prog *prev; 1361 + 1362 + /* Acquire a guard reference on prog to prevent it from being freed during the loop */ 1363 + if (prog) { 1364 + prog = bpf_prog_inc_not_zero(prog); 1365 + if (IS_ERR(prog)) 1366 + return PTR_ERR(prog); 1367 + } 1368 + 1369 + do { 1370 + if (prog) 1371 + prog = bpf_prog_inc_not_zero(prog); 1372 + prev = xchg(&cb->prog, prog); 1373 + rcu_assign_pointer(cb->callback_fn, callback_fn); 1374 + 1375 + /* 1376 + * Release previous prog, make sure that if other CPU is contending, 1377 + * to set bpf_prog, references are not leaked as each iteration acquires and 1378 + * releases one reference. 1379 + */ 1380 + if (prev) 1381 + bpf_prog_put(prev); 1382 + 1383 + } while (READ_ONCE(cb->prog) != prog || READ_ONCE(cb->callback_fn) != callback_fn); 1384 + 1385 + if (prog) 1386 + bpf_prog_put(prog); 1387 + 1388 + return 0; 1389 + } 1390 + 1357 1391 static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn, 1358 1392 struct bpf_prog *prog) 1359 1393 { 1360 - struct bpf_prog *prev; 1361 1394 struct bpf_async_cb *cb; 1362 1395 int ret = 0; 1363 1396 ··· 1411 1378 ret = -EPERM; 1412 1379 goto out; 1413 1380 } 1414 - prev = cb->prog; 1415 - if (prev != prog) { 1416 - /* Bump prog refcnt once. Every bpf_timer_set_callback() 1417 - * can pick different callback_fn-s within the same prog. 1418 - */ 1419 - prog = bpf_prog_inc_not_zero(prog); 1420 - if (IS_ERR(prog)) { 1421 - ret = PTR_ERR(prog); 1422 - goto out; 1423 - } 1424 - if (prev) 1425 - /* Drop prev prog refcnt when swapping with new prog */ 1426 - bpf_prog_put(prev); 1427 - cb->prog = prog; 1428 - } 1429 - rcu_assign_pointer(cb->callback_fn, callback_fn); 1381 + ret = bpf_async_update_prog_callback(cb, callback_fn, prog); 1430 1382 out: 1431 1383 __bpf_spin_unlock_irqrestore(&async->lock); 1432 1384 return ret; ··· 1471 1453 .arg3_type = ARG_ANYTHING, 1472 1454 }; 1473 1455 1474 - static void drop_prog_refcnt(struct bpf_async_cb *async) 1475 - { 1476 - struct bpf_prog *prog = async->prog; 1477 - 1478 - if (prog) { 1479 - bpf_prog_put(prog); 1480 - async->prog = NULL; 1481 - rcu_assign_pointer(async->callback_fn, NULL); 1482 - } 1483 - } 1484 - 1485 1456 BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) 1486 1457 { 1487 1458 struct bpf_hrtimer *t, *cur_t; ··· 1521 1514 goto out; 1522 1515 } 1523 1516 drop: 1524 - drop_prog_refcnt(&t->cb); 1517 + bpf_async_update_prog_callback(&t->cb, NULL, NULL); 1525 1518 out: 1526 1519 __bpf_spin_unlock_irqrestore(&timer->lock); 1527 1520 /* Cancel the timer and wait for associated callback to finish ··· 1554 1547 cb = async->cb; 1555 1548 if (!cb) 1556 1549 goto out; 1557 - drop_prog_refcnt(cb); 1550 + bpf_async_update_prog_callback(cb, NULL, NULL); 1558 1551 /* The subsequent bpf_timer_start/cancel() helpers won't be able to use 1559 1552 * this timer, since it won't be initialized. 1560 1553 */