Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

riscv, bpf: make some atomic operations fully ordered

The BPF atomic operations with the BPF_FETCH modifier along with
BPF_XCHG and BPF_CMPXCHG are fully ordered but the RISC-V JIT implements
all atomic operations except BPF_CMPXCHG with relaxed ordering.

Section 8.1 of the "The RISC-V Instruction Set Manual Volume I:
Unprivileged ISA" [1], titled, "Specifying Ordering of Atomic
Instructions" says:

| To provide more efficient support for release consistency [5], each
| atomic instruction has two bits, aq and rl, used to specify additional
| memory ordering constraints as viewed by other RISC-V harts.

and

| If only the aq bit is set, the atomic memory operation is treated as
| an acquire access.
| If only the rl bit is set, the atomic memory operation is treated as a
| release access.
|
| If both the aq and rl bits are set, the atomic memory operation is
| sequentially consistent.

Fix this by setting both aq and rl bits as 1 for operations with
BPF_FETCH and BPF_XCHG.

[1] https://riscv.org/wp-content/uploads/2017/05/riscv-spec-v2.2.pdf

Fixes: dd642ccb45ec ("riscv, bpf: Implement more atomic operations for RV64")
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/r/20240505201633.123115-1-puranjay@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

authored by

Puranjay Mohan and committed by
Alexei Starovoitov
20a759df 80c5a07a

+10 -10
+10 -10
arch/riscv/net/bpf_jit_comp64.c
··· 504 504 break; 505 505 /* src_reg = atomic_fetch_<op>(dst_reg + off16, src_reg) */ 506 506 case BPF_ADD | BPF_FETCH: 507 - emit(is64 ? rv_amoadd_d(rs, rs, rd, 0, 0) : 508 - rv_amoadd_w(rs, rs, rd, 0, 0), ctx); 507 + emit(is64 ? rv_amoadd_d(rs, rs, rd, 1, 1) : 508 + rv_amoadd_w(rs, rs, rd, 1, 1), ctx); 509 509 if (!is64) 510 510 emit_zextw(rs, rs, ctx); 511 511 break; 512 512 case BPF_AND | BPF_FETCH: 513 - emit(is64 ? rv_amoand_d(rs, rs, rd, 0, 0) : 514 - rv_amoand_w(rs, rs, rd, 0, 0), ctx); 513 + emit(is64 ? rv_amoand_d(rs, rs, rd, 1, 1) : 514 + rv_amoand_w(rs, rs, rd, 1, 1), ctx); 515 515 if (!is64) 516 516 emit_zextw(rs, rs, ctx); 517 517 break; 518 518 case BPF_OR | BPF_FETCH: 519 - emit(is64 ? rv_amoor_d(rs, rs, rd, 0, 0) : 520 - rv_amoor_w(rs, rs, rd, 0, 0), ctx); 519 + emit(is64 ? rv_amoor_d(rs, rs, rd, 1, 1) : 520 + rv_amoor_w(rs, rs, rd, 1, 1), ctx); 521 521 if (!is64) 522 522 emit_zextw(rs, rs, ctx); 523 523 break; 524 524 case BPF_XOR | BPF_FETCH: 525 - emit(is64 ? rv_amoxor_d(rs, rs, rd, 0, 0) : 526 - rv_amoxor_w(rs, rs, rd, 0, 0), ctx); 525 + emit(is64 ? rv_amoxor_d(rs, rs, rd, 1, 1) : 526 + rv_amoxor_w(rs, rs, rd, 1, 1), ctx); 527 527 if (!is64) 528 528 emit_zextw(rs, rs, ctx); 529 529 break; 530 530 /* src_reg = atomic_xchg(dst_reg + off16, src_reg); */ 531 531 case BPF_XCHG: 532 - emit(is64 ? rv_amoswap_d(rs, rs, rd, 0, 0) : 533 - rv_amoswap_w(rs, rs, rd, 0, 0), ctx); 532 + emit(is64 ? rv_amoswap_d(rs, rs, rd, 1, 1) : 533 + rv_amoswap_w(rs, rs, rd, 1, 1), ctx); 534 534 if (!is64) 535 535 emit_zextw(rs, rs, ctx); 536 536 break;