tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

locking/atomic: Add generic support for sync_try_cmpxchg() and its fallback

Provide the generic sync_try_cmpxchg() function from the
raw_ prefixed version, also adding explicit instrumentation.

The patch amends existing scripts to generate sync_try_cmpxchg()
locking primitive and its raw_sync_try_cmpxchg() fallback, while
leaving existing macros from the try_cmpxchg() family unchanged.

The target can define its own arch_sync_try_cmpxchg() to override the
generic version of raw_sync_try_cmpxchg(). This allows the target
to generate more optimal assembly than the generic version.

Additionally, the patch renames two scripts to better reflect
whet they really do.

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org

authored by

Uros Bizjak and committed by
Ingo Molnar e01cc1e8 fdb8b7a1

+43 -18
+14 -1
include/linux/atomic/atomic-arch-fallback.h
··· 428 428 429 429 #define raw_sync_cmpxchg arch_sync_cmpxchg 430 430 431 + #ifdef arch_sync_try_cmpxchg 432 + #define raw_sync_try_cmpxchg arch_sync_try_cmpxchg 433 + #else 434 + #define raw_sync_try_cmpxchg(_ptr, _oldp, _new) \ 435 + ({ \ 436 + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ 437 + ___r = raw_sync_cmpxchg((_ptr), ___o, (_new)); \ 438 + if (unlikely(___r != ___o)) \ 439 + *___op = ___r; \ 440 + likely(___r == ___o); \ 441 + }) 442 + #endif 443 + 431 444 /** 432 445 * raw_atomic_read() - atomic load with relaxed ordering 433 446 * @v: pointer to atomic_t ··· 4662 4649 } 4663 4650 4664 4651 #endif /* _LINUX_ATOMIC_FALLBACK_H */ 4665 - // 2fdd6702823fa842f9cea57a002e6e4476ae780c 4652 + // eec048affea735b8464f58e6d96992101f8f85f1
+9 -1
include/linux/atomic/atomic-instrumented.h
··· 4998 4998 raw_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ 4999 4999 }) 5000 5000 5001 + #define sync_try_cmpxchg(ptr, ...) \ 5002 + ({ \ 5003 + typeof(ptr) __ai_ptr = (ptr); \ 5004 + kcsan_mb(); \ 5005 + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ 5006 + raw_sync_try_cmpxchg(__ai_ptr, __VA_ARGS__); \ 5007 + }) 5008 + 5001 5009 5002 5010 #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ 5003 - // 1568f875fef72097413caab8339120c065a39aa4 5011 + // 2cc4bc990fef44d3836ec108f11b610f3f438184
+18 -15
scripts/atomic/gen-atomic-fallback.sh
··· 223 223 224 224 gen_try_cmpxchg_fallback() 225 225 { 226 + local prefix="$1"; shift 226 227 local cmpxchg="$1"; shift; 227 - local order="$1"; shift; 228 + local suffix="$1"; shift; 228 229 229 230 cat <<EOF 230 - #define raw_try_${cmpxchg}${order}(_ptr, _oldp, _new) \\ 231 + #define raw_${prefix}try_${cmpxchg}${suffix}(_ptr, _oldp, _new) \\ 231 232 ({ \\ 232 233 typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \\ 233 - ___r = raw_${cmpxchg}${order}((_ptr), ___o, (_new)); \\ 234 + ___r = raw_${prefix}${cmpxchg}${suffix}((_ptr), ___o, (_new)); \\ 234 235 if (unlikely(___r != ___o)) \\ 235 236 *___op = ___r; \\ 236 237 likely(___r == ___o); \\ ··· 260 259 fi 261 260 262 261 printf "#else\n" 263 - gen_try_cmpxchg_fallback "${cmpxchg}" "${order}" 262 + gen_try_cmpxchg_fallback "" "${cmpxchg}" "${order}" 264 263 printf "#endif\n\n" 265 264 } 266 265 267 - gen_try_cmpxchg_fallbacks() 266 + gen_try_cmpxchg_order_fallbacks() 268 267 { 269 268 local cmpxchg="$1"; shift; 270 269 ··· 273 272 done 274 273 } 275 274 276 - gen_cmpxchg_local_fallbacks() 275 + gen_def_and_try_cmpxchg_fallback() 277 276 { 277 + local prefix="$1"; shift 278 278 local cmpxchg="$1"; shift 279 + local suffix="$1"; shift 279 280 280 - printf "#define raw_${cmpxchg} arch_${cmpxchg}\n\n" 281 - printf "#ifdef arch_try_${cmpxchg}\n" 282 - printf "#define raw_try_${cmpxchg} arch_try_${cmpxchg}\n" 281 + printf "#define raw_${prefix}${cmpxchg}${suffix} arch_${prefix}${cmpxchg}${suffix}\n\n" 282 + printf "#ifdef arch_${prefix}try_${cmpxchg}${suffix}\n" 283 + printf "#define raw_${prefix}try_${cmpxchg}${suffix} arch_${prefix}try_${cmpxchg}${suffix}\n" 283 284 printf "#else\n" 284 - gen_try_cmpxchg_fallback "${cmpxchg}" "" 285 + gen_try_cmpxchg_fallback "${prefix}" "${cmpxchg}" "${suffix}" 285 286 printf "#endif\n\n" 286 287 } 287 288 ··· 305 302 done 306 303 307 304 for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do 308 - gen_try_cmpxchg_fallbacks "${cmpxchg}" 305 + gen_try_cmpxchg_order_fallbacks "${cmpxchg}" 309 306 done 310 307 311 - for cmpxchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local"; do 312 - gen_cmpxchg_local_fallbacks "${cmpxchg}" "" 308 + for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do 309 + gen_def_and_try_cmpxchg_fallback "" "${cmpxchg}" "_local" 313 310 done 314 311 315 - for cmpxchg in "sync_cmpxchg"; do 316 - printf "#define raw_${cmpxchg} arch_${cmpxchg}\n\n" 312 + for cmpxchg in "cmpxchg"; do 313 + gen_def_and_try_cmpxchg_fallback "sync_" "${cmpxchg}" "" 317 314 done 318 315 319 316 grep '^[a-z]' "$1" | while read name meta args; do
+2 -1
scripts/atomic/gen-atomic-instrumented.sh
··· 169 169 done 170 170 done 171 171 172 - for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do 172 + for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" \ 173 + "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local" "sync_try_cmpxchg"; do 173 174 gen_xchg "${xchg}" "" 174 175 printf "\n" 175 176 done