tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

crypto: x86/aes-gcm - fix PREEMPT_RT issue in gcm_crypt()

On PREEMPT_RT, kfree() takes sleeping locks and must not be called with
preemption disabled. Therefore, on PREEMPT_RT skcipher_walk_done() must
not be called from within a kernel_fpu_{begin,end}() pair, even when
it's the last call which is guaranteed to not allocate memory.

Therefore, move the last skcipher_walk_done() in gcm_crypt() to the end
of the function so that it goes after the kernel_fpu_end(). To make
this work cleanly, rework the data processing loop to handle only
non-last data segments.

Fixes: b06affb1cb58 ("crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM")
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Closes: https://lore.kernel.org/linux-crypto/20240802102333.itejxOsJ@linutronix.de
Signed-off-by: Eric Biggers <ebiggers@google.com>
Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

authored by

Eric Biggers and committed by
Herbert Xu 00141249 4b7acc85

+28 -31
+28 -31
arch/x86/crypto/aesni-intel_glue.c
··· 1366 1366 err = skcipher_walk_aead_encrypt(&walk, req, false); 1367 1367 else 1368 1368 err = skcipher_walk_aead_decrypt(&walk, req, false); 1369 + if (err) 1370 + return err; 1369 1371 1370 1372 /* 1371 1373 * Since the AES-GCM assembly code requires that at least three assembly ··· 1383 1381 gcm_process_assoc(key, ghash_acc, req->src, assoclen, flags); 1384 1382 1385 1383 /* En/decrypt the data and pass the ciphertext through GHASH. */ 1386 - while ((nbytes = walk.nbytes) != 0) { 1387 - if (unlikely(nbytes < walk.total)) { 1388 - /* 1389 - * Non-last segment. In this case, the assembly 1390 - * function requires that the length be a multiple of 16 1391 - * (AES_BLOCK_SIZE) bytes. The needed buffering of up 1392 - * to 16 bytes is handled by the skcipher_walk. Here we 1393 - * just need to round down to a multiple of 16. 1394 - */ 1395 - nbytes = round_down(nbytes, AES_BLOCK_SIZE); 1396 - aes_gcm_update(key, le_ctr, ghash_acc, 1397 - walk.src.virt.addr, walk.dst.virt.addr, 1398 - nbytes, flags); 1399 - le_ctr[0] += nbytes / AES_BLOCK_SIZE; 1400 - kernel_fpu_end(); 1401 - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); 1402 - kernel_fpu_begin(); 1403 - } else { 1404 - /* Last segment: process all remaining data. */ 1405 - aes_gcm_update(key, le_ctr, ghash_acc, 1406 - walk.src.virt.addr, walk.dst.virt.addr, 1407 - nbytes, flags); 1408 - err = skcipher_walk_done(&walk, 0); 1409 - /* 1410 - * The low word of the counter isn't used by the 1411 - * finalize, so there's no need to increment it here. 1412 - */ 1413 - } 1384 + while (unlikely((nbytes = walk.nbytes) < walk.total)) { 1385 + /* 1386 + * Non-last segment. In this case, the assembly function 1387 + * requires that the length be a multiple of 16 (AES_BLOCK_SIZE) 1388 + * bytes. The needed buffering of up to 16 bytes is handled by 1389 + * the skcipher_walk. Here we just need to round down to a 1390 + * multiple of 16. 1391 + */ 1392 + nbytes = round_down(nbytes, AES_BLOCK_SIZE); 1393 + aes_gcm_update(key, le_ctr, ghash_acc, walk.src.virt.addr, 1394 + walk.dst.virt.addr, nbytes, flags); 1395 + le_ctr[0] += nbytes / AES_BLOCK_SIZE; 1396 + kernel_fpu_end(); 1397 + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); 1398 + if (err) 1399 + return err; 1400 + kernel_fpu_begin(); 1414 1401 } 1415 - if (err) 1416 - goto out; 1402 + /* Last segment: process all remaining data. */ 1403 + aes_gcm_update(key, le_ctr, ghash_acc, walk.src.virt.addr, 1404 + walk.dst.virt.addr, nbytes, flags); 1405 + /* 1406 + * The low word of the counter isn't used by the finalize, so there's no 1407 + * need to increment it here. 1408 + */ 1417 1409 1418 1410 /* Finalize */ 1419 1411 taglen = crypto_aead_authsize(tfm); ··· 1435 1439 datalen, tag, taglen, flags)) 1436 1440 err = -EBADMSG; 1437 1441 } 1438 - out: 1439 1442 kernel_fpu_end(); 1443 + if (nbytes) 1444 + skcipher_walk_done(&walk, 0); 1440 1445 return err; 1441 1446 } 1442 1447