tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

x86/csum: Remove unnecessary odd handling

The special case for odd aligned buffers is unnecessary and mostly
just adds overhead. Aligned buffers is the expectations, and even for
unaligned buffer, the only case that was helped is if the buffer was
1-byte from word aligned which is ~1/7 of the cases. Overall it seems
highly unlikely to be worth to extra branch.

It was left in the previous perf improvement patch because I was
erroneously comparing the exact output of `csum_partial(...)`, but
really we only need `csum_fold(csum_partial(...))` to match so its
safe to remove.

All csum kunit tests pass.

Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: David Laight <david.laight@aculab.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

authored by

Noah Goldstein and committed by
Linus Torvalds 5d4acb62 5eff55d7

+4 -32
+4 -32
arch/x86/lib/csum-partial_64.c
··· 11 11 #include <asm/checksum.h> 12 12 #include <asm/word-at-a-time.h> 13 13 14 - static inline unsigned short from32to16(unsigned a) 14 + static inline __wsum csum_finalize_sum(u64 temp64) 15 15 { 16 - unsigned short b = a >> 16; 17 - asm("addw %w2,%w0\n\t" 18 - "adcw $0,%w0\n" 19 - : "=r" (b) 20 - : "0" (b), "r" (a)); 21 - return b; 22 - } 23 - 24 - static inline __wsum csum_tail(u64 temp64, int odd) 25 - { 26 - unsigned int result; 27 - 28 - result = add32_with_carry(temp64 >> 32, temp64 & 0xffffffff); 29 - if (unlikely(odd)) { 30 - result = from32to16(result); 31 - result = ((result >> 8) & 0xff) | ((result & 0xff) << 8); 32 - } 33 - return (__force __wsum)result; 16 + return (__force __wsum)((temp64 + ror64(temp64, 32)) >> 32); 34 17 } 35 18 36 19 /* ··· 30 47 __wsum csum_partial(const void *buff, int len, __wsum sum) 31 48 { 32 49 u64 temp64 = (__force u64)sum; 33 - unsigned odd; 34 - 35 - odd = 1 & (unsigned long) buff; 36 - if (unlikely(odd)) { 37 - if (unlikely(len == 0)) 38 - return sum; 39 - temp64 = ror32((__force u32)sum, 8); 40 - temp64 += (*(unsigned char *)buff << 8); 41 - len--; 42 - buff++; 43 - } 44 50 45 51 /* 46 52 * len == 40 is the hot case due to IPv6 headers, but annotating it likely() ··· 45 73 "adcq $0,%[res]" 46 74 : [res] "+r"(temp64) 47 75 : [src] "r"(buff), "m"(*(const char(*)[40])buff)); 48 - return csum_tail(temp64, odd); 76 + return csum_finalize_sum(temp64); 49 77 } 50 78 if (unlikely(len >= 64)) { 51 79 /* ··· 115 143 : [res] "+r"(temp64) 116 144 : [trail] "r"(trail)); 117 145 } 118 - return csum_tail(temp64, odd); 146 + return csum_finalize_sum(temp64); 119 147 } 120 148 EXPORT_SYMBOL(csum_partial); 121 149