diff options
| author | Eric Biggers <ebiggers@google.com> | 2025-02-19 10:23:41 -0800 |
|---|---|---|
| committer | Herbert Xu <herbert@gondor.apana.org.au> | 2025-03-02 15:19:44 +0800 |
| commit | 641938d3bba64287f199431fafd917bc7b7c9d1a (patch) | |
| tree | 003569b2d86dd9c9434b2aec491afecfd5ce2b26 /crypto/skcipher.c | |
| parent | fa94e45436c15421284c5cd24d497675b1f46433 (diff) | |
crypto: scatterwalk - don't split at page boundaries when !HIGHMEM
When !HIGHMEM, the kmap_local_page() in the scatterlist walker does not
actually map anything, and the address it returns is just the address
from the kernel's direct map, where each sg entry's data is virtually
contiguous. To improve performance, stop unnecessarily clamping data
segments to page boundaries in this case.
For now, still limit segments to PAGE_SIZE. This is needed to prevent
preemption from being disabled for too long when SIMD is used, and to
support the alignmask case which still uses a page-sized bounce buffer.
Even so, this change still helps a lot in cases where messages cross a
page boundary. For example, testing IPsec with AES-GCM on x86_64, the
messages are 1424 bytes which is less than PAGE_SIZE, but on the Rx side
over a third cross a page boundary. These ended up being processed in
three parts, with the middle part going through skcipher_next_slow which
uses a 16-byte bounce buffer. That was causing a significant amount of
overhead which unnecessarily reduced the performance benefit of the new
x86_64 AES-GCM assembly code. This change solves the problem; all these
messages now get passed to the assembly code in one part.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Diffstat (limited to 'crypto/skcipher.c')
| -rw-r--r-- | crypto/skcipher.c | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/crypto/skcipher.c b/crypto/skcipher.c index e9b803fe2441..b0e1f3c12cde 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -206,8 +206,8 @@ static int skcipher_next_fast(struct skcipher_walk *walk) diff = offset_in_page(walk->in.offset) - offset_in_page(walk->out.offset); - diff |= (u8 *)scatterwalk_page(&walk->in) - - (u8 *)scatterwalk_page(&walk->out); + diff |= (u8 *)(sg_page(walk->in.sg) + (walk->in.offset >> PAGE_SHIFT)) - + (u8 *)(sg_page(walk->out.sg) + (walk->out.offset >> PAGE_SHIFT)); skcipher_map_src(walk); walk->dst.virt.addr = walk->src.virt.addr; |