SIMD kernels performance enhancements#135
Merged
Merged
Conversation
AES-CTR ran several times slower than the raw AES-NI core it shares with ECB: the SIC mode drove the 8-wide engine through three Pascal passes per 128 bytes (scalar counter generation into a stack buffer, AES on the scratch, then a separate wide XOR). Replace it with a single register-resident asm kernel: counter blocks built in-register (MOVQ/PINSRQ), 8-wide AES via the shared AesNiEightRoundsOnly chain, PXOR straight to output, looping internally. The kernel joins the fused-kernel family (IFusedCtrKernel + TAesNiCtrKernel, registered in TFusedKernelRegistry, force-linked via ClpFusedKernelDefaults); TSicBlockCipher acquires it via TryAcquireCtr like the AEAD modes. The AES engine stays a pure block transform - CTR borrows the encrypt key schedule through the existing TryGetEncKeysPtr. Rename the shared engine resolver TAesFusedAeadX86Backend -> TAesNiFusedX86Backend (now used by CTR too) with a CTR CPU gate (AES-NI + SSE4.1, no PCLMULQDQ). CTR throughput rises several-fold to near the raw 8-wide AES-NI rate (multiple GB/s, close to ECB). x86-64 only; i386 falls back to the existing bulk path. A new AESSICTests large-buffer case cross-checks the kernel against an independent ECB keystream reference across all key sizes and batch/tail sizes.
The fused AES-NI + PCLMULQDQ GCM kernel was invoked one 128-byte batch per call from Pascal, so counter generation, context-struct packing, and the xmm save/restore ran ~8192x per MB. Push the batch loop into the kernel: it now generates the GCM counters in-register, rotates the GHASH-lags-by-one pipeline internally, and advances the buffer/counter state in place across the whole run, leaving only the prime and drain to the mode. One kernel call replaces the per-batch orchestration. Both arches build the counter blocks in-register from the running 32-bit counter and the J0 template (x86_64 into xmm0..7; i386, with no spare GP registers, four-per-half into xmm0..3 via PINSRD), so no counter scratch buffer or memory round-trip is needed and the shared context record is identical in shape across arches. Collapse the now-redundant single-batch path: one context record (TGcmFusedCtx), one proc set, an unconditional-loop include on both arches, and ProcessCtrGhashBatch dropped from IFusedGcmKernel. Remove FillNextCtrBlocks8Raw, which the new driver orphaned.
…pans The fused POLYVAL fast path passed plaintext/AAD blocks to the GHASH-based kernel in natural byte order, but POLYVAL requires each block byte-reversed before the GHASH fold (RFC 8452 Appendix A) -- which the scalar path supplies via FillReverse. As a result any hashed span >= 128 bytes (>= 8 blocks, where the fused Horner-by-8 kernel engages) produced an incorrect tag and ciphertext on CPUs with the fused kernel, non-interoperable with other GCM-SIV implementations. Below 8 blocks the scalar path runs and output was correct. The defect was untested on two counts: every RFC 8452 known-answer vector is <= 64 bytes and so never reaches the 8-block batch, and the randomised round-trip cannot detect it -- encrypt and decrypt recompute the same wrong tag and still round-trip cleanly. Fix: byte-reverse each block into the batch scratch before invoking the fused kernel, mirroring the scalar FillReverse. Add GcmSivBulkParityTests, which cross-compares fused vs scalar ciphertext+tag across AAD/plaintext lengths straddling the 8-block boundary -- the comparison a round-trip cannot make.
…verse The GCM-SIV POLYVAL fast path reversed each block in Pascal before the shared GHASH batch kernel, which then reversed it again on load via PSHUFB -- a redundant round-trip. Give the shared GHASH batch body an optional GCM_GHASH_RAW_INPUT mode that skips its per-block input byte-reverse (the running-state load and final store stay byte-reversed), and have the GCM-SIV POLYVAL kernel select it and pass blocks in natural memory order. The kernel consumes them directly as field-form operands -- the same element the scalar path reaches -- with no Pascal pre-reverse and 8 fewer PSHUFB in the hot loop.
Rename the CBC encrypt kernel unit, interface, class, factory, and registry entry points to drop "Encrypt" so a decrypt body can be added later without another rename, mirroring the OCB and CCM kernels. ClpIFusedCbcEncryptKernel -> ClpIFusedCbcKernel ClpAesNiCbcEncryptKernel -> ClpAesNiCbcKernel IFusedCbcEncryptKernel[Factory], TAesNiCbcEncryptKernel[Factory], ProcessCbcEncryptBlocks, and the Register/Unregister/Snapshot/ GetRegistered/TryAcquire registry surface lose the "Encrypt" infix. The factory now takes a TFusedModeDirection and returns False for any direction other than Encrypt, so a decrypt request falls back to the per-block path until a decrypt kernel exists. The encrypt implementation itself is unchanged: the serial-chain include files, the fused context record, and the mode's bulk path stay direction-specific.
CCM and GCM-SIV each carried an identical private EnsureCapacity and open-coded the same grow + Move + advance-length triple at every buffer append. Lift the growth into TArrayUtilities.EnsureCapacity and add AppendTo overloads (single byte / source span) that grow, write at the used length, and advance it. Both modes now append in one call and drop their local EnsureCapacity; CCM's decrypt staging reuses the shared EnsureCapacity too. No behaviour change - same doubling growth, same never-shrink reuse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds fused AES-NI kernels that interleave the AES round chain with each
mode's auxiliary operation, cutting per-message overhead across the AEAD
suite. Every kernel ships for x86_64 and i386, gated on CPU detection with
a scalar fallback (behaviour identical across the fused/scalar paths).
Fused kernels
ntzcomputed in-kernel, one-shot span dispatchCorrectness
Refactors & utilities
TBytesaccumulators replace per-opTMemoryStream(drops the readback copy); shared
AppendTo/EnsureCapacityingestionTArrayUtilities.Fill: memset fast path + typedByte/UInt32/UInt64overloads
TByteXorX86Backend