Skip to content

Speed up tarfile.calc_chksums() #160991

Open

Description

@turtle-byte-17

Feature or enhancement

Proposal:

tarfile.calc_chksums() runs once per header block, every time a header is read
(TarInfo.frombuf) or written (TarInfo._create_header). It currently builds
two 504-element tuples with struct.unpack_from (one unsigned, one signed) and
sums each one in Python:

unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
return unsigned_chksum, signed_chksum

Both totals can be had from one C-level sum(buf): a byte b >= 128 adds b
to the unsigned sum and b - 256 to the signed one, so
signed == unsigned - 256 * (number of bytes >= 128). Almost every real header
is pure ASCII, in which case the two sums are equal and bytes.isascii()
detects that cheaply. Otherwise the high bytes are counted with
bytes.translate(None, ...).

The patch (+13 lines, no deletions) keeps the original code verbatim as the
fallback, and only takes the fast path when type(buf) is bytes and len(buf) == BLOCKSIZE, which is the only thing tarfile itself ever passes.
Any other input (memoryview, array, bytearray, other lengths) behaves exactly as
before, including raising struct.error below 512 bytes.

+_HIGH_BYTES = bytes(range(128, 256))
+
 def calc_chksums(buf):
     ...
+    if type(buf) is bytes and len(buf) == BLOCKSIZE:
+        # Fast path for the only input tarfile itself passes. The chksum
+        # field (buf[148:156]) counts as eight spaces, i.e. 256. A byte with
+        # the high bit set adds b to the unsigned sum and b - 256 to the
+        # signed one, so signed == unsigned - 256 * (number of such bytes).
+        unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
+        if buf.isascii():
+            return unsigned_chksum, unsigned_chksum
+        head = buf[:148] + buf[156:]
+        high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
+        return unsigned_chksum, unsigned_chksum - 256 * high
     unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
     signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
     return unsigned_chksum, signed_chksum

Benchmarks

The absolute gain per archive is small (milliseconds), so I'm not claiming
this fixes a bottleneck. It is a modest, free speedup on a function called
about once or twice per archive member (PAX archives carry an extra x
header per member).

calc_chksums alone, ASCII header, ns per call (stock / patched):

Python stock patched
3.11.10 4877 1027
3.14.10 3008 826
3.15.2 3474 827

Headers containing high bytes are about 2.6-3.3x faster.

End to end (tarfile.open + iterate + read every member, CPU time via
time.process_time, 25 ABBA-interleaved rounds, geometric mean of the
per-workload median speedups, stock / patched):

Python read list (getmembers) stream (r|) write
3.9 1.35x 1.51x 1.07x 1.40x
3.11 1.52x 1.79x 1.08x 1.63x
3.12 1.34x 1.52x 1.05x 1.38x
3.13 1.36x 1.54x 1.05x 1.38x
3.14 1.32x 1.50x 1.05x 1.33x
3.15 1.40x 1.61x 1.39x 1.38x
3.16.0a1+dev (built from main, 4a9b981) 1.34x 1.55x 1.34x 1.35x

Each cell aggregates 20 synthetic workload/operation rows: many small members,
few large members, ustar/gnu/pax formats, high-bit names, and plain/gz/bz2/xz
variants. No row's 95% bootstrap confidence interval reaches 1.0; the smallest
gain is about 1.02x (bz2 stream, where decompression dominates) and the
largest about 2.0x.

For scale, on a synthetic archive of 20,000 x 100-byte members, reading every
member went from 199 ms to 134 ms on the 3.16 build (1.48x), and getmembers()
from 158 ms to 94 ms. Archives with only a few hundred members save a few
milliseconds. Peak memory (tracemalloc) is unchanged to within 3 KB on 3.11,
3.15 and 3.16.

Verification

  • Exhaustive equivalence against the original function on 1,118,530 headers
    per interpreter (every byte value at every position of 7 base blocks, plus
    random, clustered-high-byte and k-high-byte buffers): 0 mismatches on 3.9,
    3.11, 3.12, 3.13, 3.14 and 3.15.
  • The full test_tarfile suite passes identically with and without the patch
    (3.11.10: 591 tests, 3.12.15: 633, 3.13.16: 666, 3.14.10: 768, and 795 on a
    3.16.0a1+dev build of main, with _lzma and _zstd available). test_shutil
    also passes identically on that build.
  • Differential run of stock vs patched tarfile over about 4.1 million cases
    (about 120 base archives including the test data, ustar/gnu/pax, Sun-style
    signed-checksum headers; every single-bit flip in the first headers,
    truncation at every length, header mutations, r:*, r:, r|,
    gz/bz2/xz, extractall): 0 differences. Archives written by the patched
    tarfile are byte-identical in every format. The corpus exercises the
    unsigned-match, signed-only-match and bad-checksum branches
    (about 2.09M, 3,238 and 200k headers in one run).
  • To check the harness could actually fail, I injected six deliberate bugs
    into the patch (signed == unsigned, checksum field not excluded, off-by-one
    high-byte count, wrong high-byte threshold, wrong head slice, inverted
    isascii); all six were caught.

Activity

  1. added
    type-featureA feature request or enhancement
    performancePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 31, 2026
  2. Anurag-S1ngh commented on Oct 31, 2026

    @Anurag-S1ngh

    Hi, I'd like to take this issue. Could you assign it to me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    ⌂ Proxy