Batch small inputs into larger chunks #299

New issue

Closed

opened 2026-04-19 07:20:15 +00:00 by fuzzy · 1 comment

fuzzy commented

2026-04-19 07:20:15 +00:00

Owner

No description provided.

fuzzy added this to the Phase 6: Performance Optimization and Tuning milestone

2026-04-19 07:20:15 +00:00

fuzzy self-assigned this

2026-04-19 07:20:15 +00:00

fuzzy referenced this issue

2026-04-19 07:20:15 +00:00

Optimize buffer allocation strategies #300

fuzzy closed this issue

2026-04-19 15:43:44 +00:00

fuzzy commented

2026-04-19 15:44:53 +00:00

Author

Owner

Implemented in commit 0712bd8:

Added batchEnabled, minChunkSize, maxBatchSize configuration options
Enhanced ChunkSplitter to read at least minChunkSize bytes (unless EOF) and cap at maxBatchSize
Updated ParallelCompressor and ParallelDecompressor to propagate batching settings
Fixed decompression buffer sizing by passing chunkSize to DecompressionJob

The batching reduces per‑chunk overhead by aggregating small reads before compression, significantly improving throughput for many small reads.

Implemented in commit 0712bd8: - Added `batchEnabled`, `minChunkSize`, `maxBatchSize` configuration options - Enhanced `ChunkSplitter` to read at least `minChunkSize` bytes (unless EOF) and cap at `maxBatchSize` - Updated `ParallelCompressor` and `ParallelDecompressor` to propagate batching settings - Fixed decompression buffer sizing by passing `chunkSize` to `DecompressionJob` The batching reduces per‑chunk overhead by aggregating small reads before compression, significantly improving throughput for many small reads.

fuzzy referenced this issue from a commit

2026-04-19 15:45:06 +00:00

feat: implement input batching and comprehensive benchmarking suite