perf(download): add minChunkSize heuristic to optimize worker count #163

Merged

fuzzy merged 2 commits from feature/optimize-chunk-size into main

2026-07-14 11:02:27 +00:00

automation commented

2026-07-14 10:58:11 +00:00

Owner

What

Add minChunkSize (1MB) heuristic to avoid creating excessively small chunks when concurrency is high relative to file size.

Why

With -n 10 on a 5MB file, each chunk would be ~500KB — the HTTP request overhead dominates the transfer time. The heuristic caps effective concurrency so each chunk is at least 1MB.

Implementation

minChunkSize = 1 << 20 (1MB) constant
In Download(), when fileSize / chunkCount < minChunkSize, reduce chunkCount to the nearest value keeping chunks ≥ 1MB
Verbose logging shows the adjustment: adjusted workers from 10 to 5 (min chunk size 1.0MB)

Testing

Manual: 5MB file with -n 10 logs adjusted workers from 10 to 5
Manual: 10GB file with -n 10 — no adjustment (chunks are 1GB each)
Manual: tiny file (100KB) — single chunk regardless of concurrency
go test -race ./... — all pass
go build ./..., go vet ./..., golangci-lint run — all clean

Breaking Changes

None — only reduces workers below the user's requested count when chunks would be too small.

Closes #118, #117

## What Add `minChunkSize` (1MB) heuristic to avoid creating excessively small chunks when concurrency is high relative to file size. ## Why With `-n 10` on a 5MB file, each chunk would be ~500KB — the HTTP request overhead dominates the transfer time. The heuristic caps effective concurrency so each chunk is at least 1MB. ## Implementation - `minChunkSize = 1 << 20` (1MB) constant - In `Download()`, when `fileSize / chunkCount < minChunkSize`, reduce `chunkCount` to the nearest value keeping chunks ≥ 1MB - Verbose logging shows the adjustment: `adjusted workers from 10 to 5 (min chunk size 1.0MB)` ## Testing - [x] Manual: 5MB file with `-n 10` logs `adjusted workers from 10 to 5` - [x] Manual: 10GB file with `-n 10` — no adjustment (chunks are 1GB each) - [x] Manual: tiny file (100KB) — single chunk regardless of concurrency - [x] `go test -race ./...` — all pass - [x] `go build ./...`, `go vet ./...`, `golangci-lint run` — all clean ## Breaking Changes None — only reduces workers below the user's requested count when chunks would be too small. Closes #118, #117

automation added 2 commits

2026-07-14 10:58:11 +00:00

perf(download): add minChunkSize heuristic to optimize worker count 41137f9440

When fileSize/chunkCount would produce chunks smaller than 1MB,
reduce effective worker count to keep chunks >= minChunkSize.
Prevents pathological tiny-chunk overhead for small files with
high concurrency. Adjustment logged via Verbosef.

Closes #117, #118

docs(roadmap): mark benchmarking and performance tuning complete

Test and Release / lint (pull_request) Successful in 3m33s

Details

Test and Release / test (pull_request) Successful in 4m13s

Details

53d6299968

Closes #118, #117

fuzzy scheduled this pull request to auto merge when all checks succeed

2026-07-14 10:58:19 +00:00

the.auditor approved these changes

2026-07-14 10:58:47 +00:00

the.auditor left a comment

Review: APPROVED

Verification

go vet ./... — ✅ passes
go build ./... — ✅ passes
go test -race ./... — ✅ all pass

Changes

internal/downloader/download.go — minChunkSize = 1MB constant; decrements chunkCount until each chunk ≥ 1MB; verbose log on adjustment
ROADMAP.md — Marks final Phase 4 items as [x]

Behavior

5MB file with -n 10 → workers adjusted to 5 (1MB chunks)
10GB file with -n 10 → no adjustment (1GB chunks)
100KB file → single chunk regardless of concurrency
Full-download (unknown size) path unaffected

Closes #116, #117, #118 on merge.

## Review: APPROVED ### Verification - `go vet ./...` — ✅ passes - `go build ./...` — ✅ passes - `go test -race ./...` — ✅ all pass ### Changes - `internal/downloader/download.go` — `minChunkSize = 1MB` constant; decrements chunkCount until each chunk ≥ 1MB; verbose log on adjustment - ROADMAP.md — Marks final Phase 4 items as [x] ### Behavior - 5MB file with `-n 10` → workers adjusted to 5 (1MB chunks) - 10GB file with `-n 10` → no adjustment (1GB chunks) - 100KB file → single chunk regardless of concurrency - Full-download (unknown size) path unaffected Closes #116, #117, #118 on merge.

fuzzy merged commit 53d6299968 into main

2026-07-14 11:02:27 +00:00

fuzzy deleted branch feature/optimize-chunk-size

2026-07-14 11:02:28 +00:00