perf(download): reduce copy buffer size to 4KB, stream directly to disk #162

Merged

fuzzy merged 2 commits from feature/memory-optimization into main

2026-07-14 10:55:17 +00:00

automation commented

2026-07-14 09:54:45 +00:00

Owner

What

Reduce per-chunk copy buffer from 32KB (io.Copy default) to 4KB. Data already streams directly to disk with no in-memory accumulation.

Why

Lower memory footprint during parallel downloads. With 3 concurrent workers, peak buffer usage drops from 96KB to 12KB.

Implementation

copyBufSize = 4 * 1024 constant
runRangeDownload: io.CopyBuffer(ws, io.LimitReader(reader, expected), buf) replaces io.CopyN(ws, reader, expected) (which allocated 32KB internally)
runFullDownload: io.CopyBuffer(file, reader, buf) replaces io.Copy(file, reader)

Testing

go test -race ./... — all pass
go build ./..., go vet ./..., golangci-lint run — all clean

Breaking Changes

None

Closes #115, #113, #114

## What Reduce per-chunk copy buffer from 32KB (io.Copy default) to 4KB. Data already streams directly to disk with no in-memory accumulation. ## Why Lower memory footprint during parallel downloads. With 3 concurrent workers, peak buffer usage drops from 96KB to 12KB. ## Implementation - `copyBufSize = 4 * 1024` constant - `runRangeDownload`: `io.CopyBuffer(ws, io.LimitReader(reader, expected), buf)` replaces `io.CopyN(ws, reader, expected)` (which allocated 32KB internally) - `runFullDownload`: `io.CopyBuffer(file, reader, buf)` replaces `io.Copy(file, reader)` ## Testing - [x] `go test -race ./...` — all pass - [x] `go build ./...`, `go vet ./...`, `golangci-lint run` — all clean ## Breaking Changes None Closes #115, #113, #114

automation added 2 commits

2026-07-14 09:54:45 +00:00

perf(download): reduce copy buffer size to 4KB, stream directly to disk f6b6632a1c

Replace io.CopyN (32KB internal buffer) and io.Copy with io.CopyBuffer
using a 4KB buffer per chunk download. Data already streams from HTTP
response body directly to file via WriteAt — no accumulation in memory.

Closes #115, #113, #114

docs(roadmap): mark memory optimization complete

Test and Release / lint (pull_request) Successful in 3m54s

Details

Test and Release / test (pull_request) Successful in 4m14s

Details

b532ae3fa4

Closes #115, #113, #114

fuzzy scheduled this pull request to auto merge when all checks succeed

2026-07-14 10:54:57 +00:00

fuzzy self-assigned this

2026-07-14 10:55:01 +00:00

the.auditor approved these changes

2026-07-14 10:55:14 +00:00

the.auditor left a comment

Review: APPROVED

Verification

go vet ./... — ✅ passes
go build ./... — ✅ passes
go test -race ./... — ✅ all pass

Changes

internal/downloader/download.go — copyBufSize = 4KB constant; io.CopyBuffer replaces io.CopyN/io.Copy for explicit buffer control
ROADMAP.md — Marks Phase 4 memory optimization items as [x]

Memory impact

io.CopyN internally allocates 32KB buffer per call
io.CopyBuffer with 4KB = 8x reduction per-chunk
3 workers: 96KB → 12KB peak buffer

Closes #113, #114, #115 on merge.

## Review: APPROVED ### Verification - `go vet ./...` — ✅ passes - `go build ./...` — ✅ passes - `go test -race ./...` — ✅ all pass ### Changes - `internal/downloader/download.go` — `copyBufSize = 4KB` constant; `io.CopyBuffer` replaces `io.CopyN`/`io.Copy` for explicit buffer control - ROADMAP.md — Marks Phase 4 memory optimization items as [x] ### Memory impact - `io.CopyN` internally allocates 32KB buffer per call - `io.CopyBuffer` with 4KB = 8x reduction per-chunk - 3 workers: 96KB → 12KB peak buffer Closes #113, #114, #115 on merge.

fuzzy merged commit b532ae3fa4 into main

2026-07-14 10:55:17 +00:00

fuzzy deleted branch feature/memory-optimization

2026-07-14 10:55:18 +00:00