From Multiple Atomics to Clean Progress Tracking

There was a period in my life when I often wrote scripts that processed or transferred data in parallel (one of such stories). Those scripts were too small to mess with proper monitoring and Grafana dashboards. Still, I needed a simple way to see if things were moving forward and at what rate, to be able to tweak various parameters.

Since my scripts were concurrent, I usually found myself writing code like this:

var successCnt, errorCnt, skippedCnt, inProgressCnt atomic.Int64

// In worker goroutines
successCnt.Add(1)
// ... somewhere else ...
errorCnt.Add(1)
// ... and again ...
skippedCnt.Add(1)

// Spawn a goroutine to print progress periodically
go func() {
    for ctx.Err() == nil {
        fmt.Printf("Success: %d, Errors: %d, Skipped: %d, In Progress: %d\n",
            successCnt.Load(), errorCnt.Load(), skippedCnt.Load(), inProgressCnt.Load(),
		)

		time.Sleep(5*time.Second)
    }
}()

This worked to some extent, but had several annoyances:

• The output wasn’t very readable, especially when there were many counters or values were long
• Each new script needed to repeat the same progress tracking and printing code
• It wasn’t convenient to add new counters

Instead, I wanted something that would:

• Create new counters dynamically on first use
• Be plug-and-play: no configuration, no boilerplate
• Print progress cleanly and in-place
• Preserve existing log output of the application (wouldn’t overwrite it with in-place updates)
• Be fast and lock-free

I haven’t considered fancy TUI frameworks for in-place updates. Usually I just launch such scripts in kuberenetes as one-off pods, and use kubectl logs -f to monitor them. So I needed something that just outputs progress directly to stdout or stderr.

A Better Way

Here’s what I came up with. This little demo shows how progress is updated in-place, while exiting log output stays preserved:

Code-wise, developers just need to create a dspc.Progress instance and use Inc function to increment/decrement named counters. The PrettyPrintEvery function periodically prints the values of all known counters.

// Create an instance and print progress to stdout every 100ms
var progress dspc.Progress
defer progress.PrettyPrintEvery(os.Stdout, 100*time.Millisecond, "Progress:")()

// In worker goroutines
progress.Inc("done", 1)
progress.Inc("errors", 1)
progress.Inc("errors[timeout]", 1)

After using this in several scripts, I found it useful and convenient enough to be published as an open-source package. It’s called dspc (dead simple progress counter) and is available on GitHub

Thread-Safety Without Locks

A few words about the implementation. It’s still a collection of atomic variables, but this time they’re created dynamically on demand and stored in a Go map. So when a counter is present in the map, incrementing it is just a single atomic operation.

The tricky part here is adding new counters to the map without using locks. The solution uses a combination of two patterns:

1. Copy-on-Write — when a new counter needs to be added, we create a complete copy of the map with the new counter added
2. Compare-and-Swap — we use atomic.CompareAndSwap to safely switch to the new version of the map

While this means adding a new counter requires copying the whole map, in practice this has a virtually zero cost since the number of different counter categories is very small in real-world scenarios.

This implementation is at least 2x faster than a mutex-protected map in concurrent scenarios. But what was even more surprising to me — in a single-threaded scenario it’s a bit faster than a raw, unprotected map[string]int.

Conclusion

While dspc is a small tool solving a specific problem, it demonstrates how a thoughtful abstraction can improve developer experience. The lock-free implementation using copy-on-write provides excellent performance characteristics due to the typically small number of distinct counters. The source code is available at https://github.com/destel/dspc. Feel free to use it in your scripts or adapt the copy-on-write pattern for similar scenarios.