Notes on software & other things

On-the-Fly Content Type Sniffing and Validation in Go

Validating, sanitizing, and safely hosting user-generated content is a large and complex topic. Today we’ll look at just a single aspect of it – using magic bytes to sniff the content type of user-uploaded files and reject ones that do not match our validation rules.

This post focuses on two things:

  • How to sniff the content type without buffering an entire file in memory
  • How to make the code ergonomic and reusable

Using http.DetectContentType

There is a http.DetectContentType function in the standard library that does exactly what we need. Here’s its full description from the documentation:

DetectContentType implements the algorithm described at https://mimesniff.spec.whatwg.org/ to determine the Content-Type of the given data. It considers at most the first 512 bytes of data. DetectContentType always returns a valid MIME type: if it cannot determine a more specific one, it returns “application/octet-stream”.

Let’s see how we could use this in practice. Consider a typical file upload handler that copies files directly to S3. We’ll use the AWS SDK for Go v2’s transfermanager package, with an already configured S3 client called s3Client:

func handleUpload(w http.ResponseWriter, r *http.Request) {
	// Validate request headers
	// ...

	// Copy request body right into S3
	uploader := transfermanager.New(s3Client)

	_, err := uploader.UploadObject(r.Context(), &transfermanager.UploadObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("filename.jpg"),
		Body:   r.Body,
	})
	if err != nil {
		w.WriteHeader(500)
	}
}

Let’s think about what it would look like if we used http.DetectContentType manually to only allow uploading images. Remember, we don’t want to buffer the entire file in memory:

	// Peek at the first 512 bytes without consuming them
	reqBody := bufio.NewReaderSize(r.Body, 512)
	prefix, err := reqBody.Peek(512)
	if err != nil && err != io.EOF {
		w.WriteHeader(500)
		return
	}

	// Detect and validate the content type
	contentType := http.DetectContentType(prefix)
	if !strings.HasPrefix(contentType, "image/") {
		w.WriteHeader(400)
		return
	}

	// Pass reqBody to the uploader. It still contains the complete body.

For files shorter than 512 bytes, Peek returns the available prefix and io.EOF. That’s enough for detection. Since peeking does not consume the bytes, subsequent reads from reqBody return the complete file.

This works. But now the handler has to know how many bytes to inspect, how to handle a short file, and which reader to pass to the uploader afterward.

What Would We Want Instead?

We’d like a helper that lets us define our own content type validation rules while abstracting away buffering and detection. In the snippet below, we wrap the request body with NewContentTypeReader and pass a callback that returns nil to accept the detected content type or an error to reject it. Here, we only accept types that start with image/.

var ErrNotImage = errors.New("not an image")

func handleUpload(w http.ResponseWriter, r *http.Request) {
	// Validate request headers
	// ...

	// A custom reader that detects and validates the content type
	reqBody := NewContentTypeReader(r.Body, func(contentType string) error {
		if !strings.HasPrefix(contentType, "image/") {
			return ErrNotImage
		}
		return nil
	})

	// Copy request body right into S3
	uploader := transfermanager.New(s3Client)

	_, err := uploader.UploadObject(r.Context(), &transfermanager.UploadObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("filename.jpg"),
		Body:   reqBody,
	})
	if errors.Is(err, ErrNotImage) {
		// Caught the error from the custom reader
		w.WriteHeader(400)
		return
	}
	if err != nil {
		w.WriteHeader(500)
	}
}

Making this helper a reader lets it fit right into the existing upload flow: the uploader reads the body, and validation failures come back through its usual error path. http.MaxBytesReader uses the same idea – it checks a size limit as the body is read and reports a violation as a read error.

If the callback rejects the content type, the first non-empty read from reqBody returns zero bytes and ErrNotImage. Subsequent reads return the same error, so the uploader never receives any rejected bytes.

Implementing the Reader Wrapper

The wrapper uses the same bufio.Reader.Peek approach, but defers it until the first non-empty Read. Once the callback accepts the content type, reads proceed through the buffered reader.

If a recoverable I/O error interrupts peeking, bufio.Reader retains the partial prefix so a later read can resume without losing any bytes.

type ctReader struct {
	reader  *bufio.Reader
	handler func(contentType string) error
	err     error
}

// NewContentTypeReader sniffs and validates the content type before returning any bytes.
// A nil handler disables validation. A handler error permanently rejects the stream.
func NewContentTypeReader(r io.Reader, handler func(contentType string) error) io.Reader {
	return &ctReader{
		reader:  bufio.NewReaderSize(r, 512),
		handler: handler,
	}
}

func (r *ctReader) Read(p []byte) (int, error) {
	if r.err != nil {
		return 0, r.err
	}
	if len(p) == 0 {
		return 0, nil
	}

	if r.handler != nil {
		prefix, err := r.reader.Peek(512)
		if err != nil && err != io.EOF {
			return 0, err
		}

		r.err = r.handler(http.DetectContentType(prefix))
		r.handler = nil
		if r.err != nil {
			return 0, r.err
		}
	}

	return r.reader.Read(p)
}

Returning zero bytes on rejection matters: consumers process returned bytes even when a read also returns an error. For example, io.Copy writes those bytes, and io.ReadFull can drop the error entirely if it has filled its destination buffer. Remembering the validation error also prevents later reads from continuing through a rejected stream.

Once the buffered prefix is consumed, large reads go directly to the underlying reader, so the small buffer doesn’t get in the way of the uploader.

Conclusion

We’ve built a small reader wrapper that lets our upload handler express the validation rule while keeping the I/O details out of business code. Remember that magic byte detection is just one layer of defense. For production systems, combine it with file size limits, malware scanning, and proper access controls.

MaxBytes Middleware in Go: The Same Trap, Again

A strict request body limit is a sensible default for an API, until a file upload endpoint needs more room. Adding a larger limit to that route looks like an easy fix, but uploads still hit the global limit. In this follow-up to the timeout middleware post, we’ll build MaxBytes middleware that supports per-route overrides.

Preserving Order in Concurrent Go Apps

Processing items concurrently in Go is easy enough: have a pool of goroutines read from one channel and write to another. Results arrive in completion order, which is fine until we need input order instead. Let’s build and benchmark three ways to make that happen with infinite stream support and no per-item allocations.

My Side Project: How Low Can You Go?

After a decade of Go backends, I wanted to build something where design, animation, and user experience came first. The result is a daily number game with a vintage cartoon style: pick the lowest number, but if someone else picks it too, you’re out.