Validating, sanitizing, and safely hosting user-generated content is a large and complex topic. Today we’ll look at just a single aspect of it – using magic bytes to sniff the content type of user-uploaded files and reject ones that do not match our validation rules.
This post focuses on two things:
- How to sniff the content type without buffering an entire file in memory
- How to make the code ergonomic and reusable
Using http.DetectContentType
There is a http.DetectContentType function in the standard library that does exactly what we need. Here’s its full description from the documentation:
DetectContentType implements the algorithm described at https://mimesniff.spec.whatwg.org/ to determine the Content-Type of the given data. It considers at most the first 512 bytes of data. DetectContentType always returns a valid MIME type: if it cannot determine a more specific one, it returns “application/octet-stream”.
Let’s see how we could use this in practice. Consider a typical file upload handler that copies files directly to S3. We’ll use the AWS SDK for Go v2’s transfermanager package, with an already configured S3 client called s3Client:
func handleUpload(w http.ResponseWriter, r *http.Request) {
// Validate request headers
// ...
// Copy request body right into S3
uploader := transfermanager.New(s3Client)
_, err := uploader.UploadObject(r.Context(), &transfermanager.UploadObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String("filename.jpg"),
Body: r.Body,
})
if err != nil {
w.WriteHeader(500)
}
}
Let’s think about what it would look like if we used http.DetectContentType manually to only allow uploading images. Remember, we don’t want to buffer the entire file in memory:
// Peek at the first 512 bytes without consuming them
reqBody := bufio.NewReaderSize(r.Body, 512)
prefix, err := reqBody.Peek(512)
if err != nil && err != io.EOF {
w.WriteHeader(500)
return
}
// Detect and validate the content type
contentType := http.DetectContentType(prefix)
if !strings.HasPrefix(contentType, "image/") {
w.WriteHeader(400)
return
}
// Pass reqBody to the uploader. It still contains the complete body.
For files shorter than 512 bytes, Peek returns the available prefix and io.EOF. That’s enough for detection. Since peeking does not consume the bytes, subsequent reads from reqBody return the complete file.
This works. But now the handler has to know how many bytes to inspect, how to handle a short file, and which reader to pass to the uploader afterward.
What Would We Want Instead?
We’d like a helper that lets us define our own content type validation rules while abstracting away buffering and detection. In the snippet below, we wrap the request body with NewContentTypeReader and pass a callback that returns nil to accept the detected content type or an error to reject it. Here, we only accept types that start with image/.
var ErrNotImage = errors.New("not an image")
func handleUpload(w http.ResponseWriter, r *http.Request) {
// Validate request headers
// ...
// A custom reader that detects and validates the content type
reqBody := NewContentTypeReader(r.Body, func(contentType string) error {
if !strings.HasPrefix(contentType, "image/") {
return ErrNotImage
}
return nil
})
// Copy request body right into S3
uploader := transfermanager.New(s3Client)
_, err := uploader.UploadObject(r.Context(), &transfermanager.UploadObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String("filename.jpg"),
Body: reqBody,
})
if errors.Is(err, ErrNotImage) {
// Caught the error from the custom reader
w.WriteHeader(400)
return
}
if err != nil {
w.WriteHeader(500)
}
}
Making this helper a reader lets it fit right into the existing upload flow: the uploader reads the body, and validation failures come back through its usual error path. http.MaxBytesReader uses the same idea – it checks a size limit as the body is read and reports a violation as a read error.
If the callback rejects the content type, the first non-empty read from reqBody returns zero bytes and ErrNotImage. Subsequent reads return the same error, so the uploader never receives any rejected bytes.
Implementing the Reader Wrapper
The wrapper uses the same bufio.Reader.Peek approach, but defers it until the first non-empty Read. Once the callback accepts the content type, reads proceed through the buffered reader.
If a recoverable I/O error interrupts peeking, bufio.Reader retains the partial prefix so a later read can resume without losing any bytes.
type ctReader struct {
reader *bufio.Reader
handler func(contentType string) error
err error
}
// NewContentTypeReader sniffs and validates the content type before returning any bytes.
// A nil handler disables validation. A handler error permanently rejects the stream.
func NewContentTypeReader(r io.Reader, handler func(contentType string) error) io.Reader {
return &ctReader{
reader: bufio.NewReaderSize(r, 512),
handler: handler,
}
}
func (r *ctReader) Read(p []byte) (int, error) {
if r.err != nil {
return 0, r.err
}
if len(p) == 0 {
return 0, nil
}
if r.handler != nil {
prefix, err := r.reader.Peek(512)
if err != nil && err != io.EOF {
return 0, err
}
r.err = r.handler(http.DetectContentType(prefix))
r.handler = nil
if r.err != nil {
return 0, r.err
}
}
return r.reader.Read(p)
}
Returning zero bytes on rejection matters: consumers process returned bytes even when a read also returns an error. For example, io.Copy writes those bytes, and io.ReadFull can drop the error entirely if it has filled its destination buffer. Remembering the validation error also prevents later reads from continuing through a rejected stream.
Once the buffered prefix is consumed, large reads go directly to the underlying reader, so the small buffer doesn’t get in the way of the uploader.
Conclusion
We’ve built a small reader wrapper that lets our upload handler express the validation rule while keeping the I/O details out of business code. Remember that magic byte detection is just one layer of defense. For production systems, combine it with file size limits, malware scanning, and proper access controls.