Streaming UTF-8 Validation in C Across Buffer Boundaries

A network read, file buffer or decompression callback can end halfway through a UTF-8 sequence. Validating each buffer as an independent string rejects that legitimate prefix. Waiting until the complete message is available solves the boundary problem by retaining all its bytes, which is unnecessary when the validator only needs a few fields of state.

This implementation separates feeding bytes from declaring the end of a message. Each feed continues the previous one. Finalization catches a sequence that was still waiting for bytes when the stream ended. The interface accepts a byte pointer and length, so an embedded zero byte does not terminate validation.

Define the accepted byte language

The accepted input follows RFC 3629's UTF-8 byte syntax. ASCII uses one byte; other sequences use two, three or four. Continuation bytes occupy the range 80 through BF. Leads C0, C1 and F5 through FF cannot begin a valid sequence.

Length alone is insufficient. The representation must be the shortest one for its value, exclude surrogate code points and stay at or below U+10FFFF. In particular, the first continuation is restricted after E0, ED, F0 and F4. The code checks those restrictions immediately and also verifies the completed value.

The API contract

Initialize one Utf8State per independent message. Supply consecutive chunks in order, then call utf8_finish after the final chunk. UTF8_OK from a feed means the bytes seen so far are a permissible prefix; it does not declare an incomplete message valid. Finalization is required.

The state pointer must refer to a live, initialized object. For a nonzero length, the data pointer must identify at least that many readable bytes. A null data pointer is permitted with length zero. The function neither allocates memory nor keeps the input pointer after it returns. Callers remain responsible for object bounds and for synchronizing access to a state object.

This C11 program explicitly requires eight-bit bytes. It checks whether the next chunk's length would overflow the cumulative size_t offset before reading the chunk. That condition returns UTF8_LIMIT and refuses the whole call. It is a bookkeeping limit, not a claim that the supplied bytes are malformed.

Complete C implementation and fixed-byte demo

The demo avoids source-file encoding assumptions by spelling out every input byte in hexadecimal. Compile the complete listing as a C11 program. When incorporating the API into another translation unit, separate the functions from the demonstration; the supplied UTF8_NO_MAIN guard also allows the test harness to include the same source.

#include <limits.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>

#if CHAR_BIT != 8
#error This validator requires 8-bit bytes.
#endif

typedef enum { UTF8_OK, UTF8_INVALID, UTF8_LIMIT } Utf8Status;

typedef struct {
    unsigned int needed;
    unsigned long codepoint, minimum;
    size_t offset, sequence_start, error_offset;
    Utf8Status status;
} Utf8State;

void utf8_init(Utf8State *s)
{
    *s = (Utf8State){0};
}

static Utf8Status utf8_fail(Utf8State *s, Utf8Status status, size_t at)
{
    s->status = status;
    s->error_offset = at;
    return status;
}

Utf8Status utf8_feed(Utf8State *s, const unsigned char *data, size_t len)
{
    size_t i;
    if (s->status != UTF8_OK)
        return s->status;
    if (len > SIZE_MAX - s->offset)
        return utf8_fail(s, UTF8_LIMIT, s->offset);

    for (i = 0; i < len; ++i) {
        unsigned int b = data[i];
        if (s->needed == 0) {
            s->sequence_start = s->offset;
            if (b <= 0x7F) {
                /* ASCII, including a zero byte, is valid. */
            } else if (b >= 0xC2 && b <= 0xDF) {
                s->codepoint = b & 0x1F; s->minimum = 0x80;
                s->needed = 1;
            } else if (b >= 0xE0 && b <= 0xEF) {
                s->codepoint = b & 0x0F; s->minimum = 0x800;
                s->needed = 2;
            } else if (b >= 0xF0 && b <= 0xF4) {
                s->codepoint = b & 0x07; s->minimum = 0x10000;
                s->needed = 3;
            } else {
                return utf8_fail(s, UTF8_INVALID, s->offset);
            }
        } else {
            if ((b & 0xC0) != 0x80)
                return utf8_fail(s, UTF8_INVALID, s->sequence_start);
            /* Restrict the first continuation after E0, ED, F0 and F4. */
            if ((s->minimum == 0x800 && s->needed == 2 &&
                 ((s->codepoint == 0 && b < 0xA0) ||
                  (s->codepoint == 0x0D && b > 0x9F))) ||
                (s->minimum == 0x10000 && s->needed == 3 &&
                 ((s->codepoint == 0 && b < 0x90) ||
                  (s->codepoint == 4 && b > 0x8F))))
                return utf8_fail(s, UTF8_INVALID, s->sequence_start);
            s->codepoint = (s->codepoint << 6) | (b & 0x3F);
            --s->needed;
            if (s->needed == 0 &&
                (s->codepoint < s->minimum || s->codepoint > 0x10FFFF ||
                 (s->codepoint >= 0xD800 && s->codepoint <= 0xDFFF)))
                return utf8_fail(s, UTF8_INVALID, s->sequence_start);
        }
        ++s->offset;
    }
    return UTF8_OK;
}

Utf8Status utf8_finish(Utf8State *s)
{
    if (s->status != UTF8_OK)
        return s->status;
    if (s->needed != 0)
        return utf8_fail(s, UTF8_INVALID, s->sequence_start);
    return UTF8_OK;
}

#ifndef UTF8_NO_MAIN
int main(void)
{
    const unsigned char first[] = {0x41, 0xF0};
    const unsigned char second[] = {0x9F, 0x98};
    const unsigned char third[] = {0x80, 0x42};
    const unsigned char bad[] = {0x41, 0xED, 0xA0, 0x80};
    const unsigned char tail[] = {0xE2, 0x82};
    Utf8State s;

    utf8_init(&s);
    utf8_feed(&s, first, sizeof first);
    utf8_feed(&s, second, sizeof second);
    utf8_feed(&s, third, sizeof third);
    puts(utf8_finish(&s) == UTF8_OK ? "split: valid" : "split: invalid");
    utf8_init(&s);
    utf8_feed(&s, bad, sizeof bad);
    if (utf8_finish(&s) == UTF8_INVALID)
        printf("surrogate: invalid at byte %zu\n", s.error_offset);
    utf8_init(&s);
    utf8_feed(&s, tail, sizeof tail);
    if (utf8_finish(&s) == UTF8_INVALID)
        printf("tail: invalid at byte %zu\n", s.error_offset);
    return 0;
}
#endif

Trace a sequence across three buffers

The valid demonstration represents ASCII A, the four bytes F0 9F 98 80, then ASCII B. The four-byte sequence is intentionally divided among three feeds. These are actual byte arrays, not hexadecimal text that the validator must parse.

Feed bytes	offset after	needed after	Accumulated value
41 F0	2	3	0x00, pending
9F 98	4	1	0x07D8, pending
80 42	6	0	U+1F600 completed; then ASCII B

needed records how many continuation bytes remain. codepoint accumulates payload bits, and minimum records the smallest value allowed for that sequence length. The partial value shifts left by six bits before the next continuation's six payload bits are added. With the accepted leads, this accumulator fits within the minimum range guaranteed for unsigned long.

Only continuation state survives a boundary; no previous buffer must remain alive. The next ASCII byte is recognized only after the sequence completes. The accumulated value and minimum are meaningful for a pending sequence; their old values can remain in the structure while needed is zero.

Errors and absolute offsets

offset counts consumed bytes from the beginning of the message, including bytes of a pending sequence. An invalid lead reports its own position. If an existing sequence becomes malformed, error_offset reports that sequence's starting byte. A truncated tail uses the same starting position during finalization. These are zero-based byte offsets, not character indexes.

Failure is sticky. Once the status becomes invalid or limited, later calls return that status and preserve the recorded error. Reinitialize to validate another message. This prevents a later chunk or a finalization call from accidentally erasing a previous failure.

Bytes	Reason for rejection	error_offset
C0 80	Forbidden lead for an overlong representation	0
E0 80 80	First continuation too small	0
ED A0 80	Surrogate encoding	0
F4 90 80 80	Value above U+10FFFF	0
80	Continuation without a lead	0
E2 82 then finish	Truncated sequence	0

The surrogate demonstration has an initial ASCII byte, so its malformed sequence begins at offset one. The truncated demonstration starts at zero. Running the exact listing prints:

split: valid
surrogate: invalid at byte 1
tail: invalid at byte 0

What the verification exercises

The listing compiled without diagnostics with TinyCC using C11 mode and warnings treated as errors. Its demonstration output matched execution. An independent harness compared validity and failure offsets with Python's strict UTF-8 decoder across 147,570 checks: every one-byte input, every two-byte input both whole and split, boundary vectors at every split position, and seeded random inputs under several chunk layouts.

The generated corpus mixed arbitrary bytes, correctly encoded scalar sequences and mutated encodings. Separate contract checks exercised empty feeds, reaching the largest representable offset, rejecting an overflowing chunk, sticky failures and truncated finalization. This provides repeatable evidence for the implementation; it does not constitute a proof over every possible message or every C environment.

Compile and run the demo, for example:

cc -std=c11 -Wall -Wextra -Wpedantic main.c -o utf8-demo
./utf8-demo

Validation is only one layer

The Unicode Standard's encoding and conformance definitions distinguish well-formed encoding from further text processing. This validator accepts well-formed encodings of scalar values, including unassigned values and noncharacters. It does not ask whether a character is useful or permitted by an application's identifier policy.

It also accepts an encoded BOM and zero bytes; an enclosing protocol may impose additional rules. It performs no normalization, case mapping, grapheme counting, display-width calculation or replacement of malformed input. Keep those decisions in separate stages with their own specifications.

For the C language requirements behind unsigned integers, shifts and integer limits, consult the public C11 committee draft N1570. For context on the blog's earlier string material, see strings in C; this byte-and-length interface deliberately does not rely on a terminating null byte.