C String strtok() Function

What You’ll Learn

The strtok() function breaks a C string into tokens using delimiter characters such as spaces, commas, or tabs. It is a classic tool for parsing command-line arguments, CSV fields, and log lines—once you understand that it modifies the original buffer.

Definition and Usage

strtok scans a modifiable string and returns pointers to consecutive tokens. Delimiter characters are replaced with '\0', so each token becomes its own null-terminated substring inside the same array.

The first call uses the string pointer. Every following call passes NULL so strtok continues where it left off. When no tokens remain, it returns NULL.

📝 Syntax

Standard C declaration:

char *strtok(char *str, const char *delimiters);

Parameters

• str — on the first call, the modifiable string to tokenize; on later calls, NULL.
• delimiters — C-string listing characters that separate tokens (space, comma, tab, etc.).

Return Value

• Pointer to the next token (null-terminated substring inside the original buffer).
• NULL when no more tokens are found.

Header

• #include <string.h>

Examples Gallery

Compile with gcc strtok.c -std=c11 -o strtok. Use a char[] buffer for every example that calls strtok directly.

#include <stdio.h>
#include <string.h>

int main(void) {
    char sentence[] = "This is a sample sentence.";
    const char *delimiters = " ";

    char *token = strtok(sentence, delimiters);

    while (token != NULL) {
        printf("Token: %s\n", token);
        token = strtok(NULL, delimiters);
    }

    return 0;
}

Token: This
Token: is
Token: a
Token: sample
Token: sentence.

#include <stdio.h>
#include <string.h>

int main(void) {
    char line[] = "apple, banana; cherry  date";

    char *token = strtok(line, " ,;");

    while (token != NULL) {
        printf("[%s]\n", token);
        token = strtok(NULL, " ,;");
    }

    return 0;
}

[apple]
[banana]
[cherry]
[date]

#include <stdio.h>
#include <string.h>

int main(void) {
    const char *original = "red|green|blue";
    char buffer[64];

    strcpy(buffer, original);

    char *color = strtok(buffer, "|");

    printf("Original still: %s\n", original);

    while (color != NULL) {
        printf("Color: %s\n", color);
        color = strtok(NULL, "|");
    }

    return 0;
}

Original still: red|green|blue
Color: red
Color: green
Color: blue

#include <stdio.h>
#include <string.h>

int main(void) {
    char record[] = "Mari,28,Chennai";
    char *field;
    int column = 0;

    field = strtok(record, ",");

    while (field != NULL) {
        printf("Column %d: %s\n", column++, field);
        field = strtok(NULL, ",");
    }

    return 0;
}

Column 0: Mari
Column 1: 28
Column 2: Chennai

#include <stdio.h>
#include <string.h>

int main(void) {
    char data[] = "  alpha  beta";
    const char *cursor = data;

    size_t skip = strspn(cursor, " \t");
    size_t len  = strcspn(cursor + skip, " \t");

    printf("Span token: %.*s\n", (int)len, cursor + skip);
    printf("Buffer unchanged mid-parse: [%s]\n", data);

    char *tok = strtok(data, " \t");
    printf("strtok first: %s\n", tok);
    printf("Buffer after strtok: [%s]\n", data);

    return 0;
}

Span token: alpha
Buffer unchanged mid-parse: [  alpha  beta]
strtok first: alpha
Buffer after strtok: [alpha\0 beta]

🚀 Common Use Cases

• Word splitting — break sentences or commands into individual tokens.
• CSV and config lines — parse key=value or comma-separated records.
• Path components — split on '/' or '\\' (simple cases).
• Log parsing — extract timestamp, level, and message fields.
• Teaching parsers — introduce tokenization before building full lexer logic.

📝 Notes

• Modifies the input buffer by inserting '\0' at delimiter positions.
• Requires a modifiable char[] buffer—not a string literal pointer.
• Uses internal static state; not safe across threads or nested tokenization of two strings.
• POSIX provides strtok_r() with an explicit context pointer for reentrancy.
• Consecutive delimiters produce no empty tokens; they are treated as one separator run.

⚡ Optimization

strtok is efficient for one-pass splitting of moderate strings. For complex parsing (quoted fields, nested delimiters), a dedicated parser or manual cursor with strspn/strcspn scales better. Copy the source once if you need both tokens and the original layout.

💡 Best Practices