Handling Sequence Data with tf.keras.preprocessing.sequence

A model cannot learn directly from words; it needs a stable mapping from tokens to integer IDs. Keras’s TextVectorization layer builds that mapping and applies the same text normalization and tokenization at training and inference time.

import tensorflow as tf
from tensorflow import keras

texts = tf.constant([
    "Small functions are easier to test.",
    "Clear names make code easier to read.",
    "Tests make refactoring safer.",
])

vectorizer = keras.layers.TextVectorization(
    output_mode="int",
    split="whitespace",
)

vectorizer.adapt(texts)

sequences = vectorizer(texts)
print(sequences.numpy())

With integer output, ID 0 is reserved for padding and ID 1 represents an out-of-vocabulary token. Ordinary vocabulary entries begin at 2. A sequence such as [5, 8, 3] therefore means “the fifth, eighth, and third vocabulary entries,” not that the words have any numeric relationship.

Inspect the vocabulary when debugging. The order determines the IDs, so saving the vectorization layer with the model is safer than rebuilding a mapping separately in application code.

vocabulary = vectorizer.get_vocabulary()

for token_id, token in enumerate(vocabulary[:12]):
    print(token_id, repr(token))

Adapt the layer on training text only. If validation or test text contributes tokens during adaptation, information from those splits has leaked into preprocessing, even if the labels remain untouched.

train_texts = tf.constant([
    "A parser turns text into tokens.",
    "Tokens can be mapped to integer IDs.",
])

vectorizer = keras.layers.TextVectorization(output_mode="int")
vectorizer.adapt(train_texts)

train_ids = vectorizer(train_texts)
validation_ids = vectorizer(
    tf.constant(["An unseen word still receives an ID."])
)

Words absent from the adapted vocabulary map to the same out-of-vocabulary ID. Set max_tokens when you need a bounded vocabulary; less frequent tokens then share that fallback ID rather than expanding the model’s input space indefinitely.

Building a vocabulary that fits

A vocabulary limit is a modeling choice, not just a memory setting. Common words keep distinct IDs; rare words collapse to the out-of-vocabulary ID. A smaller vocabulary reduces the size of the embedding table, but can erase distinctions that matter in a specialized corpus.

vocabulary_limit = 10_000

vectorizer = keras.layers.TextVectorization(
    max_tokens=vocabulary_limit,
    output_mode="int",
    standardize="lower_and_strip_punctuation",
    split="whitespace",
)
vectorizer.adapt(train_texts)

actual_vocabulary_size = len(vectorizer.get_vocabulary())
print(actual_vocabulary_size)

max_tokens includes reserved entries, so the number of ordinary tokens is smaller than the limit. Use the vocabulary the layer actually learned when sizing a downstream embedding; do not assume every permitted slot was filled.

vocabulary_size = len(vectorizer.get_vocabulary())

model = keras.Sequential([
    vectorizer,
    keras.layers.Embedding(
        input_dim=vocabulary_size,
        output_dim=64,
        mask_zero=True,
    ),
    keras.layers.GlobalAveragePooling1D(),
    keras.layers.Dense(1, activation="sigmoid"),
])

For a controlled domain, a fixed vocabulary can make the mapping explicit and reproducible. Tokens not listed still map to the out-of-vocabulary entry, so keep the normalization and tokenization rules consistent with the vocabulary’s contents.

domain_vocabulary = [
    "compile",
    "exception",
    "function",
    "module",
    "parser",
    "syntax",
]

vectorizer = keras.layers.TextVectorization(
    vocabulary=domain_vocabulary,
    output_mode="int",
    standardize="lower_and_strip_punctuation",
    split="whitespace",
)

When inspecting a learned vocabulary, remember that its order reflects token frequency, not alphabetical order or semantic importance. If two runs must produce identical IDs, adapt with the same training data and configuration, then save the fitted layer or its vocabulary alongside the model.

Padding sequences without guesswork

Padding gives a batch of sequences a rectangular shape. With integer output, TextVectorization uses 0 for padding, so a short sequence can be extended without inventing another token ID. The simplest fixed-shape option is output_sequence_length.

vectorizer = keras.layers.TextVectorization(
    max_tokens=10_000,
    output_mode="int",
    output_sequence_length=128,
)
vectorizer.adapt(train_texts)

batch = vectorizer(train_texts[:3])
print(batch.shape)
print(batch.numpy())

The chosen length should come from the data, not from a convenient round number. Measure tokenized sequence lengths on the training split and inspect their distribution. A percentile is often a practical starting point: it preserves that share of examples at their full length while keeping most batches compact.

import numpy as np

probe = keras.layers.TextVectorization(
    max_tokens=10_000,
    output_mode="int",
)
probe.adapt(train_texts)

encoded = probe(train_texts)
lengths = tf.reduce_sum(
    tf.cast(encoded != 0, tf.int32),
    axis=1,
).numpy()

for percentile in (50, 90, 95, 99):
    print(percentile, int(np.percentile(lengths, percentile)))

Use the measured value when constructing the production layer, then adapt it on the same training text and with the same tokenization settings. The fixed length controls the output shape; examples longer than it are clipped, so choose the cutoff with the next section’s truncation behavior in mind.

sequence_length = 128  # Set this from the training-length distribution.

vectorizer = keras.layers.TextVectorization(
    max_tokens=10_000,
    output_mode="int",
    output_sequence_length=sequence_length,
)
vectorizer.adapt(train_texts)

Padding is harmless only when downstream layers know to ignore it. An embedding layer can produce a mask for ID 0; sequence layers that support masks can then skip padded positions instead of treating them as real words.

model = keras.Sequential([
    vectorizer,
    keras.layers.Embedding(
        input_dim=len(vectorizer.get_vocabulary()),
        output_dim=64,
        mask_zero=True,
    ),
    keras.layers.Bidirectional(
        keras.layers.LSTM(32),
    ),
    keras.layers.Dense(1, activation="sigmoid"),
])

For a simple average over token embeddings, use a mask-aware pooling layer. Otherwise, zero-padded positions can affect the denominator and dilute the representation of shorter examples.

model = keras.Sequential([
    vectorizer,
    keras.layers.Embedding(
        input_dim=len(vectorizer.get_vocabulary()),
        output_dim=64,
        mask_zero=True,
    ),
    keras.layers.GlobalAveragePooling1D(),
    keras.layers.Dense(1, activation="sigmoid"),
])

Truncating long sequences with care

A fixed sequence length is also a deletion rule. If a document exceeds the limit, some tokens disappear; the model cannot distinguish “not safe” from “safe” if truncation removes the negation. The important question is not merely how many tokens fit, but which part of the example survives.

output_sequence_length makes the shape convenient, but its truncation behavior is implicit in the layer’s implementation. Check it with a deliberately long, recognizable example rather than assuming which end is retained.

probe = keras.layers.TextVectorization(
    output_mode="int",
    output_sequence_length=4,
)
probe.adapt(tf.constant(["alpha beta gamma delta epsilon zeta"]))

print(probe(tf.constant(["alpha beta gamma delta epsilon zeta"])).numpy())
print(probe.get_vocabulary())

For ordinary documents, retaining the beginning may preserve a title or opening summary; for logs, conversations, and many event streams, the newest tokens may matter more. Make the choice explicit by requesting ragged integer sequences, then padding them with the desired truncation direction.

sequence_length = 128

vectorizer = keras.layers.TextVectorization(
    max_tokens=10_000,
    output_mode="int",
    ragged=True,
)
vectorizer.adapt(train_texts)

ragged_ids = vectorizer(train_texts)
rows = ragged_ids.to_list()

# Keep the most recent tokens; pad shorter rows on the right.
sequences = keras.utils.pad_sequences(
    rows,
    maxlen=sequence_length,
    padding="post",
    truncating="pre",
    value=0,
)

print(sequences.shape)

Use truncating="post" instead when the start of each sequence is the important part. The padding side and truncation side are separate decisions: right-padding adds zeros at the end, while pre-truncation removes tokens from the beginning. Keep the padding ID at zero so embedding masks continue to work.

sequences = keras.utils.pad_sequences(
    rows,
    maxlen=sequence_length,
    padding="post",
    truncating="post",
    value=0,
)

For long documents, cutting off everything outside one window can discard useful evidence. Split documents into overlapping windows and associate the original label with each window when that label applies locally. For document-level labels, aggregate window predictions or representations at the document level rather than quietly treating every chunk as an independent example.

def make_windows(token_ids, window_size, stride):
    return [
        token_ids[start:start + window_size]
        for start in range(0, max(len(token_ids) - window_size + 1, 1), stride)
        if token_ids[start:start + window_size]
    ]

windows = make_windows(rows[0], window_size=128, stride=96)

Overlapping windows avoid losing a phrase that crosses a boundary, but they also create correlated training examples. Keep all windows from one source document in the same data split; splitting after windowing can place near-duplicate text in both training and validation data.

Putting preprocessing into a Keras pipeline

Keep the vectorizer inside the model when you want training and inference to use exactly the same tokenization, vocabulary, and sequence length. Adapt it on training text first, then connect it to a model that accepts strings rather than precomputed integer arrays.

sequence_length = 128

vectorizer = keras.layers.TextVectorization(
    max_tokens=10_000,
    output_mode="int",
    output_sequence_length=sequence_length,
)
vectorizer.adapt(train_texts)

inputs = keras.Input(shape=(), dtype=tf.string, name="text")
token_ids = vectorizer(inputs)
x = keras.layers.Embedding(
    input_dim=len(vectorizer.get_vocabulary()),
    output_dim=64,
    mask_zero=True,
)(token_ids)
x = keras.layers.GlobalAveragePooling1D()(x)
outputs = keras.layers.Dense(1, activation="sigmoid")(x)

model = keras.Model(inputs, outputs)
model.compile(
    optimizer="adam",
    loss="binary_crossentropy",
    metrics=["accuracy"],
)

The dataset can now contain raw strings and labels. Batching and prefetching still apply normally; the model performs vectorization as it processes each batch.

train_ds = (
    tf.data.Dataset.from_tensor_slices((train_texts, train_labels))
    .shuffle(buffer_size=len(train_texts))
    .batch(32)
    .prefetch(tf.data.AUTOTUNE)
)

validation_ds = (
    tf.data.Dataset.from_tensor_slices((validation_texts, validation_labels))
    .batch(32)
    .prefetch(tf.data.AUTOTUNE)
)

model.fit(
    train_ds,
    validation_data=validation_ds,
    epochs=5,
)

Because preprocessing is part of the model, prediction takes the same kind of input as the original data. There is no separate vocabulary lookup to keep synchronized with application code.

scores = model.predict([
    "The parser reports an unexpected token.",
    "The test suite passes after the change.",
])
print(scores)

Save the fitted model rather than reconstructing its vectorizer from scratch. The saved model carries the adapted vocabulary and its configuration, so loading it restores the text-to-ID mapping used during training.

model.save("text_classifier.keras")

loaded_model = keras.models.load_model("text_classifier.keras")
scores = loaded_model.predict(["A new example arrives here."])

For larger datasets, keep an eye on where the input pipeline runs. A vectorization layer inside the model favors a self-contained inference artifact; preprocessing in a tf.data mapping step can be useful when you want to cache encoded examples or overlap CPU preprocessing with accelerator training.

vectorizer.adapt(train_texts)

encoded_train_ds = (
    tf.data.Dataset.from_tensor_slices((train_texts, train_labels))
    .map(
        lambda text, label: (vectorizer(text), label),
        num_parallel_calls=tf.data.AUTOTUNE,
    )
    .cache()
    .shuffle(buffer_size=len(train_texts))
    .batch(32)
    .prefetch(tf.data.AUTOTUNE)
)