The Time Machine dataset consists of H. G. Wells' book *The Time Machine*, which contains just over $$30,000$$ words. It is used as a small, introductory text dataset to demonstrate the fundamental steps of reading and preprocessing raw text for sequence data models before scaling up to the significantly larger datasets typical of real-world applications.

The Time Machine Dataset

After loading raw text into memory, it is commonly preprocessed to standardize the format and reduce vocabulary complexity before feeding it into sequence models. A straightforward preprocessing strategy uses a regular expression to replace every non-alphabetical character (including punctuation, digits, and whitespace) with a single space, and then converts all remaining characters to lowercase. For instance, applying the pattern [^A-Za-z]+ followed by lowercasing transforms 'The Time Machine, by H. G. Wells [1898]' into 'the time machine by h g wells '. This produces a uniform stream of space-separated lowercase words, eliminating noise from punctuation and capitalization that is unnecessary for many language modeling tasks.

Simplified Raw Text Preprocessing

The TimeMachine class provides a build method that combines all text-to-sequence conversion steps into a single end-to-end pipeline. Given the raw text and an optional pre-existing vocabulary, build first preprocesses the text (lowercasing and stripping non-alphabetical characters), then tokenizes the cleaned text into individual characters. If no vocabulary is supplied, it constructs one from the resulting tokens using the Vocab class. Finally, it maps every token to its corresponding integer index via the vocabulary, producing two outputs: corpus, a flat list of integer token indices, and vocab, the vocabulary object. This unified method ensures that preprocessing, tokenization, and numericalization are performed in a consistent, reproducible sequence.

TimeMachine Build Pipeline

The initial step in preparing text for sequence modeling is to read the raw text from a dataset into memory as a continuous string. For example, a complete text file, such as a book, can be downloaded and loaded as a single sequence of characters before any further processing occurs.

Claude

Sequence models are statistical supervised learning functions designed to process, predict, or classify based on sequence data. A fundamental task in this domain is to estimate the joint probability of an entire sequence. When dealing with natural language data composed of discrete tokens, such as words, these estimated functions are commonly referred to as language models. Sequence models provide the capacity to evaluate the likelihood of sequences (for example, comparing the naturalness of outputs from translation systems), sample new sequences, and optimize for the most likely sequence outputs.

Sequence Models

Dive into Deep Learning

https://machinelearningmastery.com/sequence-prediction/

Predictions with Sequences

Sequence prediction is used to predict the next value of a given input sequence. One example is product recommendations, which predicts a customer's next purchase given a sequence of their past purchases. Another example is stock market prediction.

Sequence Prediction Models

Sequence classification predicts a class label for a given input sequence. Sentiment analysis is one example, which takes a sequence of text and predicts the label for the sentiment (positive or negative). 

Sequence Classification Models

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

You have trained word embeddings using a text dataset of $$m_1$$ words. You are considering using these word embeddings for a language task, for which you have a separate labeled dataset of $$m_2$$ words. Keeping in mind that using word embeddings is a form of transfer learning, under which of these circumstance would you expect the word embeddings to be helpful?

Sequence Model Question #1

In the word2vec algorithm, you estimate $$P\left(t|c\right)$$, where t is the target word and c is a context word. How are t and c chosen from the training set?

Sequence Model Question #2

Suppose you learn a word embedding for a vocabulary of 10000 words. Then the embedding vectors should be 10000 dimensional, so as to capture the full range of variation and meaning in those words.

Sequence Moel Question #4

In beam search, if you increase the beam width B, which of the following would you expect to be true? Check all that apply.

Sequence Model Question #3

Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

In sequence modeling architectures, standard notation is used to represent the inputs and outputs. A source sequence is typically denoted as $$x_1...x_m$$, where $$m$$ represents the length of the source sequence. Its corresponding sequence of embedding vectors is denoted as $$\mathbf{e}_{1}^{x}...\mathbf{e}_{m}^{x}$$. Similarly, the target sequence that the model aims to generate is denoted as $$y_1...y_n$$, with length $$n$$, and its corresponding embedding sequence is represented as $$\mathbf{e}_{1}^{y}...\mathbf{e}_{n}^{y}$$.

Notation for Source and Target Sequences

In sequence modeling, interpolation and extrapolation represent two fundamentally different challenges with a significant gap in difficulty. Interpolation involves estimating values that fall within the temporal range of data the model has already observed, while extrapolation requires predicting future, unseen values beyond the observed range. Because extrapolation demands the model generalize beyond its training distribution, it is substantially harder than interpolation. This asymmetry has a critical practical implication: when working with sequential data, one must always respect the temporal order of observations during training—that is, a model should never be trained on future data to predict the past.

Interpolation vs. Extrapolation in Sequence Models

For causal sequence models—those in which time progresses naturally forward—estimating values in the forward direction (predicting the future from past observations) is typically much easier and more practical than estimating in the reverse direction (reconstructing the past from future observations). This asymmetry arises because causal models are designed to capture how past events generate future outcomes, making the forward conditional distribution the natural object of estimation. Attempting to invert this process is inherently more challenging because the mapping from effects back to causes is often one-to-many and ill-conditioned.

Learn Before

Related

Learn After