To facilitate model training and evaluation, a data iterator is defined for the machine translation dataset. This iterator, often implemented as a get_dataloader method, uses a boolean flag to slice the underlying data arrays, returning a tensor loader for either the training or validation subset of the dataset.

Claude

The conceptual preprocessing steps for the English-French dataset can be implemented in a `_preprocess` method added to the `MTFraEng` class. This method standardizes the raw text by first replacing non-breaking spaces (`\u202f` and `\xa0`) with regular spaces. It then converts the text to lowercase and iterates through the characters, selectively inserting a space before punctuation marks (e.g., `, . ! ?`) if a space does not already precede them.

```python
@d2l.add_to_class(MTFraEng)
def _preprocess(self, text):
    # Replace non-breaking space with space
    text = text.replace('\u202f', ' ').replace('\xa0', ' ')
    # Insert space between words and punctuation marks
    no_space = lambda char, prev_char: char in ',.!?' and prev_char != ' '
    out = [' ' + char if i > 0 and no_space(char, text[i - 1]) else char
           for i, char in enumerate(text.lower())]
    return ''.join(out)
```

Learn Before

Related