Learn Before
Consider a tokenization method that segments text into individual English words and punctuation marks. For instance, 'It’s great.' becomes ['It', '’s', 'great', '.']. True or False: Following this method, the phrase 'We're going home.' would be tokenized as ['We', '’re', 'going', 'home.'].
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A tokenization process is designed to segment text into individual English words and punctuation marks. For example, the phrase 'It’s great.' is tokenized into
['It', '’s', 'great', '.']. Based on this rule, how would the sentence 'The student's book isn't here.' be tokenized?Applying Word and Punctuation Tokenization
Consider a tokenization method that segments text into individual English words and punctuation marks. For instance, 'It’s great.' becomes
['It', '’s', 'great', '.']. True or False: Following this method, the phrase 'We're going home.' would be tokenized as['We', '’re', 'going', 'home.'].