Applying Word and Punctuation Segmentation
A text processing method involves segmenting a sentence into a sequence of its individual words and punctuation marks. Under this method, a contraction is separated into its component parts. For example, the sentence "It's amazing." would be segmented into the sequence: {It, ’, s, amazing, .}. Following this exact method, provide the complete sequence of segments for the sentence: "We're not going; it's too late!"
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A piece of text is segmented into a sequence of smaller units by separating it into individual words and treating each punctuation mark as its own distinct unit. Given this method, which of the following options correctly represents the segmentation of the sentence: "She said, 'It's great!'"?
Applying Word and Punctuation Segmentation
Analyzing a Tokenization Function's Output