Learn Before
Distinction and Interchangeability of 'Tokens' and 'Words' in NLP
In Natural Language Processing (NLP), tokens are considered the fundamental units of text, produced through a process called tokenization. Although the terms 'token' and 'word' are frequently used interchangeably in discussions, it is important to recognize that they have closely related yet distinct meanings within the field.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Natural language processing in ACM Computing Classification
NLP references
Models used in NLP
Text normalization
Part-of-speech Tagging
Sentiment Analysis
Topic Model
Parsing
High Dimensional Outputs
Historical Perspective: Natural Language Processing
Machine Reading and Comprehension
Minimum Edit Distance
Variation Factors of Input Texts
Period Disambiguation
Features Design for NLP Classification Problems
Vector Semantics and Embeddings
Words and Vectors
English Word Classes
Logical Representations of Sentence Meaning
First-Order Logic
Information Extraction
Word Senses
Semantic Roles: Labeling
Semantic Roles ( Thematic Roles )
Question Answering
Information Retrieval
Dialogue Systems
Properties of Human Conversation
Prompt Tuning
Types of NLP Model Paradigms
Types of Training Objectives of Pre-trained LM
Major Tuning Strategy Types
Articulatory Phonetics
Phonetics
Word embedding
A Survey of Data Augmentation Approaches for NLP
Data Augmentation in NLP
Spelling correction and the noisy channel
Constituency
Text Classification
Information Extraction (IE)
A Survey of Natural Language Based Financial Forecasting
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction
A Survey of the State-of-the-Art Models in Neural Abstractive Text Summarization
From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Information
Machine Translation (MT)
Temporal Reasoning
Knowledge Graph
Dynamic Neural Network in Natural Language Processing
Label Preservation
Deep Learning Algorithms in Data Augmentation
Applications of Data Augmentation
Coreference Resolution
Explainable AI for Natural Language Processing
Corpora
Racism in NLP
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Low-Resource Scenario in Natural Language Processing
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Low-Resource NLP
Continual Learning
Continual Lifelong Learning in Natural Language Processing: A Survey
Object Naming in Language and Vision
A Survey on Hate Speech Detection using Natural Language Processing
Hate Speech Detection using Natural Language Processing
A Survey of Text Games for Reinforcement Learning informed by Natural Language
Natural Language Text Games for Reinforcement Learning
Data-Driven Sentence Simplification: Survey and Benchmark
Deep Learning for Text Style Transfer: A Survey
Text Style Transfer (TST)
Representing Numbers in NLP: a Survey and a Vision
Number representation in NLP
Semantic Textual Similarity (STS)
Paraphrase Identification (PI)
Machine Comprehension (MC)
Sentence Representation Model Categorizations
Automatic Detection of Machine Generated Text: A Critical Survey
Automatic Detection of Machine Generated Text
Fine-grained Financial Opinion Mining: A Survey and Research Agenda
Natural Language Processing in Finance
Phonology / Phonetics
Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering
Sentence Pair Modelling
A Survey of Active Learning for Text Classification using Deep Neural Networks
A Survey of Knowledge-Enhanced Text Generation
Knowledge-enhanced Text Generation
The Pollyanna Hypothesis
On Positivity Bias in Negative Reviews
Widely Used English Review Datasets
A Survey on Dialogue Summarization: Recent Advances and New Frontiers
Survey on Dialogue Summarization: Recent Advances and New Frontiers
Potential Biases of Natural Language Processing
The Pre-training and Fine-tuning Paradigm
Tokens and Words in NLP
Distinction and Interchangeability of 'Tokens' and 'Words' in NLP
Code-Switching in NLP and Linguistics
Automatic Speech Recognition
Text to Speech
Training Dataset