Tokenizing means to split up text by word or by sentence. This process provides the first step for data analysis. To use tokenizing features, import: 
```python
from nltk.tokenize import sent_tokenize, word_tokenize
```
```sent_tokenize``` will split up the text by sentence and ```word_tokenize``` will split up the text by word. Both functions return a list of strings.

Tokenizing

Named entities are noun phrases that refer to specific locations, people, organizations, and so on. NER automatically finds the named entities in texts and determines what kind of named entity they are. ```nltk.ne_chunk()``` can recognize named entities. 

```python
def extract_ne(quote):
        words = word_tokenize(quote, language=language)
        tags = nltk.pos_tag(words)
        tree = nltk.ne_chunk(tags, binary=True)
        return set(" ".join(i[0] for i in t)
            for t in tree
            if hasattr(t, "label") and t.label() == "NE")
```


Using Named Entity Recognition (NER)



A lemma is a word that represents a whole group of words, and that group of words is called a lexeme. Lemmatization is the task of determining that two words have the same root, despite their surface differences. For instance, the word 'friends' will become 'friend' and 'worst' will become 'bad'. To start lemmatizing, use:
```python
from nltk.stem import WordNetLemmatizer
```
To lemmatize a sentence, use ```word_tokenize``` first then iterate through the list with a loop.


Lemmatization

Chunking allows you to identify phrases by making use of POS tags. Naturally, to chunk a text, word_tokenize must be imported. 

```python
from nltk.tokenize import word_tokenize
```
Let ```text``` be the body of text to chunk.
```python
tokenized_text = word_tokenize(text) # tokenized_text will be a list of words separated into different strings
nltk.download("averaged_perceptron_tagger") 
POS_tags = nltk.pos_tag(tokenized_text) # returns list of tuples with each word paired with a POS
```
The next step is to form a grammar rule by which the sentence should be phrased, or "chunked."
```python
grammar = "NP: {<DT>?<JJ>*<NN>}"
```
This rule defines a Noun Phrase(NP), which means it can start with an optional determiner, then have any number of adjectives, then ends with a noun. 

Then create a chunk parser with this grammar
```python
chunk_parser = nltk.RegexpParser(grammar)
tree = chunk_parser.parse(POS_tags)
tree.draw()
```

Chunking in NLTK

Tagging is the process of identifying and labeling Parts of Speech (POS). POS is a grammatical term that deals with the roles words play when using them together in sentences. There are eight parts of speech in English (Noun, Pronoun, Adjective, Verb, Adverb, Preposition, Conjunction, Interjection). Another POS not included in the list is Determiners, which are articles like 'a' or 'the'. 

To tag a chunk of text, use ```word_tokenize``` to separate the words in the string and store them in a list. Then call 
```
nltk.pos_tag()
```
on the new list of words. This will return a tuple with a word paired to their POS, such as
```
[('If', 'IN'),
 ('you', 'PRP'),
 ('wish', 'VBP')]
```

Tagging in Programming Languages

- Tokenizing
- Tagging
- Lemmatizing
- Chunking
- Using Named Entity Recognition (NER)


University of Michigan - Ann Arbor

Natural Language Processing is a process of manipulating or understanding speech or text with software or machine. NLTK contains libraries for statistical language processing. There are many NLP libraries out there for use but NLTK is by far the most diverse. Other NLP libraries are best suited for specific tasks.

Learn Before

Related

Learn After