To achieve a unified solution for incorporating numeracy in NLP task the following are required:


Benchmarking platform to evaluate numeracy in different subtask settings.


Proper design principles to be considered while creating the language model like tradeoffs, exact/approximation, bias/variance, specificity/expressivity and many more.


Successful number representation could open up opportunities in Finance and Accounting.

One important conclusion is that the subword segmentation with lookup embeddings is suboptimal for numbers. 

Syracuse University

NLP Systems have dealt with various significant tasks such as word embeddings, sentiment / emotion detection and many more. But there aren’t many developments made in the direction of numeracy with Natural Language. Numeracy is considered to be important foundation block  that is everywhere in a Language and often Neglected while developing NLP systems.

For example: ‘I woke up at 11’ statement requires comprehending with context and common sense. The number 11 infers late morning which is commonsense. The number 11 can be found in different context like ‘I earn $11 per month’ where context matters.

Number representation in NLP

https://aclanthology.org/2021.naacl-main.53/

Representing Numbers in NLP: a Survey and a Vision

There are two categories of the tasks for numeracy

1. Granularity: Encoding of number is Approximate or Exact
2. Units: Numbers are either Abstract or Grounded

The following are the along the axes of granularity and units:

Simple Arithmetic: Task of arithmetic operation on numbers like addition and subtraction. 

Numeration: This is decoding that refers to the task of mapping a string to numeric value.

Magnitude Comparison: It is the task of comparing values of numbers. 

Arithmetic Word Problems: They are simple arithmetic word problems/tasks. Eg. Mary had two cookies. She gave away one. How many does she have left.

Exact Facts: It is the commonsense knowledge treated to the language model. Eg: Bird has two legs.

Measurement Estimation: Task where the objects are measured in certain dimensions. Eg: weight of a bicycle.

Numerical Language Modeling: Task of making numeric predictions for the unlabelled text in natural language. Eg: In arithmetic: 2+3 = [MASK]

Downstream Applications: Some applications include detecting sarcasm, understanding financial statements, Answering questions and more. 

Some other tasks: Paraphrasing, Quantity Entailment, Numeral Understanding, counting and Domain Specific tasks.

Numeracy Tasks

The methods are classified into String and Real Representation as each has its own way of representing numerical value in their respective form.

String-based methods that consider Notation, Tokenization, Pooling while generating the language methods.

Real-based methods that take Scale, Direction, Binning into consideration

The image contains all the methods arranged according to the taxonomy of Exact and Approximate granularity and results across all the dimension of task available.

Methods applied on Numeracy Tasks 

For String-based methods: 


Scientific Notation works better than Decimal Notation
Character Level Tokenization performs better that Subword Level Tokenization.

For Real-based methods:


Log scale works better than Linear scale.
Binning like Dense Cross Entropy Loss works better than Continuous Value Prediction.
Binning the distributions based on precision level is better than modeling continuous predictions.


Encoding and Decoding:


Value Embedding can be used to encode and decode numbers.DICE Encoder can be used for encoding which is not easily decodable. 


Mixing multiple methods:


Ensembling embeddings could be helpful sometimes. Some methods like Exponent embeddings ensembled with DigitRNN barely outperforms the standalone exponent embeddings.

Learn Before

Related