-    Books
-    Articles
-    Research papers

Information Theory references

A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Information Entropy

A nat is a unit of information or entropy, analogous to a bit, but based on natural logarithms (base $$e$$) rather than base 2. The relationship between nats and bits is defined by the base change: one nat is equivalent to $$\frac{1}{\log(2)} \approx 1.44$$ bits.

In information theory, the predictability of data is inversely related to the amount of information required to communicate it. A data stream that is highly predictable (e.g., repeating identical tokens) conveys little to no new information and is therefore easy to compress. Conversely, unpredictable events carry more information and are harder to compress.

Relationship Between Predictability and Compressibility

Information theory is the mathematical understanding of parameters and rules of transmitting and processing information through communication systems.

University of Michigan - Ann Arbor

Claude

- Discrete mathematics
- Probability and statistics
- Mathematical software
- Information theory
- Mathematical analysis
- Continuous mathematics

Mathematics of computing in ACM Computing Classification

- Information Theory
- Quantum Information 

Major Theories in Information Science

https://www.britannica.com/science/information-theory/Discrete-noiseless-communication-and-the-concept-of-entropy

Britannica Article on Info Theory

Dive into Deep Learning

Statistics is the process of gathering, analyzing, and interpreting information in order to help people make decisions when faced with uncertainty.

Statistics

The linear algebra necessary to further explain Machine Learning in our capacity, requires a basic understanding of 
- Linear Systems of Equations
- Linear Transformations
- Vector Spaces
- Orthogonality
- Eigenvalues


Linear Algebra

Information Theory

- Computation of transforms
- Computations in finite fields
- Computations on matrices
- Computations on polynomials
- Number-theoretic computations
- Interpolation
- Numerical differentiation
- Interval arithmetic
- Arbitrary-precision arithmetic
- Automatic differentiation
- Mesh generation
- Discretization

Learn Before

Related

Learn After