1. Establishes the most relevant high-level features.

2. Novel representations with high chance that after mapping will look like real 

Representational Learning - Strengths

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

From input text, meanings of the words were learnt automatically and useful representations were generated.

Representation Learning in NLP

A deep learning model is a powerful type of machine learning model differentiated by its architecture, which consists of many successive transformations of data chained together from top to bottom. This depth of continuous data transformations allows these models to tackle complex tasks by automatically learning multi-level representations. By doing so, deep learning effectively replaces not only the labor-intensive process of manual feature engineering, but also the shallow models that are typically used at the end of traditional machine learning pipelines.

Deep Learning Model

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Johns Hopkins University

San Diego State University

Claude

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

Machine learning is a branch of artificial intelligence focused on building systems that improve their task performance by learning from data or other experience. It draws on ideas from statistics, optimization, and data analysis to make predictions, recognize patterns, and support decisions.

Machine Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

If the model is based on fixed calculations we will have the same results every time(i.e. taking average value of each pixel), thus that's not generative. It is inevitable to use random element influencing each samples that are produced by our model. 
(**quick note** this comparison can be  similar to *linear* vs *non-linear* activation functions in deep learning)

*Probabilistic* rather than *Deterministic*

Most problems that you might have faced in the machine learning are discriminative in their nature. Unlike generative modeling here each sample has a **label** .  You can consider discriminative modeling as supervised learning.

The discriminative model learns the decision function f(X) directly from the data, or the conditional probability distribution P(Y|X) as the prediction model. The discriminative model is concerned with what output Y should be predicted for a given input X. Common discriminative models are: K-nearest neighbor, perceptron, linear regression, linear discriminant analysis (LDA), LR, SVM, decision tree, neural network, boosting, conditional random field, maximum entropy model.

In order to understand difference between *discriminative* and *generative* modeling you can use these definitions:
 

Discriminative Modeling

1. Complete understanding of how the data was generated, not merely categorizing objects.

2. Seems highly promising in the Reinforcement Learning 

3. It is mightier to acquire machine with  the form of human intelligence. 

Why *Generative Modeling* ? 

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

Quick Recap For Some Probability Concepts

Representational Learning

1. Variational AutoEncoders (_VAE_)
2. Generative Adversarial Networks (_GAN_)
3. Diffusion Inversion
4. Approximate Bayesian computation (_ABC_) framework

Generative Modeling Architectures

https://www.amazon.com/Generative-Deep-Learning-Teaching-Machines/dp/1492041947

David Foster's Generative Deep Learning

Deep Belief Networks (DBNs) were one of the first nonconvolutional models to successfully admit training of deep architectures. They are generative models with several layers of latent or hidden variables in which the latent variables are typically binary, while the visible units may be binary or real. Every unit in each layer is connected to every unit in each neighboring layer.

Deep Belief Networks (DBNs)

Practitioners usually evaluate generative models by having experimental subjects visually assess the samples. This can help to notice under and overfitting, but struggles because bad models may still look solid visually. In this case, databases like MNIST are used to continue to check.

Evaluating Generative Models

Generative Adversarial Networks are a type of generative modeling approach. In this process, the generator network competes with the discriminator network, which attempts to distinguish between samples drawn from either the training data or the generator. This allows the discriminator to produce a probability that a particular training example is real rather than a fake sample drawn from the model. 

Generative Adversarial Networks

A generator network using a convolutional structure is often useful for generating images. We utilize the "transpose" of the convolution operator, often yielding more realistic images using fewer parameters.

Convolutional Generative Networks

A **Generative Stochastic Network**, or **GSN**, is a generative model with latent or hidden variables $$\bold h$$ in the Markov chain, in addition to the ordinary visible random variables $$\bold x$$.

Generative Stochastic Networks (GSNs)

Here is the example to help make the idea of a generative model comprehensible for you. Suppose you have dog images and you want to generate a dog image that looks real but that dog actually doesn't exist. This can be achieved by learning the characteristics of dog images. This is a typical problem which we can solve using generative modeling.

Here is a visual clue for you:

Generative Model Example

•	The standard procedure for drawing samples from a normal distribution with mean µ and covariance Σ  is to feed samples z from a normal distribution with zero mean and identity covariance into a very simple generator. 
•	Pseudorandom number generators can also use nonlinear transformations of simple distributions. In this case, we can use the inverse transform sampling techniques. If we can specify p(x), integrate over x, and invert the resulting function. In this case, we don’t need to use machine learning methods.


How to generate samples from not complicated distributions using generator networks?

• To generate samples from more complicated distributions that are diﬃcult to specify directly, diﬃcult to integrate over, or whose resulting integrals are diﬃcult to invert, we use a feedforward network to represent a parametric family of nonlinear functions g, and use training data to infer the parameters selecting the desired function

Generate samples from complicated distributions

• When emitting the parameters of a conditional distribution over x, it can generate discrete data as well as continuous data. However, these data could be algebraically manipulated by humans, which is a weakness.

• On the other hand, when the generator net provides samples directly, it can generate only continuous data, and are not forced to use conditional distribution. The problem with direct sampling is that the model could no longer be trained using back-propagation. 


Emitting the parameters of a conditional distribution versus directly emitting samples

Some generative models allow the probability density function to be evaluated explicitly; others do not. Some are structured probabilistic models which can be described in terms of graphs and factors; others cannot, though they still represent probability distributions.

Variations of generative models

Which of the following models are machine learning generative models?

Generative models

The learning process of generative modeling requires optimizing intractable criteria. In the context of differentiable generator nets, the criteria are intractable because the data does not specify both the inputs $$z$$ and the outputs $$x$$ of the generator net. However, in the case of supervised learning, both the inputs $$x$$ and the outputs $$y$$ are given, and the optimization procedure needs only to learn how to produce the specified mapping.

Difficulty of Generative Modeling Compared to Supervised Learning

Supervised learning is a method for learning a mapping from inputs x to outputs y from labeled examples (x, y). Common supervised learning algorithms include linear regression, logistic regression, and neural networks.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

A development set, sometimes called a holdout validation set, is a data split used to tune hyperparameters, compare feature choices, and make other model-design decisions during development.

Development Set

A test set is used to estimate how well a trained system performs, but it should not be used to choose the model, features, or hyperparameters.

Purpose of a Test Set

When a project must satisfy more than one evaluation criterion, a common strategy is to set one metric as a minimum acceptable requirement and then optimize a different metric among the solutions that meet that requirement. For example, a team might require inference time to stay under a limit and then select the model with the highest accuracy.

Combining Metrics With a Threshold and an Objective

For a brand-new machine learning task, it is usually very hard to know ahead of time which method will work best. Even skilled researchers often test many different ideas before they find one that performs well enough.

It Is Hard to Predict the Best Machine Learning Strategy Before Trying It

Machine learning development usually proceeds in cycles. A practical loop is to propose an approach, implement it, run an experiment to measure the result, and then use what you learned to choose the next change. Because each round reveals new information, teams often test many ideas before settling on one that works well.

An Iterative Machine Learning Workflow

In machine learning, two broad causes of poor performance are bias and variance. Separating them helps you choose the right next step, such as changing the model, gathering more data, or adjusting the training approach.

Bias and Variance as Two Major Sources of Error

With enough labeled input-output examples, an end-to-end model can learn to map inputs directly to outputs that are more complex than a single number. Those outputs can include text, images, audio, or other structured results.

What End-to-End Learning Can Predict

A pipeline can be poorly designed if the full system performs much worse than human level even though each stage is close to human level on the information it receives. In that case, the weakness is often that important information is being lost, so the design should be reconsidered.

When a Pipeline Is Missing Information

Text polarity detection is the task of deciding whether a piece of text expresses a favorable or unfavorable opinion, such as determining whether a product review praises or criticizes the item.

Text Polarity Detection

Machine learning strategy is the discipline of deciding which changes are most likely to improve a model, based on evidence from errors, data splits, and project constraints. It helps you prioritize the next experiment instead of guessing blindly.

Machine Learning Strategy

Recent progress in deep learning has mainly come from larger data sets and greater computing power.

Learn Before

Related

Learn After