The `pandas.get_dummies()` function is used to convert categorical variables into binary indicator columns. To apply categorical imputation, the `dummy_na=True` argument can be passed, which instructs the function to treat `NaN` as a valid category and generate a dedicated dummy column to explicitly flag missing entries.

Claude

University of Michigan - Ann Arbor

In order to avoid the problem of false assumed relationships present with LabelEncoder, you can use OneHotEncoder to transform a column into a series of columns where each column represents the binary choice of 'did the original column contain value x'. This prevents the creation of false ordinal relationship. However, OneHotEncoder only takes columns of integer values greater than 1, but you can easily transform categorical variables using LabelEncoder or pd.get_dummies

sklearn.preprocessing.OneHotEncoder

Missing data is important to handle early on in the data processing stage because it can easily affect regressions and other types of analysis at later stages of the pipeline. Pandas provides a variety of tools for dealing with missing values. 

Missing values will typically be stored as a ```np.nan``` and will appear in the printed data frames as ```NaN```. To drop all rows with ```NaN```, you can use the ```df.dropna(how="any")```. This will remove all rows that have at least 1 ```NaN```. 

You can also fill all NA values with the ```df.fillna(value = 5)``` and it will set all ```NaN``` entries to 5 in the data frame. You can also perform these operations on subsets of the data frame for more specific manipulation. 


Missing Data Handling

Pandas DataFrame is two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). A Data frame is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and columns.

Pandas DataFrame

A dummy variable, or indicator variable, is created to numerically represent categorical data. When a categorical feature has multiple possible discrete values, it is converted into a series of separate binary columns. For a given record, the column corresponding to its correct category is assigned a value of $$1$$, while all other dummy columns are set to $$0$$.

Dummy Variable

A common imputation heuristic for missing categorical fields is to treat the absence of data itself as a distinct category. For example, if a column contains both valid string categories and NaN entries, it can be transformed into multiple binary dummy variables—where one specific column serves to indicate whether the original value was missing by using $$1$$ for missing and $$0$$ otherwise.

Categorical Imputation via Dummy Variables

https://pandas.pydata.org/docs/index.html

Pandas documentation

Dive into Deep Learning

Scikit-learn enables allows for the transformation of categorical data into quantitative data which can be used in machine learning model training. However, an important thing to note about LabelEncoder is that it will transform your categorical data into ordinal numbers which may cause your model interpret do believe there are relationships between unrelated categories.

sklearn.preprocessing.LabelEncoder

Link: [kaggle.com](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding) 

One hot encoding picture

pandas.get_dummies

Imputation is a critical data science strategy used to handle missing values by replacing them with estimated substitutions rather than discarding the data entirely. The specific estimation heuristic depends on the context; for instance, missing numerical data might be replaced by the column's mean, while missing categorical entries might be treated as their own distinct category.

Data Imputation

When dealing with missing values in a dataset, deletion is a straightforward handling strategy that involves entirely removing the rows or columns that contain `NaN` or empty entries. While this method successfully eliminates missing data, it carries the risk of discarding otherwise valuable information.

Data Deletion for Missing Values

For missing numerical values, a standard imputation heuristic is to substitute `NaN` entries with the mean of all available values in that column. In `pandas`, this is achieved by chaining the `.fillna()` and `.mean()` methods, such as `inputs.fillna(inputs.mean())`, which replaces all missing occurrences with the calculated column average.

pandas.DataFrame.fillna

To extract data from a Pandas DataFrame you can use slicing. Two of the most popular and useful slicing techniques are the .loc and .iloc features of Pandas. The main difference between .loc and .iloc is that .loc is label based while .iloc is integer position based. 

Pandas DataFrame Slicing

.groupby() is a useful function to sort a dataframe into groups based on certain criteria. For example, if you have a dataframe of team rosters, you can group by team name or college and each group will consist only of entries with the same team or college.

```python
#group by team
df.groupby(‘Team’)
#group by college
df.groupby(‘College’)
#group by team and college 
df.groupby([‘Team’, ‘College’])
```

Pandas GroupBy

- .drop (Removes values by row or column label)
- .dropna (Removes missing data)
- .drop_duplicates (Removes duplicate values)

Removing Values from DataFrame

Indexing is one of the ways that you can slice and dice your data frames into subsets. There are a couple choices on how to index, and a brief introduction is below. 

Pandas DataFrame Indexing

Beyond standard categorical and numeric variables, deep learning practitioners frequently work with myriad unstructured data formats. These complex data types include text strings for natural language processing, images and point clouds for computer vision, and audio data. Handling these diverse formats requires specialized approaches and advanced tools compared to traditional tabular data.

Learn Before

Related