Imputation is a critical data science strategy used to handle missing values by replacing them with estimated substitutions rather than discarding the data entirely. The specific estimation heuristic depends on the context; for instance, missing numerical data might be replaced by the column's mean, while missing categorical entries might be treated as their own distinct category.

Data Imputation

When dealing with missing values in a dataset, deletion is a straightforward handling strategy that involves entirely removing the rows or columns that contain `NaN` or empty entries. While this method successfully eliminates missing data, it carries the risk of discarding otherwise valuable information.

Data Deletion for Missing Values

The `pandas.get_dummies()` function is used to convert categorical variables into binary indicator columns. To apply categorical imputation, the `dummy_na=True` argument can be passed, which instructs the function to treat `NaN` as a valid category and generate a dedicated dummy column to explicitly flag missing entries.

pandas.get_dummies

For missing numerical values, a standard imputation heuristic is to substitute `NaN` entries with the mean of all available values in that column. In `pandas`, this is achieved by chaining the `.fillna()` and `.mean()` methods, such as `inputs.fillna(inputs.mean())`, which replaces all missing occurrences with the calculated column average.

pandas.DataFrame.fillna

Missing data is important to handle early on in the data processing stage because it can easily affect regressions and other types of analysis at later stages of the pipeline. Pandas provides a variety of tools for dealing with missing values. 

Missing values will typically be stored as a ```np.nan``` and will appear in the printed data frames as ```NaN```. To drop all rows with ```NaN```, you can use the ```df.dropna(how="any")```. This will remove all rows that have at least 1 ```NaN```. 

You can also fill all NA values with the ```df.fillna(value = 5)``` and it will set all ```NaN``` entries to 5 in the data frame. You can also perform these operations on subsets of the data frame for more specific manipulation. 


University of Michigan - Ann Arbor

Claude

There are many tools for tidying data in Pandas and the subsequent child nodes will cover the specific usage of the following concepts:
- Data Selection
- Data Merging
- Data Grouping
- Missing Data 
- Date Time Conversions

Tools for Tidying Data

Dive into Deep Learning

Selecting Data can be done very easily as data stored in Pandas data frames have a coordinate pair associated with it and also have a string based indexing, which utilizes the column names and index values. 

To select an entire column, use ``` python df["column_name"]```. To select a range of columns use ``` python df[0:3] ```. This will select the values ranging from column 0 to column 3. 

Data selection can also be done using the rows names, or selecting a range of values using string or integer names, for example: ``` python df["20130102":"20130104"]```
```
                  A         B         C         D
2013-01-02  1.212112 -0.173215  0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929  1.071804
2013-01-04  0.721555 -0.706771 -1.039575  0.271860
```

You can also directly access a specific index using one of the following 4 functions in the Child node.

Selecting Data in Pandas

Missing Data Handling

Data in Pandas data objects are easily merged together. Given a series, we can concatenate it into a date frame. 
Given 
```python 
pieces = [df[:3], df[3:7], df[7:]]
``` 
```python 
pd.concat(pieces)
``` 
This would  concatenate the pieces based off their original index values and piece the data frame back together again. 



Merging in Pandas

Other than merging, we can also perform join operations similar to SQL join function calls. 

```python
pd.merge(left, right, on="key")
```

This would join the left and right dat frames based on the variable named "key". 

For more specifics on merging details, such as inner or outer joins, please consult the specific documentation, found here :https://pandas.pydata.org/docs/user_guide/merging.html#merging-join. 

Joining in Pandas

Data doesn't always come in the format that is very useful or helpful for performing analysis and sometimes can be re-organized to make things easier to view. Reshaping can change the relative position of certain information to help draw emphasis between different pieces of data. Consider the following example:
```python
                     A         B
first second                    
bar   one     0.721555 -0.706771
      two    -1.039575  0.271860
baz   one    -0.424972  0.567020
      two     0.276232 -1.087401
```
Using the ```.stack()``` method the data can be modified to look like this
```python
first  second   
bar    one     A   -0.727965
               B   -0.589346
       two     A    0.339969
               B   -0.693205
baz    one     A   -0.339355
               B    0.593616
       two     A    0.884345
               B    1.591431
```
You can also include an argument when using the stack method to specify which columns you want to stack. The same logic and usage applies when using the inverse function ```.unstack()```.

Both ```.stack()``` and ```.unstack()``` will return a sorted copy of the dataframe. 

Learn Before

Related

Learn After