Data & Preprocessing

What is a Dataset ?
In Machine Learning, a dataset is just a collection of data that we use to train and test our models.

Think of it like an Excel sheet:

Rows → each row is one data point (called a sample).
Columns → each column contains information about that sample.
Features (Inputs)

Features are the independent variables — the input data we provide to the ML model.
They describe the properties of each sample.
Example:
Age
Salary
Height
Exam scores Features are like the ingredients we give to a recipe.
Labels (Outputs / Targets)

The label is what we want the model to predict.
It is also called the dependent variable or target.
Example:
Will the student pass or fail? (Yes/No)
Price of a house
Disease diagnosis (Positive/Negative)
Labels are like the final dish that we expect after cooking with the ingredients.

Example Dataset

Age	Salary	Hours Studied	Passed (Label)
20	30,000	5	Yes
22	25,000	2	No
19	40,000	6	Yes
21	35,000	1	No
Features → Age, Salary, Hours Studied
Label → Passed (Yes/No)

###################################################
Loading CSV - Worked Example
We have a dataset of students stored in a CSV file (students.csv). Each row represents one student, and the dataset contains:

Age → student’s age
Salary → part-time salary in INR
Hours_Studied → number of hours studied per day
Passed → whether the student passed the exam (Yes or No)
Our task is to load this dataset into Python using Pandas and take a quick look at it.

########################

# Step 1: Import pandas
import pandas as pd

# Step 2: Load the dataset (make sure students.csv is in the same folder as this script)
df = pd.read_csv("students.csv")

# Step 3: View the first few rows
print("First 5 rows of the dataset:")
print(df.head())

# Step 4: Separate features and label
features = df[["Age", "Salary", "Hours_Studied"]]
label = df["Passed"]

print("\nFeatures (X):")
print(features.head())

print("\nLabel (y):")
print(label.head())
#################################################

Types of data
When working with datasets, data can come in different forms.
Understanding the type of data helps us decide how to preprocess and use it in ML models.

1. Numerical Data (Quantitative)
Data represented as numbers.

Age = 25 → Discrete
Salary = $55,000 → Discrete
Clarification: You might think salary could be continuous, but in practice, 
it's paid in specific currency units (e.g., dollars and cents). 
Since there are finite, countable steps between values (you can't be paid $55,000.12345), we treat it as discrete.
Weight = 72.8 kg → Continuous

2. Categorical Data (Qualitative)

Data represented as categories or labels.

Gender = {Male, Female, Other} → Unordered
Department = {HR, IT, Finance} → Unordered
Customer Satisfaction = {Low, Medium, High} → Ordered

3. Text Data

Data in the form of sentences, paragraphs, or words.

Reviews: "This product is amazing!"
Tweets, comments, articles.

4. Image Data  ###########

Data in the form of pictures or visuals. To a computer, an image is a grid of pixels,
where each pixel has a numerical value representing its color and intensity.

Handwritten digit images
Medical scans (X-rays, MRIs)
Satellite photos for weather prediction
Product photos on an e-commerce site
Security camera footage
###########################################
MCQ 1 : 
A health-tracking app records the number of steps a user takes in a day. 
Sometimes a person may walk 5,000 steps, other times 12,300 steps. The values are always whole counts.

Answer = Numeric ( Discreate ) because the number of steps is a numerical discrete variable, it consists of whole counts.
#####################3
MCQ 2 : 
A university collects data about students’ blood groups for medical records. 
Possible values are {A, B, AB, O}. There is no ranking or priority among these categories.

Answer = Categorical (unordered ) because blood groups are categorical (unordered), they are distinct categories with no inherent ranking.
##############################
MCQ 3 :
Researchers collect thousands of tweets about a new smartphone launch. 
Each tweet is a short text message, often with slang, hashtags, or emojis.

Answer = Text ... because tweets are text data, consisting of words, symbols, and emojis.
#############################33

Handling missing values :

In real-world datasets, missing values are very common. Machine learning models cannot work properly with missing values, so we need to handle them before training.

1. How to Identify Missing Values

In Pandas, missing values are usually represented as:

NaN (Not a Number)
None
Empty cells
Output:

Name	Age	Department	Salary
Rahul	25	  IT	      50000
Priya	NaN	  HR	      45000
Arjun	28	  NaN      	60000
Sneha	35	  IT         NaN

2. Detect Missing Values

# Check for missing values
print(df.isnull())

# Count missing values per column
print(df.isnull().sum())
3. Handling Missing Values : Removing Missing Values

Drop rows with missing values:
df_cleaned = df.dropna()
Drop columns with too many missing values:
df_cleaned = df.dropna(axis=1)
4. Handling Missing Values : Filling Missing Values (Imputation)

Fill with a constant:
df['Age'] = df['Age'].fillna(0)  # Fill missing ages with 0
Fill with mean/median/mode (common for numerical data):
df['Salary'] = df['Salary'].fillna(df['Salary'].mean())
Fill categorical missing values with the most frequent value:
df['Department'] = df['Department'].fillna(df['Department'].mode()[0])
###########################################33
HERE IS AN EXAMPLE.....
Handling missing values - Worked Example
We have a dataset of employees stored in a CSV file (employees.csv). Some values are missing (NaN).

Each row represents one employee, and the dataset contains:

Name → employee’s name
Age → employee’s age
Department → department in the company
Salary → employee’s salary in INR
Our task is to load the dataset, detect missing values, and handle them so that our ML models can work correctly.

#######  HERE IS THE CODE ....  ####
import pandas as pd

# Step 1: Load dataset
df = pd.read_csv("employees.csv")
print("Original Dataset:")
print(df)

# Step 2: Check missing values
print("\nMissing Values per Column:")
print(df.isnull().sum())

# Step 3: Fill missing numerical values with mean
df['Age'] = df['Age'].fillna(df['Age'].mean())
df['Salary'] = df['Salary'].fillna(df['Salary'].mean())

# Step 4: Fill missing categorical values with mode
df['Department'] = df['Department'].fillna(df['Department'].mode()[0])

print("\nDataset after Handling Missing Values:")
print(df)

###############################
Handling missing values - Practice Problem
You are given a dataset called students.csv. This dataset contains information about students in a school, and some values are missing (NaN).

Task:

Load the dataset using Pandas.
Display the first 5 rows.
Check for missing values in each column.
Handle missing values:
Fill missing numerical columns (Math_Score, English_Score, Science_Score) with the mean of that column.
Fill missing categorical columns (Class) with the mode of that column.
Round numerical values to 1 decimal place.
Display the dataset after handling missing values.
####  HERE IS THE CODE #####
import pandas as pd

def handle_missing_values(file_path):
    # Step 1: Load dataset
    df = pd.read_csv("students.csv")
    print("Original Dataset:")
    print(df.head())

    # Step 2: Check missing values
    print("\nMissing Values per Column:")
    print(df.isnull().sum())

    # Step 3: Fill missing numerical values with mean
    df['Math_Score'] = df['Math_Score'].fillna(df['Math_Score'].mean())
    df['English_Score'] = df['English_Score'].fillna(df['English_Score'].mean())
    df['Science_Score'] = df['Science_Score'].fillna(df['Science_Score'].mean())

    # Step 4: Fill missing categorical values with mode
    df['Class'] = df['Class'].fillna(df['Class'].mode()[0])

    # Step 5: Round numerical columns to 1 decimal place
    df[['Math_Score', 'English_Score', 'Science_Score']] = df[['Math_Score', 'English_Score', 'Science_Score']].round(1)

    # Step 6: Display dataset after handling missing values
    print("\nDataset after Handling Missing Values (rounded to 1 decimal):")
    print(df)

    return df

# Call the function in main
if __name__ == "__main__":
    handle_missing_values("students.csv")
#########################
What is Outlier
An outlier is a value in your data that is very different from the other values.
Think of it like a “weird guest” in a group: they just don’t fit in with the rest.
image
Example:
We have a dataset of students’ test scores:

Name	Score
Alice	50
Bob	52
Carol	49
Dave	51
Eve	100
Most scores are around 50, but Eve scored 100, that’s an outlier.
Note: Outliers are easy to see when you look at the data or plot it on a graph.

2. Why Outliers Matter

Outliers can change the average a lot.
They can confuse machine learning models, especially if the model expects “normal” values.
Sometimes outliers are mistakes in data, and sometimes they are real but rare events.
Example:

Average score without Eve: (50+52+49+51)/4 = 50.5
Average score with Eve: (50+52+49+51+100)/5 = 60.4 → big jump because of one outlier!
3. How to Handle Outliers Once we spot an outlier, we decide what to do with it:

Method	What it Means	Example with Eve’s Score
Leave it	Keep it if it’s real and important	Keep 100
Remove it	Delete if it’s a mistake	Remove 100
Adjust it	Replace with a nearby value to reduce impact	Change 100 → 52
Note: Always ask why the outlier exists before deciding what to do.

Key takeaway:
Outliers are values that don’t fit in with the rest. You can see them, think about why they exist, and decide how to handle them. 
Start with visuals and small datasets before learning formulas.

################################33
Dealing with Outliers - Practice Problem
We are given a dataset called working_hours.csv.
This dataset contains information about employees and how many hours they work per week.
Sometimes, datasets contain outliers, unusual values that don’t make sense compared to the rest of the data.
For example, if most employees work between 30–60 hours per week, but one record shows 20 hours or 100 hours, those are clear outliers.
Task

Load the dataset using Pandas.
Display the first 5 rows of the dataset to understand its structure.
Visualize the Hours_Worked column using a boxplot to identify outliers.
Remove outliers by keeping only employees who worked between 30 and 60 hours per week.
Display the cleaned dataset without outliers.

##### HERE IS CODE ####
import pandas as pd
import matplotlib
matplotlib.use('Agg')  # Use non-interactive backend
import matplotlib.pyplot as plt

def analyze_data(filePath):

    # Step 1: Load csv file
    df = pd.read_csv(filePath)
    print("\nFirst 5 rows of the dataset:\n")
    print( df.head() )   # ← Fill in

    # Step 2: Visualize with outliers
    plt.boxplot(df["Hours_Worked"])  
    plt.title("Working Hours Distribution (with Outliers)")
    plt.savefig("hours_outliers.png")
    plt.close()
    print("\nBoxplot saved as 'hours_outliers.png'")

    # Step 3: Remove outliers (keep only 30–60)
    cleaned_df = df[( df["Hours_Worked"] >= 30) & ( df["Hours_Worked"] <= 60 )]   # ← Fill in
    print(df.shape)
    print(cleaned_df)
    print("\nDataset after removing outliers:\n")
    
    print( cleaned_df.shape )   # ← Fill in

    # return cleaned_df

def main():
    analyze_data("working_hours.csv")

if __name__ == "__main__":
    main()
###################################
Dealing with Outliers - Practice Problem 2
We are given a dataset called employee_salaries.csv.
This dataset contains information about employees and their salaries.

Sometimes, datasets contain outliers, which are unusual values compared to the rest of the data.
For example, if most employees earn between 30,000 and 100,000 but one record shows 10,000 or 200,000, those are clear outliers.

Task

Load the dataset using Pandas.
Display the first 5 rows of the dataset to understand its structure.
Visualize the Salary column using a boxplot to identify outliers.
Remove outliers by keeping only employees who have salaries between 30,000 and 100,000.
Display the cleaned dataset without outliers.
Return the cleaned DataFrame.

### HERE IS THE SOLUTION ####

import pandas as pd   # TODO: Import pandas library
import matplotlib.pyplot as plt

def analyze_salaries(filePath):   # TODO: Add function parameter (file path)

    # Step 1: Load csv file
    df = pd.read_csv("employee_salaries.csv")   # TODO: Load the CSV file using pandas
    print("\nFirst 5 rows of the dataset:\n")
    print(df.head())   # TODO: Print the first 5 rows

    # Step 2: Visualize with outliers
    plt.boxplot(df["Salary"])   
    plt.title("Salary Distribution (with Outliers)")
    plt.savefig("salary_outliers.png")   
    plt.close()
    print("\nBoxplot saved as 'salary_outliers.png'")

    # Step 3: Remove outliers (keep only 30000–100000)
    # TODO: Filter dataset to keep only rows where Salary is between 30000 and 100000
    cleaned_df = df[(df["Salary"] >= 30000) & (df["Salary"] <= 100000)]   
    print("\nDataset after removing outliers:\n")
    print(cleaned_df)
    print(cleaned_df.shape)
    # TODO return the cleaned DataFrame
    return cleaned_df


def main():
    # TODO: Call the function with the dataset path
    analyze_salaries("employee_salaries.csv")   

if __name__ == "__main__":
    main()
##########################33333333#####################
Mini Project - Clean and Explore Employee Dataset
You are given a dataset called employee_practice.csv with the following columns:

Name	Age	Department	Salary	Years_Experience	Promotion_Eligibility
John	28	HR	50000	2	No
Sophia	34	IT	80000	5	Yes
Amit	25	Finance	60000	3	No
Emma	40	HR	90000	6	Yes
Liam	30	IT	75000	4	No
Task

1. Load the dataset

Load employee_practice.csv using pandas.
2. Identify Features & Label

Features: All columns except Promotion_Eligibility
Label: Promotion_Eligibility
Print both features and label.
3. Handle Missing Values

For numerical columns (Age, Years_Experience, Salary): fill missing values with median.
For categorical columns (Department, Promotion_Eligibility): fill missing values with mode.
Print missing values before and after handling.
4. Detect & Handle Outliers

Salary: cap values < 30000 → 30000, values > 150000 → 150000
Years_Experience: cap > 30 → 30
Print outliers before handling.
5. Visualize Salary

Use a boxplot to show Salary distribution after handling outliers.

### HERE IS THE CODE  ######

import pandas as pd
import matplotlib
import matplotlib.pyplot as plt

# -----------------------------
# Step 1: Load Dataset
# -----------------------------
df = pd.read_csv("employee_practice.csv")  # <-- Fill: load CSV file 
print("First 5 rows:\n", df.head())

# -----------------------------
# Step 2: Identify Features & Label and Print features columns and label name
# -----------------------------
features = df.drop("Promotion_Eligibility", axis=1)  
label = df["Promotion_Eligibility"]  

print("\nFeatures:\n", features.columns)  # <-- Fill: print feature columns 
print("Label:\n", label.name)   # <-- Fill: print label name 

# -----------------------------
# Step 3: Handle Missing Values
# -----------------------------
print("\nMissing Values Before:\n", df.isnull().sum())  # <-- Fill: check missing values 

# Fill numerical missing values with median
df["Age"] = df["Age"].fillna(df["Age"].median())  # <-- Fill: handle missing Age 
df["Years_Experience"] = df["Years_Experience"].fillna(df["Years_Experience"].median())  # <-- Fill: handle missing Years_Experience 

# Fill categorical missing values with mode
df["Department"] = df["Department"].fillna(df["Department"].mode()[0])  # <-- Fill: handle missing Department 
df["Promotion_Eligibility"] = df["Promotion_Eligibility"].fillna(df["Promotion_Eligibility"].mode()[0])  # <-- Fill: handle missing Promotion_Eligibility 

print("\nMissing Values After:\n", df.isnull().sum())

# -----------------------------
# Step 4: Detect & Handle Outliers (Simple Method)
# -----------------------------
# Identify Salary outliers (e.g., Salary < 30000 or Salary > 150000)
salary_outliers = df[(df["Salary"] < 30000) | (df["Salary"] > 150000)]  # <-- Fill: set low and high threshold values
print("\nSalary Outliers:\n", salary_outliers)

# Remove Salary outliers 
df = df[(df["Salary"] >= 30000) & (df["Salary"] <= 150000)]

# Handle Years_Experience outliers (remove > 30 years)
df = df[df["Years_Experience"] <= 30]


# -----------------------------
# Step 5: Handling Outliers
# -----------------------------
print("\nDescriptive Statistics:\n", df.describe())  # <-- Fill: get descriptive stats 
print("\nEmployees per Department:\n", df["Department"].value_counts())  # <-- Fill: count employees 

plt.figure(figsize=(6,4))
plt.boxplot(df["Salary"])  # <-- Fill: column to visualize "Salary"
plt.title("Salary Distribution After Handling Outliers")
plt.ylabel("Salary")   # <-- Fill: y-axis label ("Salary")
plt.savefig("salary_boxplot.png")  # Save the image
plt.show()

def main():
    analyze_data("employee_practice.csv")

if __name__ == "__main__":
    main()

#####################################