Data Analysis &
Visualization — Mid-Sem Guide

Goal: Discover natural groupings in data without labels.

Applications: Customer segmentation, document grouping, gene expression analysis, anomaly detection, image segmentation.

Characteristics of Good Clustering:

1. High intra-cluster similarity (within group, objects are similar).
2. Low inter-cluster similarity (between groups, objects are dissimilar).
3. Scalable to large datasets.
4. Ability to handle noise and outliers.
5. Interpretable results.

Distance Measures:

Types of Clustering:

Steps:

1. Choose k (number of clusters).
2. Randomly initialize k centroids.
3. Assignment step: Assign each data point to the nearest centroid.
4. Update step: Recalculate each centroid as the mean of all points assigned to it.
5. Repeat steps 3–4 until centroids no longer change (convergence).

Limitations:

• Must specify k in advance.
• Sensitive to initial centroid placement.
• Assumes spherical clusters of similar size.
• Sensitive to outliers (outlier can pull centroid).
• Not suitable for non-convex shaped clusters.

Key Parameters:

• ε (epsilon): Radius of the neighborhood around a point.
• MinPts: Minimum number of points required to form a dense region (core point).

Point Types:

1. Core Point: Has at least MinPts points within distance ε (including itself).
2. Border Point: Within ε of a core point, but has fewer than MinPts neighbors.
3. Noise Point (Outlier): Not a core point and not within ε of any core point.

Algorithm:

1. Start with an unvisited point.
2. If it's a core point, create a new cluster and expand it by adding all density-reachable points.
3. If it's a border point, assign to nearest cluster.
4. If it's a noise point, label it as noise.
5. Repeat for all unvisited points.

Advantages over K-Means: Discovers clusters of arbitrary shape, automatically finds number of clusters, handles noise/outliers well.

Disadvantage: Struggles with varying density clusters and high-dimensional data.

How it works:

1. Place a kernel (smooth bump function) centered at each data point.
2. Sum all kernels — regions with many points have high density.
3. Peaks in the density estimate suggest cluster centers.

Bandwidth (h): Controls smoothness. Small h → spiky/overfit. Large h → over-smooth. Chosen by cross-validation.

Use in Clustering: KDE identifies modes (peaks) in the density, which correspond to cluster centers. Useful for finding natural groups without specifying k.

1. Agglomerative (Bottom-up) — Most Common:

1. Start: each point is its own cluster.
2. Find the two closest clusters and merge them.
3. Repeat until all points are in one cluster.
4. Cut the dendrogram at desired level to get clusters.

Linkage Criteria (how to measure distance between clusters):

2. Divisive (Top-down): Start with all points in one cluster, recursively split into smaller clusters. Less common.

Strengths: No need to specify k, produces intuitive dendrogram, can capture hierarchical structure.
Weaknesses: O(n²) time complexity, sensitive to noise, cannot undo a wrong merge.

How It Works:

1. Bootstrap Sampling (Bagging): Create n random subsets of the training data (with replacement). Train one decision tree on each subset.
2. Feature Randomness: At each node split, only a random subset of features is considered (typically √total_features for classification). This decorrelates the trees.
3. Prediction:
   • Classification: Majority vote across all trees.
   • Regression: Average of all tree predictions.

Training Data
     │
     ├── Bootstrap Sample 1 → Decision Tree 1 → Prediction 1
     ├── Bootstrap Sample 2 → Decision Tree 2 → Prediction 2
     ├── Bootstrap Sample 3 → Decision Tree 3 → Prediction 3
     │        ...                  ...               ...
     └── Bootstrap Sample n → Decision Tree n → Prediction n
                                                      │
                                              Majority Vote / Average
                                                      │
                                               FINAL PREDICTION

Key Hyperparameters:

• n_estimators: Number of trees. More trees = better but slower.
• max_features: Number of features considered at each split.
• max_depth: Maximum depth of each tree (controls overfitting).

Advantages:

• Handles high-dimensional data well.
• Resistant to overfitting (due to averaging).
• Can compute feature importance.
• Works well even without parameter tuning.
• Handles missing values and maintains accuracy for large datasets.

Disadvantages:

• Not interpretable (black box).
• Computationally expensive for very large forests.
• Slower than a single decision tree.

Classification — Two-Step Process:

1. Training (Model Construction): A classifier is built using labeled training data. The model learns patterns that distinguish between classes.
2. Testing (Prediction): The trained model is applied to new, unseen data to predict class labels. Accuracy is evaluated on the test set.

Signs of Overfitting: Very high training accuracy, much lower test accuracy. The gap between training error and test error is large.

Causes: Too many features, too deep a model, too small a training dataset, training for too long.

Methods to Avoid Overfitting:

1. Cross-Validation: Use k-fold CV to evaluate model on multiple subsets of data.
2. Pruning: In decision trees, remove branches that have little power to classify data.
3. Regularization: Add a penalty for model complexity (L1/Lasso, L2/Ridge).
4. Feature Selection: Remove irrelevant or redundant features.
5. Get More Data: More training examples help the model generalize.
6. Ensemble Methods: Combine many models (e.g., Random Forest) — averaging reduces variance.
7. Early Stopping: In neural networks, stop training when validation error starts increasing.
8. Dropout: In neural networks, randomly deactivate neurons during training.

Basic Concept: Given a new unknown point, find the k most similar (nearest) known points and use their characteristics to predict or describe the unknown.

Applications:

• Geographic analysis (nearest hospital, nearest store).
• Recommendation systems (users similar to you liked X).
• Anomaly detection (point with no near neighbors = outlier).
• Data imputation (fill missing values with nearest neighbor's value).

Types of NN Analysis:

1. 1-NN: Find the single nearest neighbor.
2. K-NN: Find k nearest neighbors and vote.
3. Average NN: Use average of multiple neighbors.
4. Radius-NN: All points within a distance r.

How ANN Works:

1. For a new query point q, find all neighbors within a radius or find k nearest neighbors.
2. Compute the average feature values (or average distance) of those neighbors.
3. Assign the class that is most represented on average.
4. Or, for regression, predict the average target value of the neighbors.

Example:

ANN vs Standard NN: Standard NN uses only the single closest point and is sensitive to noise. ANN uses multiple neighbors, making it more robust. However, ANN requires more computation.

Algorithm Steps:

1. Choose the value of k.
2. Calculate the distance from the new point to every training point.
3. Sort the distances and identify k nearest neighbors.
4. For classification: take majority vote among k neighbors.
5. For regression: compute average of k neighbors' values.

Choosing k:

• Small k (k=1): Very sensitive to noise, complex boundary.
• Large k: Smoother boundary, may include irrelevant neighbors.
• Rule of thumb: k = √n. Always choose odd k for binary classification (avoid ties).
• Best: Use cross-validation to find optimal k.

Feature Scaling in KNN: Since KNN is distance-based, attributes with larger ranges dominate. Always normalize features before applying KNN. Use Min-Max normalization: x' = (x − min) / (max − min).

Data Science for E-commerce Growth:

1. Customer Segmentation (Clustering): K-Means segments customers by RFM (Recency, Frequency, Monetary value). Different marketing strategies for each segment (VIP customers, discount-seekers, churned customers).
2. Recommendation Systems (KNN / Collaborative Filtering): "Customers like you also bought X." Netflix-style recommendations based on user similarity. Increases average order value.
3. Churn Prediction (Classification): Predict which customers are likely to stop buying. Logistic regression / Random Forest on behavioral data. Proactive retention: targeted discounts, emails.
4. Fraud Detection (Outlier Detection / Classification): Unusual transactions flagged as potential fraud. Isolation Forest, DBSCAN for anomaly detection. Saves revenue and protects customers.
5. Demand Forecasting (Time Series): ARIMA/LSTM models predict future demand. Optimizes inventory, reduces stockouts and overstock.
6. A/B Testing (Inferential Statistics): Test two versions of a webpage and determine which converts better. Statistical significance testing (p-values, confidence intervals).
7. Price Optimization (Regression): Predict optimal prices using regression models on competitor prices, demand elasticity, and seasonality.
8. AARRR Funnel (E-commerce Metrics):
   • Acquisition: How do users find you?
   • Activation: Do they have a good first experience?
   • Retention: Do they come back?
   • Referral: Do they tell others?
   • Revenue: Do they pay?