Course project for McGill COMP 551: Applied Machine Learning — Mini-Project 2, Group 49, Fall 2020 (3-person team).
This repository contains a from-scratch NumPy implementation of multi-class logistic regression (softmax regression) trained with mini-batch stochastic gradient descent (SGD) and momentum. The full write-up is available in Report/writeup.pdf.
Hand-rolling gradient descent when PyTorch exists: character building, NumPy style.
The project implements two core components from scratch:
- Multi-class softmax regression (
code/softmax_Regression.py) — one-hot encoded targets, softmax probabilities, and cross-entropy gradient. - Mini-batch SGD with momentum (
code/GradientDescent.py) — random mini-batch creation, velocity-style weight update, and a custom termination condition based on 5-fold cross-validation accuracy.
A custom grid-search hyper-parameter optimizer (code/hyper_optimization.py) searches over batch size, learning rate, and momentum using 5-fold cross-validation. Performance is compared against off-the-shelf scikit-learn K-Nearest Neighbours and Decision Tree classifiers (code/comparison_other_classifier.py).
| Dataset | Source | Instances | Features | Classes | Notes |
|---|---|---|---|---|---|
| Scikit-learn Digits | sklearn.datasets.load_digits |
1,797 | 64 (8×8 pixel images) | 10 (digits 0–9) | Already pre-processed by scikit-learn; no missing values. |
| OpenML Cardiotocography | fetch_openml(name='cardiotocography', version=2) |
2,126 | 35 | 3 (fetal states N, S, P) | Non-numeric metadata removed; min-max normalized to prevent softmax overflow. |
Dataset loading and normalization are handled in code/loader.py.
- A bias column is appended to the feature matrix during
fitandpredict. - Targets are one-hot encoded with
sklearn.preprocessing.OneHotEncoder. - Predictions are made by selecting the class with the highest softmax probability.
The weight update rule in code/GradientDescent.py is:
d_w = beta * w + (1 - beta) * grad
w = w - learning_rate * d_w
where grad is the average cross-entropy gradient over the current mini-batch and beta is the momentum coefficient.
Training stops when any of the following is true:
- The L2 norm of the gradient falls below
epsilon(default1e-5). - The number of iterations reaches
max_iter(default1e4). - The mean 5-fold cross-validation accuracy is no longer strictly increasing over the last 20 iterations.
The third condition returns the model at the best cross-validation accuracy rather than the final gradient-descent iterate, and it reduces runtime when convergence is slow.
code/hyper_optimization.py performs a grid search with 5-fold cross-validation over:
| Hyper-parameter | Search range | Step size |
|---|---|---|
| Batch size | 50 to dataset size | 100 |
| Learning rate | 0.01 to 0.1 | 0.018 |
| Momentum beta | 0.9 to 0.99 | 0.018 |
The configuration with the highest average cross-validation accuracy is selected.
| Dataset | Batch Size | Learning Rate | Momentum | Runtime | Accuracy |
|---|---|---|---|---|---|
| Digits | 950 | 0.027 | 0.9 | 3.7 s | 92.7% |
| Cardiotocography | 50 | 0.010 | 0.9 | 4.9 s | 77.8% |
L1 and L2 regularization (lambda = 0.1) were implemented but did not significantly improve accuracy; the report suggests this is likely because the softmax cross-entropy objective already has a unique global minimum in weight space.
| Dataset | K-NN (optimal k) | Decision Tree (optimal depth) | Our Softmax Regression |
|---|---|---|---|
| Digits | 98.9% (k = 1) | 91.7% (depth = 20) | 92.7% |
| Cardiotocography | 99.1% (k = 3) | 98.6% (depth = 5) | 77.8% |
On the Digits dataset, the softmax regressor outperformed the decision tree. On Cardiotocography, both off-the-shelf classifiers scored higher.
.
├── code/
│ ├── softmax_Regression.py # Softmax regression model
│ ├── GradientDescent.py # Mini-batch SGD with momentum and 5-fold CV termination
│ ├── hyper_optimization.py # Grid-search hyper-parameter optimizer
│ ├── loader.py # Dataset loading and min-max normalization
│ ├── comparison_other_classifier.py # K-NN and Decision Tree baselines
│ └── main.py # Example end-to-end run
├── Report/
│ └── writeup.pdf # Full project report
└── README.md
Run the complete pipeline from the code/ directory:
cd code
python main.pymain.py performs the following steps:
- Loads both datasets via
loader.Loader. - Min-max normalizes the Cardiotocography features.
- Trains the softmax regressor on Digits with default gradient descent.
- Runs the hyper-parameter optimizer on Digits.
- Compares K-NN and Decision Tree baselines on Cardiotocography.
To train on a different dataset or use the optimized hyper-parameters, instantiate the classes directly:
from GradientDescent import GradientDescent
from softmax_Regression import softmax_Regression
from loader import Loader
loader = Loader()
X, y = loader.numpy_dataset01() # or numpy_dataset02()
gd = GradientDescent(batch_size=950, learning_rate=0.027, momentum_beta=0.9)
model = softmax_Regression()
w_opt, encoder = model.fit(X, y, gd)
y_pred = model.predict(X, w_opt)
y_pred = encoder.inverse_transform(y_pred)- Python 3
- NumPy
- scikit-learn
- pandas
- Matplotlib
Install with pip:
pip install numpy scikit-learn pandas matplotlib- Cardiotocography dataset: https://www.openml.org/d/1466
- Full project report: Report/writeup.pdf
