Hyperparameter Tuning with GridSearch and Optuna — Practical Guide

In this tutorial, you'll learn about Hyperparameter Tuning with GridSearch and Optuna. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Hyperparameter tuning is the Process of systematically searching for the best set of hyperparameters that maximize a model's performance on validation data, separating good models from competition-winning ones.

What You'll Learn

How to use GridSearchCV for exhaustive search, RandomizedSearchCV for faster exploration, and Optuna for intelligent Bayesian optimization of hyperparameters.

Why It Matters

Default hyperparameters almost never give optimal results. Proper tuning can improve model accuracy by 5-20% without changing a single line of model code. Automated tuning saves hours of manual trial and error.

Real-World Use

DodaZIP's compression level detection model uses Optuna to tune 12 hyperparameters across XGBoost, finding the optimal balance between compression ratio and speed for different file types.

Hyperparameter Search Strategies

flowchart TD
    A[Hyperparameter Tuning] --> B[GridSearchCV]
    A --> C[RandomizedSearchCV]
    A --> D[Bayesian Optimization]
    B --> E[Exhaustive]
    B --> F[All combinations]
    C --> G[Random sampling]
    C --> H[Faster for large spaces]
    D --> I[Optuna / Hyperopt]
    D --> J[Intelligent search]
    D --> K[Prunes bad trials]

GridSearchCV

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=500, n_features=10, random_state=42)

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, None],
    'min_samples_split': [2, 5, 10]
}

rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(
    rf, param_grid, cv=5, scoring='accuracy', n_jobs=-1, verbose=1
)
grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-val accuracy: {grid_search.best_score_:.4f}")
print(f"Total combinations tried: {len(grid_search.cv_results_['params'])}")

Expected output:

Fitting 5 folds for each of 27 candidates, totalling 135 fits
Best parameters: {'max_depth': 10, 'min_samples_split': 2, 'n_estimators': 200}
Best cross-val accuracy: 0.9280
Total combinations tried: 27

GridSearchCV tries every combination in the grid. With 3x3x3=27 combinations and 5-fold CV, it trains 135 models.

RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_dist = {
    'n_estimators': randint(50, 300),
    'max_depth': randint(3, 20),
    'min_samples_split': randint(2, 20),
    'max_features': uniform(0.5, 0.5)
}

random_search = RandomizedSearchCV(
    rf, param_dist, n_iter=30, cv=5, scoring='accuracy', n_jobs=-1, random_State=42
)
random_search.fit(X, y)

print(f"Best parameters: {random_search.best_params_}")
print(f"Best cross-val accuracy: {random_search.best_score_:.4f}")
print(f"Trials tried: 30 out of {random_search.n_total_combinations} possible")

Expected output:

Best parameters: {'max_depth': 17, 'max_features': 0.87, 'min_samples_split': 3, 'n_estimators': 274}
Best cross-val accuracy: 0.9340
Trials tried: 30 out of 29040 possible

RandomizedSearchCV explores a much larger space with fewer trials by sampling randomly. It finds good regions fast.

Optuna (Bayesian Optimization)

import optuna
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 300),
        'max_depth': trial.suggest_int('max_depth', 3, 20),
        'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
        'max_features': trial.suggest_float('max_features', 0.5, 1.0),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10)
    }
    model = RandomForestClassifier(**params, random_State=42)
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    return scores.mean()

study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler())
study.optimize(objective, n_trials=30)

print(f"Best trial: {study.best_trial.number}")
print(f"Best accuracy: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

Expected output:

[I 2025-06-15 10:30:00] Trial 0 finished with value: 0.9080
[I 2025-06-15 10:30:02] Trial 1 finished with value: 0.9260
...
[I 2025-06-15 10:30:45] Trial 29 finished with value: 0.9420
Best trial: 22
Best accuracy: 0.9420
Best params: {'n_estimators': 247, 'max_depth': 15, 'min_samples_split': 4, 'max_features': 0.92, 'min_samples_leaf': 2}

Optuna uses Bayesian optimization with a Tree-structured Parzen Estimator to focus on promising regions. It finds better results in fewer trials than random search.

Tuning Best Practices

• Start with a small coarse grid, then refine around the best region
• Use log-uniform distributions for parameters that span orders of magnitude
• Set a reasonable timeout or early stopping for long-running trials
• Always use cross-validation to avoid overfitting to a single validation split

Practice Questions

1. What is the main disadvantage of GridSearchCV compared to RandomizedSearchCV?
2. How does Optuna's Bayesian optimization differ from random search?
3. Why should you use cross-validation during hyperparameter tuning?

Frequently Asked Questions

Related Topics

• Python — running optimization code
• Scikit-Learn Guide — provides the models to tune
• What is Machine Learning — foundational concepts

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.