Optuna for Agentic AI: A Practical Guide to Hyperparameter Optimization of LLM Pipelines

Optuna is a mature, MIT-licensed Python framework for automated hyperparameter search; version 4.8.0 (March 2026) is the current stable baseline. Although it was designed for machine-learning model training, its architecture — define-by-run studies, composable samplers, persistent storage, and a real-time dashboard — transfers naturally to any measurable optimization problem, including the configuration of agentic AI pipelines. The two things that actually determine whether a study produces trustworthy results are not Optuna features: you need a repeatable batch harness that re-runs a pipeline segment with different configurations, and a labeled test set with a stable scalar metric that honestly reflects quality. Without those, trial scores are noise. And budget honestly: a Claude-class study realistically costs $10–$600, not the sub-dollar figures cited in fine-tuning literature for small open models.

The applications to agentic engineering described here are novel proposed patterns without published precedent as of mid-2026 — architectural possibilities, not validated production recipes. This guide is written for a Python-fluent reader new to AutoML; code uses Optuna 4.x idioms throughout.

Introduction: Optuna and AutoML

What is hyperparameter optimization?

Most software systems — and all machine-learning models — have configuration values that are not learned from data but must be set before a run begins. In ML these are hyperparameters: learning rate, batch size, regularization strength. Hyperparameter optimization (HPO) searches the space of possible values to find a configuration that maximizes some objective metric.

The same idea applies outside ML. Any repeatable process with tunable parameters and a measurable outcome is a candidate for HPO. The "model" being tuned could be a language-model prompting pipeline, a retry-and-routing loop, or a temperature-and-top-p configuration for a chain of agents.

Where Optuna fits

AutoML is a broad field encompassing neural architecture search, feature engineering, pipeline assembly, and hyperparameter optimization. Optuna occupies the HPO corner of this space. It does not build models, select features, or choose algorithms — it efficiently searches a parameter space you define, using whichever search algorithm you configure.

Optuna was created by Preferred Networks and first released in 2019. As of mid-2026 it is at version 4.8.0, maintained as an open-source project under the MIT license, and requires Python 3.9 or higher. Its "define-by-run" design philosophy means the parameter space is constructed dynamically as each trial executes, rather than being declared as a static schema up front — especially flexible for workflows where the shape of the search space depends on other parameter choices (hierarchical or conditional spaces).

Why this matters for agentic engineering

Building agentic systems involves many configuration decisions: how many retries to allow before escalating, what temperature to use for different node roles, how much context to include in a synthesis prompt, how to weight competing objectives in a routing decision. These decisions are usually made by intuition or trial and error. Optuna offers a systematic, reproducible alternative — provided you can measure what "better" means.

Core concepts and API

Studies, trials, and objectives

The central object in Optuna is a study, which represents a complete optimization campaign. A study contains many trials, each a single evaluation of one parameter configuration. The study is created with a direction ("minimize" or "maximize") and a storage backend.

import optuna

def objective(trial):
    # Suggest a value from the search space
    temperature = trial.suggest_float("temperature", 0.0, 1.5)
    max_tokens = trial.suggest_int("max_tokens", 256, 4096, step=256)
    model_tier = trial.suggest_categorical("model_tier", ["haiku", "sonnet", "opus"])

    # Evaluate the configuration — your pipeline call goes here
    score = evaluate_pipeline(temperature, max_tokens, model_tier)
    return score

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

print(study.best_params)
print(study.best_value)

Suggest methods

Optuna provides typed suggestion methods that define the search space:

Method	Description	Example
suggest_float(name, low, high)	Continuous float	temperature, penalty weight
suggest_float(name, low, high, log=True)	Log-scale float	learning rates, small positive values
suggest_int(name, low, high)	Integer	token limits, retry counts
suggest_categorical(name, choices)	Discrete set	model names, sampler types

Samplers

A sampler determines which parameter configuration to try next, given the history of completed trials. The default sampler is TPE (Tree-structured Parzen Estimator), a Bayesian method that builds a probabilistic model of which regions of the search space tend to produce good results.

Sampler	When to use
TPE (default)	General single-objective optimization; good starting point
NSGA-II	Multi-objective optimization (Pareto front)
GPSampler + ConstantLiar	Asynchronous parallel trials
CmaEsSampler	High-dimensional continuous spaces (large trial budgets)
RandomSampler	Baselines, sanity checks
LLAMBO (OptunaHub)	Semantic/categorical-heavy spaces where LLM priors help

Caveat — TPE warm-up: TPE's probabilistic model benefits from roughly 10–25 initial trials before it reliably outperforms random sampling. This is a practitioner rule-of-thumb, not an official specification — the exact count depends on parameter-space dimensionality and structure.

Caveat — CMA-ES population size: CMA-ES maintains a population of candidate solutions; the effective batch size per generation is governed by the population-size parameter (lambda, approximately 4 + floor(3 × log(n_dims))). This is a population-sizing concept, not a fixed warm-up trial count.

Pruning

Optuna supports pruning — early termination of trials that are clearly underperforming. Pruning is relevant for long-running objective functions where it is worth stopping a bad configuration partway through. In agentic pipeline evaluation, pruning can be applied after evaluating the first few test cases in a batch, before completing the full evaluation set.

def objective(trial):
    for step, test_case in enumerate(test_cases):
        score = run_single_case(trial.params, test_case)
        trial.report(score, step)
        if trial.should_prune():
            raise optuna.TrialPruned()
    return aggregate_score

Storage backends

Optuna persists study state to a storage backend, enabling resumable studies and multi-process parallelism.

• SQLite (sqlite:///db.sqlite3) — zero-configuration, suitable for single-machine studies
• PostgreSQL/MySQL — recommended for multi-process or distributed studies
• JournalStorage — modern v4.x file-based or Redis-based journal pattern; the preferred lightweight option for local work

Caveat: RedisStorage as a direct class may be legacy or deprecated in v4.x. Prefer JournalStorage with a JournalRedisBackend when Redis persistence is needed. Confirm the current API in the v4.8.0 documentation before using Redis-backed storage in production.

# Modern v4.x lightweight storage pattern
from optuna.storages import JournalStorage, JournalFileBackend

storage = JournalStorage(JournalFileBackend("./optuna-journal.log"))
study = optuna.create_study(storage=storage, study_name="my-pipeline-study")

The Optuna ecosystem

OptunaHub

OptunaHub is a community feature-sharing registry, officially released in August 2024. It provides additional samplers, pruners, and utilities beyond the core package. Notable contributions relevant to agentic engineering:

• LLAMBO — a sampler that uses a language model's priors to guide search, particularly useful in spaces with semantic structure (e.g., choosing between prompt templates or model configurations)
• SPEA-II — an alternative multi-objective evolutionary sampler
• HypE — hypervolume-based evolutionary multi-objective optimizer
• EvoMergeSampler — evolutionary model-merging

OptunaHub modules are loaded at runtime:

import optunahub

module = optunahub.load_module("samplers/llambo")
sampler = module.LLAMBOSampler()
study = optuna.create_study(sampler=sampler)

Caveat: optunahub is in the 0.4.x series as of early 2026; confirm the exact current version on PyPI before pinning dependencies.

Optuna Dashboard

The Optuna Dashboard is a real-time visualization interface for monitoring running and completed studies. It connects directly to the study's storage backend and requires no additional instrumentation in the objective function.

# Install
pip install optuna-dashboard

# Launch (pointing at an existing SQLite study database)
optuna-dashboard sqlite:///db.sqlite3

The dashboard displays trial history, parameter-importance rankings, parallel-coordinate plots, and the empirical distribution of parameter values. For iterative tuning sessions — where understanding why certain configurations perform well matters as much as finding the optimum — the dashboard is a significant productivity aid.

Caveat: The dashboard's LLM integration feature (natural-language trial filtering, auto graph generation) is a feature of the dashboard package, not the Optuna framework core. The dashboard and the core framework are versioned independently. Confirm the exact dashboard version on PyPI.

optuna-mcp

optuna-mcp is an official Optuna project that exposes Optuna study management as a Model Context Protocol (MCP) server, enabling language-model agents to interact with studies directly. It lets an agent suggest parameters, report trial results, and query study state via MCP tool calls.

Caveat: optuna-mcp is an optional convenience layer for LLM-native workflows. It is not a primary dependency for Optuna itself, and its stability in production agentic pipelines should be independently validated before reliance. For users already operating in an MCP-native environment it warrants evaluation; it should not be treated as a requirement.

{
  "mcpServers": {
    "Optuna": {
      "command": "uvx",
      "args": ["optuna-mcp", "--storage", "sqlite:///optuna.db"]
    }
  }
}

Integration packages

Optuna v4.x splits framework-specific integrations into separate optuna-integration-* packages. Old import paths under optuna.integration.* are legacy shims and may be removed in future releases. When using any integration (e.g., PyTorch, TensorFlow, LightGBM), install the appropriate split package and use its canonical import path.

Installation and getting started

# Core installation
pip install optuna

# Optional: dashboard
pip install optuna-dashboard

# Optional: hub for community samplers
pip install optunahub

# Optional: MCP server (evaluate stability before production use)
# pip install optuna-mcp   # Or: uvx optuna-mcp (via uv)

Optuna 4.8.0 requires Python 3.9 or higher. Verify with:

python --version
python -c "import optuna; print(optuna.__version__)"

Minimal working example

A complete study with persistent storage and dashboard-ready output:

import optuna
from optuna.storages import JournalStorage, JournalFileBackend

# Create persistent storage
storage = JournalStorage(JournalFileBackend("./pipeline-study.log"))

# Define objective
def objective(trial):
    temperature = trial.suggest_float("temperature", 0.3, 1.2)
    context_tokens = trial.suggest_int("context_tokens", 512, 8192, step=512)
    retry_budget = trial.suggest_int("retry_budget", 1, 5)

    # Replace with your actual evaluation function
    score = mock_pipeline_eval(temperature, context_tokens, retry_budget)
    return score

# Create and run study
study = optuna.create_study(
    study_name="pipeline-v1",
    storage=storage,
    direction="maximize",
    load_if_exists=True  # Resume if study already exists
)

study.optimize(objective, n_trials=30, timeout=3600)

# Inspect results
print("Best parameters:", study.best_params)
print("Best score:", study.best_value)

# Parameter importance
importances = optuna.importance.get_param_importances(study)
for param, importance in importances.items():
    print(f"  {param}: {importance:.3f}")

After running at least one trial, launch the dashboard and navigate to http://localhost:8080:

optuna-dashboard sqlite:///db.sqlite3

Applications to agentic AI engineering

Important caveat: The agentic applications in this section are novel proposed patterns. As of mid-2026, no published research or production case studies document Optuna being used to optimize agent skill pipelines or similar agentic architectures. These are architectural possibilities derived from Optuna's general design and first principles of agentic engineering. Treat them as design hypotheses to validate experimentally, not as established practice.

Why agentic pipelines are optimization candidates

An agent skill pipeline has many tunable knobs: the temperature of synthesis nodes, the token budget allocated to different stages, the threshold at which a verification node triggers a re-spawn, the number of parallel branches to maintain. These are typically set by human judgment. If you can (a) run the pipeline repeatably on a fixed test set and (b) score its output with a scalar metric, Optuna can search this space more systematically than manual iteration.

A documented precedent

One documented example of an LLM agent acting as an Optuna orchestrator is described in a Hugging Face blog post, where the combination managed distributed HPO runs for LLM fine-tuning on Hugging Face Jobs, achieving a meaningful improvement in evaluation loss within two trials. This example involved open-source model fine-tuning, not prompt-only pipeline optimization — but it demonstrates the viability of the agent-as-orchestrator pattern.

For multi-objective agent evaluation, NSGA-II produces a Pareto front of non-dominated configurations — the set of points where you cannot improve one objective without degrading the other. This is useful when optimizing simultaneously for output quality and API cost:

study = optuna.create_study(
    directions=["maximize", "minimize"],  # quality, cost
    sampler=optuna.samplers.NSGAIISampler()
)
study.optimize(multi_objective_objective, n_trials=100)

# Inspect Pareto front
pareto_trials = study.best_trials
for trial in pareto_trials:
    print(f"Quality: {trial.values[0]:.3f}, Cost: ${trial.values[1]:.4f}")

Candidate optimization targets

The following are proposed novel applications. Each is architecturally coherent; none has published benchmarks.

Prompt parameter tuning

Parameters like temperature, max_tokens, and system-prompt variants are natural Optuna candidates:

def prompt_pipeline_objective(trial):
    temperature = trial.suggest_float("temperature", 0.0, 1.5)
    max_tokens = trial.suggest_int("max_tokens", 512, 4096, step=256)
    prompt_variant = trial.suggest_categorical(
        "prompt_variant",
        ["baseline", "chain-of-thought", "step-by-step", "role-primed"]
    )

    scores = []
    for test_case in LABELED_TEST_SET:
        result = call_model(
            prompt=build_prompt(prompt_variant, test_case["input"]),
            temperature=temperature,
            max_tokens=max_tokens
        )
        scores.append(score_output(result, test_case["expected"]))

    return sum(scores) / len(scores)

Retry budget and escalation thresholds

In a pipeline with fallback logic (retry with a stronger model on failure), the retry count and confidence threshold before escalation can be tuned jointly:

def retry_policy_objective(trial):
    max_retries = trial.suggest_int("max_retries", 1, 5)
    confidence_threshold = trial.suggest_float("confidence_threshold", 0.5, 0.95)
    escalation_model = trial.suggest_categorical(
        "escalation_model", ["haiku", "sonnet", "opus"]
    )
    quality, cost = evaluate_retry_policy(max_retries, confidence_threshold, escalation_model)
    return quality  # Or multi-objective: return quality, -cost

Retrieval and routing thresholds

For skills with knowledge-base routing, retrieval parameters are Optuna candidates:

def routing_objective(trial):
    kb_top_k = trial.suggest_int("kb_top_k", 1, 20)
    similarity_threshold = trial.suggest_float("sim_threshold", 0.5, 0.99)
    confidence_cutoff = trial.suggest_float("route_confidence", 0.4, 0.95)

    routing_accuracy = evaluate_routing(kb_top_k, similarity_threshold, confidence_cutoff)
    return routing_accuracy

The prerequisite stack

Before any of the above produces reliable results, four prerequisites must be in place:

1. Batch harness — a script that runs the pipeline segment repeatably with arbitrary parameter configurations, returning a score.
2. Labeled test set — a fixed set of inputs with known-good or human-rated expected outputs.
3. Scalar metric — a single number (or fixed tuple for multi-objective) that honestly captures quality.
4. Stable environment — API access, dependencies, and pipeline code deterministic enough that score variance across reruns of the same config is small relative to the differences between configs.

Without these, Optuna will dutifully optimize noise.

Integration architecture

Conceptual architecture

A minimal Optuna integration for an agent pipeline has three components:

1. Study manager — creates/loads the study, manages storage, runs study.optimize().
2. Objective function — calls your pipeline, collects scores, returns the metric.
3. Evaluation harness — the repeatable runner that isolates the pipeline segment being tuned.

┌─────────────────────────────────────────┐
│           Optuna Study Manager          │
│  study.optimize(objective, n_trials=N)  │
└───────────────────┬─────────────────────┘
                    │ trial
                    ▼
┌─────────────────────────────────────────┐
│         Objective Function              │
│  1. trial.suggest_*() → params          │
│  2. Call evaluation harness             │
│  3. Return scalar score                 │
└───────────────────┬─────────────────────┘
                    │ params
                    ▼
┌─────────────────────────────────────────┐
│         Evaluation Harness              │
│  - Fixed test set (N cases)             │
│  - 3–5 reps per case (stochastic avg)   │
│  - Labeled expected outputs             │
│  - Scalar scorer                        │
└─────────────────────────────────────────┘

Primary integration path: direct Python + shell

Agent runtime (shell tool)
    → runs Python script containing Optuna study
    → objective function calls agent pipeline subprocess or API
    → results written to storage backend
    → best_params exported to config.json
    → agent reads config.json to apply parameters

This path has zero additional dependencies beyond Optuna and is the recommended primary approach for stability.

Optional path: optuna-mcp

If your environment uses MCP, optuna-mcp provides a native interface for agent-driven study management. The agent can call MCP tools to suggest trial parameters, report results, and query the best configuration. This is architecturally clean but carries the stability caveat noted above; maintain the Python fallback.

Portability pattern (two-tier deployment)

Skills must remain executable without Optuna present. The correct framing is two deployment tiers, not "portable and valid":

• Tier 1 (optimized): Optuna installed, access to storage backend, full study reproducible. Performance reflects optimization.
• Tier 2 (portable baseline): config.json contains best_params from a completed study. Executable without Optuna. Performance is fixed at last-optimized values — no further improvement without re-running the study, and is degraded relative to Tier 1.

Version-control config.json alongside skill code, and document which study version produced it.

# config-aware skill entry point
import json
from pathlib import Path

DEFAULT_PARAMS = {
    "temperature": 0.7,
    "max_tokens": 2048,
    "retry_budget": 3,
}

def load_params() -> dict:
    params = DEFAULT_PARAMS.copy()
    config_path = Path(__file__).parent / "config.json"
    if config_path.exists():
        params.update(json.loads(config_path.read_text()))
    return params

Sampler selection for agentic spaces

The agentic parameter space typically mixes categorical choices (model tier, prompt variant) with continuous values (temperature, thresholds). TPE handles this mixed space well. LLAMBO is worth considering when the categorical choices have semantic structure exploitable by a language model. For multi-objective studies, NSGA-II is the documented choice.

An honest cost and measurement guide

The API cost reality

The most important number to internalize before starting an Optuna study on an LLM pipeline is the per-study API cost. For Claude-class models, a realistic study budget is $10 to $600. The exact cost depends on:

• Trial count — 30 trials vs. 300 trials is a 10× cost multiplier.
• Reps per trial — averaging over 5 repetitions multiplies cost by 5.
• Prompt length — long system prompts or large context windows dominate token counts.
• Model tier — the most capable models are substantially more expensive per token than the cheapest.

Important caveat: Published cost figures from LLM fine-tuning HPO examples apply to lightweight open-source models on GPU compute, not to frontier-API calls. Do not use sub-dollar figures for budget planning when the pipeline under study calls a commercial LLM API.

Minimum viable study sizing

A rough planning formula:

Total cost ≈ n_trials × reps_per_trial × cases_per_eval × avg_tokens × price_per_token

Practical cost ranges:

Scale	n_trials	n_tasks	n_reps	Estimated range
Minimal viable	20	10	3	$10–$50
Moderate	50	20	3	$100–$300
Full study	100	50	5	$300–$600+

Always compute a per-trial cost estimate before running a large study.

Cost controls

• Use cheaper-tier models for exploration trials (parameter-space search).
• Validate top-N candidates (Pareto-front members) with a more capable tier.
• Implement MedianPruner with trial.report() at intermediate steps — prunes clearly bad configurations before full evaluation cost is incurred.
• Set n_trials conservatively; add trials incrementally with load_if_exists=True.

Handling stochastic outputs

Language-model outputs are non-deterministic. Without measurement discipline, Optuna optimizes noise:

• Minimum 3–5 repetitions per trial — evaluate each configuration multiple times and return the mean score.
• Low-temperature evaluation passes — use temperature=0 or near-0 for evaluation where possible; reduces variance substantially.
• Judge LLM rubric — for subjective quality metrics, use a separate LLM judge with a structured scoring rubric (e.g. 1–5 scale on named dimensions) to convert qualitative quality to a consistent scalar.
• Variance sanity check — before running any study, run the same configuration 5 times and measure variance. If variance exceeds the expected effect size of parameter changes, the study will optimize noise.

def robust_objective(trial):
    params = {
        "temperature": trial.suggest_float("temperature", 0.0, 1.2),
        "max_tokens": trial.suggest_int("max_tokens", 512, 2048, step=256),
    }

    N_REPS = 5
    scores = []

    for rep in range(N_REPS):
        rep_scores = []
        for case in TEST_CASES:
            result = call_pipeline(params, case["input"])
            score = evaluate(result, case["expected"])
            rep_scores.append(score)

        rep_mean = sum(rep_scores) / len(rep_scores)
        scores.append(rep_mean)

        # Report for pruning
        trial.report(rep_mean, rep)
        if trial.should_prune():
            raise optuna.TrialPruned()

    return sum(scores) / len(scores)

Measurement validity

A score is only meaningful if:

1. The evaluation set is representative of real inputs.
2. The scorer is consistent (same output → approximately same score).
3. The metric is aligned with the actual goal, not a proxy that diverges from real-world quality.

Human-rated rubric scores on a fixed test set are the most honest approach, though expensive. Automated metrics (assertion pass/fail, structured evaluation) are faster but require validation that they correlate with real quality on your specific task.

When to use Optuna

Decision framework

Not every parameter-tuning problem warrants Optuna. The overhead of building a batch harness, labeling a test set, and paying for evaluation trials is only justified when the parameter space is large enough and the task important enough to benefit.

Does the pipeline have ≤4 parameters?
├── YES → Is the budget very small ( float:
        params = {
            "temperature": trial.suggest_float("temperature", 0.0, 1.5),
            "max_tokens": trial.suggest_int("max_tokens", 256, 4096, step=256),
            "model": trial.suggest_categorical("model", ["haiku", "sonnet"]),
            "retry_budget": trial.suggest_int("retry_budget", 1, 4),
        }

        all_scores = []
        for rep in range(n_reps):
            rep_scores = []
            for case in test_cases:
                try:
                    result = run_pipeline(params, case["input"])
                    score = evaluate_output(result, case["expected"])
                    rep_scores.append(score)
                except Exception as e:
                    logger.warning(f"Trial {trial.number}, rep {rep}: {e}")
                    rep_scores.append(0.0)  # Penalize failures

            rep_mean = sum(rep_scores) / len(rep_scores)
            all_scores.append(rep_mean)

            trial.report(rep_mean, rep)
            if trial.should_prune():
                raise optuna.TrialPruned()

        return sum(all_scores) / len(all_scores)

    return objective

Progressive implementation path

1. Baseline first: measure current performance, record variance. No Optuna yet.
2. Simple single-objective study: 2–3 parameters, 20–30 trials, SQLite storage. Validate that Optuna finds something better than baseline.
3. Multi-objective extension: add cost as a second objective, use NSGA-II, select from the Pareto front.
4. Continuous improvement loop: schedule periodic re-optimization as workloads evolve; always compare against the last best_params baseline.

Infrastructure to build first (in order):

1. Evaluation harness (callable, batch-compatible)
2. Test set (labeled, representative, versioned)
3. Metric definition (scalar or multi-objective tuple)
4. SQLite storage setup + config.json export utility

Forward-looking developments

Optuna v5 roadmap

The Optuna team has published a roadmap for v5, targeting a summer 2026 release. Stated goals include a prompt-optimization toolchain that would make the framework more directly relevant to language-model workflows, plus continued improvements to sampler efficiency and the dashboard.

Important caveat: Optuna v5 has not been released as of mid-2026. The summer-2026 timeline and the prompt-optimization toolchain are roadmap items, not shipped features. Do not architect current systems around v5 capabilities until the release is public and the API is stable. Design for v4.8.x.

Emerging academic patterns

Several recent research directions explore the intersection of LLMs and hyperparameter optimization:

• LLMs vs. classical HPO — work examining whether LLMs can serve as alternatives to classical HPO methods has found the two approaches have different strengths; LLM-based one-shot parameter recommendation was competitive with iterative sampling on structured ML benchmarks, suggesting the approaches are complementary rather than competing.
• AutoML-Agent — a multi-agent LLM framework for full AutoML pipelines, representing a broader trend toward agent-driven optimization.
• OptiMindTune — a three-agent system for hyperparameter optimization (recommender, evaluator, decision agent), demonstrating that agentic approaches to HPO are an active research frontier.

LLM-native optimization

The LLAMBO sampler in OptunaHub represents one direction: using an LLM's priors to guide search in spaces where parameter names and values carry semantic meaning. This is particularly relevant for agentic engineering, where parameter names like "temperature", "chain-of-thought", and "confidence_threshold" are semantically meaningful to a language model in a way they are not to classical Bayesian methods.

Conclusion

Optuna 4.8.0 is a production-ready, well-documented, MIT-licensed framework for hyperparameter optimization. Its core abstractions — studies, trials, typed suggest methods, composable samplers, and persistent storage — are clean and learnable for a Python-fluent engineer new to AutoML. The surrounding ecosystem (OptunaHub, the dashboard, optuna-mcp) extends its capabilities without complicating the core API.

For agentic AI engineering, Optuna is a plausible tool for systematically exploring prompt parameter spaces, retry policies, model-routing configurations, and multi-objective quality-cost tradeoffs. These are novel applications without published precedent; the patterns here are hypotheses to test, not established recipes.

The two most important takeaways are practical constraints, not architectural ones:

1. Measurement first. A batch harness, a labeled test set, and an honest scalar metric must exist before a study can produce meaningful results. Building these artifacts is the real work; once they exist, Optuna is straightforward to apply.
2. Budget honestly. A Claude-class study costs $10–$600, not cents. This is not an obstacle to using Optuna, but it is a planning constraint to internalize before launching a large-scale study.

Sources

1. Optuna Documentation. https://optuna.readthedocs.io/en/stable/
2. Optuna GitHub Releases. https://github.com/optuna/optuna/releases
3. Optuna Dashboard. https://github.com/optuna/optuna-dashboard
4. Yoshihiko Ozaki. "OptunaHub — A Feature Sharing Platform for Optuna, Now Available in Official Release." Optuna/Medium, August 2024. https://medium.com/optuna/optunahub-a-feature-sharing-platform-for-optuna-now-available-in-official-release-4b99efe9934d
5. OptunaHub Registry. https://hub.optuna.org/
6. Optuna MCP Server. https://github.com/optuna/optuna-mcp
7. Optuna v5 Roadmap. Optuna/Medium. https://medium.com/optuna/optuna-v5-roadmap-ac7d6935a878
8. von Csefalvay, C. "Hyperparameter Optimization with Claude Code, Optuna, and Hugging Face Jobs." Hugging Face Blog. https://huggingface.co/blog/chrisvoncsefalvay/claude-hf-jobs-optuna
9. LLAMBO on OptunaHub. https://hub.optuna.org/samplers/llambo/
10. Kochnev et al. "Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?" arXiv:2504.06006. https://arxiv.org/abs/2504.06006
11. "AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML." arXiv:2410.02958. https://arxiv.org/abs/2410.02958
12. "OptiMindTune: A Multi-Agent Framework for Intelligent Hyperparameter Optimization." arXiv:2505.19205. https://arxiv.org/abs/2505.19205
13. "Efficient Optimization Algorithms." Optuna Documentation. https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html

Appendix: Quick reference card

Installation

pip install optuna                    # Core (required)
pip install optuna-dashboard          # Dashboard UI (recommended)
pip install optunahub                 # Community samplers (optional)
# uvx optuna-mcp --storage sqlite:///optuna.db  # MCP server (optional, evaluate stability)

Key API patterns

# Create study
study = optuna.create_study(direction="maximize")          # Single-objective
study = optuna.create_study(directions=["maximize", "minimize"])  # Multi-objective

# Suggest parameters
trial.suggest_float("name", low, high)                     # Continuous
trial.suggest_float("name", low, high, log=True)           # Log-scale
trial.suggest_int("name", low, high)                       # Integer
trial.suggest_categorical("name", ["a", "b", "c"])         # Categorical

# Run optimization
study.optimize(objective, n_trials=50, timeout=3600)

# Inspect results
study.best_params       # Best parameter dict
study.best_value        # Best objective value
study.trials            # All trial objects

Sampler selection

Use case	Sampler
Default / general	TPE (default)
Multi-objective	NSGAIISampler()
Async parallel	GPSampler(constant_liar=True)
Semantic spaces	optunahub LLAMBO
Baseline comparison	RandomSampler()

Storage options

# Lightweight local (JournalStorage — v4.x recommended)
from optuna.storages import JournalStorage, JournalFileBackend
storage = JournalStorage(JournalFileBackend("./study.log"))

# SQLite (single-machine, dashboard-compatible)
storage = "sqlite:///study.db"

# PostgreSQL (multi-process)
storage = "postgresql://user:pass@host/dbname"

Cost planning formula

Estimated cost = n_trials × reps_per_trial × cases_per_eval × avg_tokens × price_per_1K_tokens / 1000

Practical range for Claude-class models: $10–$600 per study

Prerequisites checklist

• [ ] Batch harness implemented (repeatable pipeline runner)
• [ ] Labeled test set prepared (fixed inputs + expected outputs)
• [ ] Scalar metric defined and validated
• [ ] Per-trial cost estimate computed
• [ ] Storage backend chosen and tested
• [ ] 3–5 repetitions per trial planned for stochastic averaging