How optimization works

Skynet is Optimize Anything: one product that improves any artifact against your data and your metric. Every run uses the same run contract and the same metric contract. The artifact kind decides which engines can work on it.

The shared contract

Every run has two files. run.py runs the artifact on one case. metric.py scores the output.

from skynet import llm


def run(artifact, input):
    return llm(artifact, input=input)

def score(input, output, expected=None):
    ok = output == expected
    return (1.0 if ok else 0.0), {"feedback": "Match." if ok else "No match."}

• artifact is the current version that the optimizer proposes.
• input is a dict that holds the input fields of one case.
• expected is the expected value of the case, or None when the case has none.
• score returns a pair: a number, and a dict with a "feedback" string.

The skynet helper module gives llm, Image and log. A call to llm from the run code goes to the task model. A call to llm from the metric goes to the metric model.

Read Metrics and scorers for the full metric contract.

Artifact kinds and engines

The engines come in two families:

• The native engines treat the artifact as a black box. They work on Prompt, Code and Text artifacts.
• The DSPy optimizers work inside a DSPy program. They work only on DSPy program artifacts.

DSPy MIPROv2 and DSPy BootstrapFewShot need pairs (cases with expected outputs). With a single input, a DSPy program can use only Auto and DSPy GEPA.

Prompt, Code and Text artifacts

The native engines treat the artifact as a black box. An engine proposes a new version, runs run(artifact, input) on the cases, and reads the score and feedback.

The six native engines use different search methods:

• GEPA evolves versions by reflection and keeps a Pareto front.
• Best-of-N asks the reflection model for independent proposals and keeps the best one.
• AutoResearch, Meta-Harness, AutoSaddler and ShinkaEvolve use a coding agent to propose versions.

Read Engines for each engine and its settings.

DSPy program artifacts

A DSPy program is one artifact kind. Your run.py builds a dspy.Module at module level and calls it inside run. If the file has more than one module, name the one to optimize program.

What happens when a DSPy optimizer runs:

1. Skynet finds your module and wraps your run function around it. Every case goes through your own run.
2. The DSPy optimizer compiles the program on the train cases.
3. It selects on the validation cases.
4. The winner is the instruction text of each predictor, plus the full compiled state of the module. The state includes the few-shot demos.

The final scores measure the same program that Skynet saves.

DSPy BootstrapFewShot does not use a reflection model.

Auto

Auto is the default choice in the wizard.

• On a Prompt, Code or Text artifact, Auto explores with five native engines, then continues with GEPA from the best version.
• On a DSPy program, Auto runs DSPy GEPA.

Both forms of Auto read the feedback text from your metric.

Which engine to pick

Related pages

• Engines
• Datasets and splits
• Metrics and scorers
• Submit a run