Auto-Generate Agent Skills with Microsoft's SkillOpt

Every skill in this catalog so far has been hand-written by a human. microsoft/SkillOpt takes a different approach: a tool that watches the agent work, learns from the trajectories, and emits a best_skill.md artifact the agent then loads as a system prompt. The model itself stays frozen. The skill is what improves.

Why This Skill Matters

Hand-crafted skills are great until your agent does the same task ten times and you realize a better pattern would have emerged. SkillOpt codifies the discovery loop: rollout, reflect, aggregate, select, update, evaluate. You point it at a benchmark or a task, and it produces a deployable artifact.

Three things make this useful:

• No fine-tuning. The model is frozen; the skill is the optimization target. Deployment is zero added inference calls.
• Cross-agent portability. A best_skill.md from one backend (say, GPT-5.5 direct chat) loads into Codex CLI, Claude Code CLI, Cursor, and Copilot — the README reports gains across all of them.
• Validation-gated updates. Skill edits go through a held-out validation set. Edits that don't improve held-out accuracy are rejected, so the artifact doesn't drift off-distribution.

If you have an agent doing a task badly and you don't want to touch the model weights, this is the tool.

Installation

pip install skillopt

For the WebUI dashboard:

python -m skillopt_webui.app --port 8080 --host 127.0.0.1

Use --host 127.0.0.1, not the default 0.0.0.0, unless you intentionally want to expose the dashboard to your network.

Real Workflow: Improve Coding Agent Accuracy on SearchQA

A realistic scenario, not "hello world": your coding agent fails a question-answering benchmark 60% of the time, and you want to lift that without fine-tuning.

The workflow:

1. Define a scoring rollout on the SearchQA reference benchmark.

2. Run SkillOpt's training loop against frozen GPT-5.5:

   skillopt train --benchmark searchqa --backend openai --validate-on held-out
   

3. SkillOpt runs rollout → reflect → aggregate → select → update → evaluate. Each update is gated by held-out accuracy.

4. After convergence, SkillOpt writes best_skill.md (300–2,000 tokens).

5. Load best_skill.md as the agent's system prompt for the task.

Expected output: a best_skill.md artifact and a metric file showing the before/after. The README reports +23.5 accuracy points (direct chat), +24.8 (Codex CLI), +19.1 (Claude Code CLI) versus the no-skill baseline on SearchQA. Numbers vary by task; expect a meaningful lift, not a miracle.

You can also drop the artifact into Codex CLI or Claude Code CLI directly:

codex --system-prompt-file best_skill.md

The same file works because it's just markdown.

Real Workflow: Run SkillOpt-Sleep for Nightly Self-Improvement

SkillOpt-Sleep (v0.2.0+) is the offline self-evolution engine: it harvests your agent's logs from the day, mines for patterns, replays them on held-out sets, and consolidates the best edits into the next-day best_skill.md.

The workflow:

1. Let the agent run normally; ensure its logs are captured somewhere SkillOpt can read.

2. Schedule the sleep job:

   skillopt-sleep --logs ./agent-logs/2026-08-17 --output best_skill.md
   

3. The next morning, best_skill.md reflects the day's successes and rejections.

4. Load it as the system prompt; the agent starts the day a little better than yesterday.

Expected output: a refreshed best_skill.md and a brief report of what changed (edits accepted, edits rejected, held-out delta). Run it daily for compounding improvements; ignore it for a week and the artifact goes stale.

Tips

• Start with a benchmark, not a freeform task. The validation gate needs scored rollouts; freeform chat without a scoring signal won't drive improvements.
• Use the WebUI for debugging. The dashboard makes it obvious when updates stall or regress on the held-out set.
• Pin optimizer-model cost. SkillOpt needs an LLM to propose edits. The deployment is zero-cost, but the training step is not — budget for it.
• Audit best_skill.md before loading. SkillOpt optimizes held-out accuracy, not human readability. A 2,000-token skill may be opaque; review it before relying on it.

When Not to Use This

• You can't define a held-out validation set. Without a scoring signal, the optimizer has nothing to gate edits against. Find a measurable proxy first.
• You want a human-readable skill. SkillOpt's output optimizes for accuracy, not for editorial clarity. If your team needs to understand and modify the skill, start from a hand-written baseline.
• You need fine-grained control. The default training loop is conservative. Experimental multi-objective, replay, and dream-rollout knobs exist but aren't main-CLI flags — you'd be reaching into internals.
• The plugin/MCP integration matters. The PyPI wheel doesn't bundle the Claude Code / Codex / Copilot shells. Pull them from the GitHub repo, not pip.

See the leaderboard for more skills.