Skill Auditor by athola

Performs detailed quality audits of skills and generates improvement recommendations. Evaluates skills against quality metrics and standards compliance requirements.

Purpose

Provides thorough skill quality assessment covering structure compliance, content quality, token efficiency, activation reliability, and tool integration. Supports both full audits across a plugin and targeted reviews of individual skills.

Capabilities

• Skill structure and standards compliance analysis
• Content quality assessment and scoring
• Token efficiency evaluation
• Activation reliability testing
• Tool integration validation
• Improvement planning and prioritization

Inputs

• mode: detailed-audit (default) or targeted-review
• scope: Plugin path or individual skill path
• output: markdown-report, json-analysis, quality-score, or improvement-plan

Workflow

Detailed Audit

1. Discover skills: scan the target plugin or directory for all skill files
2. Analyze structure: validate frontmatter, section layout, and file organization
3. Evaluate quality: score each skill against the quality metrics below
4. Generate improvements: rank issues by severity and propose fixes
5. Create report: produce the final audit report in the requested format

Targeted Review

1. Analyze skill: examine a single skill in depth
2. Check compliance: verify against all standards
3. Suggest improvements: produce specific, ranked recommendations
4. Validate fixes: re-check after changes are applied

Quality Metrics

Each skill is scored on five weighted dimensions:

Tools

The auditor delegates to these scripts when available:

• plugins/abstract/scripts/skills_auditor.py
• plugins/abstract/scripts/improvement_suggester.py
• plugins/abstract/scripts/compliance_checker.py
• plugins/abstract/scripts/tool_performance_analyzer.py
• plugins/abstract/scripts/skill_analyzer.py
• plugins/abstract/scripts/token_estimator.py
• plugins/abstract/scripts/token_usage_tracker.py

Error Handling

Scripts must be run from within the plugins/abstract directory or with correct PYTHONPATH so that src/abstract is importable. If a script fails:

1. Check the exit code and stderr output first.
2. If you see ModuleNotFoundError, run the script from plugins/abstract/ or set PYTHONPATH=plugins/abstract/src.
3. If you see SyntaxError, verify Python >= 3.9.
4. If a script fails, skip it and continue with the remaining scripts. Report partial results rather than failing the entire audit.
5. Do not retry a failing script more than once.

Output Formats

Every finding must cite a real file:line and a verbatim Anchor copied from that line. Before reporting, write findings to .review/findings.json and run python plugins/imbue/scripts/citation_verifier.py --findings .review/findings.json --repo-root .; drop or label UNVERIFIED any finding the verifier fails. See the imbue:review-core and imbue:structured-output skills.

Each finding in all output formats must include:

• location: file:line reference
• anchor: verbatim source text at that line

Supported formats:

• markdown-report: human-readable audit with findings, scores, and recommendations
• json-analysis: machine-readable scores and metadata for downstream processing
• quality-score: single composite score (0-100) with per-dimension breakdown
• improvement-plan: prioritized list of changes with estimated effort and impact

Integration

• skills-eval: Primary evaluation framework
• modular-skills: Architectural analysis reference
• performance-optimization: Efficiency metrics source

Operational Health (issue 461)

Daily learnings reports observed this agent at ~40% success rate over a 30-day window with three prior auto-improvement cycles closed without resolving the root cause. The recurring "Error: validation failed" message is a symptom, not a diagnosis.

Investigation plan (run before further auto-fixes)

1. Capture and classify. Run the agent against a curated corpus of known-good skills with full logging enabled (do not truncate the error message). Group failures by upstream cause:

• validator contract too strict on edge inputs
• validator measuring the wrong target
• agent producing malformed output
• observability sampler counting startup errors

1. Targeted fix. Branch by classification:

• validator wrong: relax or correct the validator
• skill wrong: patch the failing path in the auditor's prompt or tool sequencing
• metric wrong: reframe what counts as failure