{
  "batch": "deep-35",
  "setup": "Hermes cloned once from NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f; installed from source with uv sync --locked --python 3.14 (Python 3.14.10). plugins --help exposed install/validate/doctor. Used supplied tryit.py with Nous tests/fakes/fake_llm_provider.py, fresh disposable Hermes homes, empty plans followed by 2-5 scripted calls including bad input; no real model/provider credentials. Plugin pins/subdirectories came from field.json. Test-only pytest, PyYAML, boto3, moto and Playwright installed in scratch. Desktop GUI unavailable; limits are recorded per entry. Runtime and test counts below are direct local observations, not entrants' claimed results. Method: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/RUN.md ; pins: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json",
  "plugins": [
    {
      "pr": 150563,
      "name": "hermes-roam",
      "validate": "Passed catalog admission (exit 0, security safe, matching tools/hooks and Desktop surface checks); enable and doctor exit 0; three tools and three hooks registered. Pinned evidence: https://raw.githubusercontent.com/vcruz305/hermes-roam/8b9fba41fea77f2fe2d6d572cffd43039d2da11f/README.md ; https://raw.githubusercontent.com/vcruz305/hermes-roam/8b9fba41fea77f2fe2d6d572cffd43039d2da11f/settings.py ; https://raw.githubusercontent.com/vcruz305/hermes-roam/8b9fba41fea77f2fe2d6d572cffd43039d2da11f/roam/archive.py.",
      "inside": "Loaded three deferred tools, lifecycle hooks, native roam CLI, slash command and setup skill. Empty plan added no eager tools; tool_search discovered all three. Status and dry-run push returned actionable missing-local_dir ConfigErrors on an unconfigured home, an invalid scope was schema-rejected, and model-initiated apply=true was refused by default. In a separate configured real-Hermes CLI flow, a disposable generated archive key and local backend successfully initialized, pushed an encrypted generation, previewed conflicts without writes, then applied a backed-up pull into a second home and reproduced its memory/SOUL. S3/R2 and remote Git were not tested live.",
      "tests": "610 passed, 2 failed, 1 skipped after installing declared/test dependencies: a 20,000-path inventory took 3.41s versus a 2s assertion under concurrent load, and human status output exceeded the promised 80 columns. Initial collection errors were missing PyYAML/moto, resolved. Separate two-home CLI sync passed.",
      "useful": 5,
      "reliable": 5,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "rank": 1,
      "best": "Encrypted, conflict-aware sync of the user's real Hermes skills, memories, persona and profile settings makes the same personal agent usable across hardware, leading this close top group for likely everyday use.",
      "worst": "The human status-width test fails and a large-home timing target missed here; real S3/R2/Git behavior remains supported by read code and tests rather than this seat's live remote run.",
      "total": 28,
      "blocked": null
    },
    {
      "pr": 149159,
      "name": "understudy",
      "validate": "Passed catalog admission (exit 0, security safe, registrations and Desktop SDK checks passed); enable and doctor exit 0; two tools registered. Noninteractive enable correctly withheld model/provider override capabilities. Pinned evidence: https://raw.githubusercontent.com/stffinfcti/understudy/e0920f9e3641f0e24afe6ee01b21322c070920fd/README.md ; https://raw.githubusercontent.com/stffinfcti/understudy/e0920f9e3641f0e24afe6ee01b21322c070920fd/engine/audition.py ; https://raw.githubusercontent.com/stffinfcti/understudy/e0920f9e3641f0e24afe6ee01b21322c070920fd/surfaces/tools.py.",
      "inside": "Loaded, with two deferred tools plus CLI, slash command, skill and Desktop report surface. Empty plan exposed no additional eager tools. Planned dry-run returned a zero-history estimate without spending; an empty candidates list was rejected by Hermes's schema validator, and a missing/path-like report id returned a clear no-audition error. Actual paid candidate/judge auditions were not performed; orchestrator behavior was exercised by the plugin's fake-LLM tests.",
      "tests": "Passed: 94 pytest tests, covering corpus sampling, blind judge, budgets, errors, pricing, persistence and report surfaces.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "rank": 6,
      "best": "Auditioning cheaper models against the user's own session history, with a self-baseline and confidence intervals, makes model choice more useful than a generic leaderboard.",
      "worst": "The advertised hard dollar cap uses past mean costs rather than reserving a worst-case charge, so expensive or unpriced in-flight calls can overshoot it, and single-step agreement remains a proxy for completing tasks.",
      "total": 24,
      "blocked": null
    },
    {
      "pr": 153714,
      "name": "tenure",
      "validate": "Passed catalog admission (exit 0, security safe, matching hooks and Desktop surface checks); enable and doctor exit 0; seven hooks registered, no tools. Pinned evidence: https://raw.githubusercontent.com/otsoa-dubief/tenure/f4a62855e592b9e170ab0c54d9278aabef05f370/README.md ; https://raw.githubusercontent.com/otsoa-dubief/tenure/f4a62855e592b9e170ab0c54d9278aabef05f370/tenure/__init__.py ; https://raw.githubusercontent.com/otsoa-dubief/tenure/f4a62855e592b9e170ab0c54d9278aabef05f370/tenure/dashboard/plugin_api.py.",
      "inside": "Loaded approval/observation hooks, native CLI, slash command, dashboard and Desktop page; no eager model tools expected or added. Three scripted Hermes calls wrote valid JSON, read it, and returned a normal not_found for a missing file. Native tenure status and undo list recorded the write; undo 1 restored the pre-write state (removed the newly created file), counted a fault and put that action type on T1 probation. Setup guidance and passport export worked; an unknown undo id failed cleanly. Desktop UI and a real human approval interaction were not exercised. Exported a fresh signed passport and five-event ledger: hermes tenure verify returned signature valid, chain intact and one type recomputed by ledger replay.",
      "tests": "2402 pytest tests passed, 5 failed, 2 skipped. Failures: two read-only dashboard assertions observed initialization writes, two no-set_config tests expected 501 but received 200 with the installed Hermes settings-writer fallback, and a large-ledger passport timing assertion exceeded 300ms. Real-Hermes file undo/probation flow passed. Passport/ledger verification also passed.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 3,
      "best": "Revocable autonomy backed by real file undo, per-action probation and an evidence ledger improves ordinary Hermes work while keeping outbound actions under human approval.",
      "worst": "A broad shell classifier and trust state machine create considerable safety complexity, and its tests expose read-side initialization writes plus settings and timing mismatches in this Hermes installation.",
      "total": 29,
      "blocked": null
    },
    {
      "pr": 150596,
      "name": "ariadne",
      "validate": "Passed catalog admission (exit 0, security safe, declarations and Desktop surface checks passed); enable and doctor exit 0; 19 hook registrations and three middleware points, no tools. Pinned evidence: https://raw.githubusercontent.com/miracle352/hermes-ariadne/128c95967a0761ad24d3d224441d1502c43bbb69/README.md ; https://raw.githubusercontent.com/miracle352/hermes-ariadne/128c95967a0761ad24d3d224441d1502c43bbb69/runtime.py ; https://raw.githubusercontent.com/miracle352/hermes-ariadne/128c95967a0761ad24d3d224441d1502c43bbb69/tests/e2e/roundtrip.py.",
      "inside": "Loaded. Empty plan correctly added no tools; it registers record/replay middleware, hooks, native CLI, slash command and Desktop page. Three planned calls wrote a note, read it, and returned a clean not_found for a missing note. The separate native two-turn end-to-end test recorded four model calls, three tools and one auxiliary call, then replayed IDENTICAL with zero drift, zero leaks and zero hook-contract violations; the output file stayed deleted, proving replay did not re-execute its writing tool. A bogus cassette id gave a clear CLI error. Desktop UI and cross-version upgrade/bisect were not run.",
      "tests": "Passed: 21 unit tests and the real-Hermes offline roundtrip end-to-end script (2 turns, 4 matching model calls, 3 matching tools).",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 2,
      "best": "Safe offline replay, request diffs and hook-contract checks let users test Hermes changes against their own sessions rather than trust an upgrade blindly.",
      "worst": "The heaviest upgrade/bisect paths still require separate Hermes builds, and recordings replay through a specific surface rather than reproducing streaming or every dispatcher hook.",
      "total": 29,
      "blocked": null
    },
    {
      "pr": 149426,
      "name": "pinpoint",
      "validate": "Passed catalog admission (exit 0, security safe, matching declarations and Desktop SDK checks); enable and doctor exit 0; two tools registered. Pinned evidence: https://raw.githubusercontent.com/stupidburgers/pinpoint/06b911e1c66908ebf340c6cb981028bc45b89c73/README.md ; https://raw.githubusercontent.com/stupidburgers/pinpoint/06b911e1c66908ebf340c6cb981028bc45b89c73/_tools.py ; https://raw.githubusercontent.com/stupidburgers/pinpoint/06b911e1c66908ebf340c6cb981028bc45b89c73/engine/pinpoint-engine.js.",
      "inside": "Loaded two deferred tools, a system-prompt discovery hint, local pin store, and Desktop browser/pane integration. Empty plan exposed no extra eager tools. Correct tool_search found both; tool_call pinpoint_pins(list) returned an honest empty store, max_findings=0 was rejected by Hermes schema validation, and a normal audit returned desktop_unavailable with a setup hint after its 4-second acknowledgement budget. No real Desktop window was available, so a completed live-page audit/design-mode loop could not be independently exercised.",
      "tests": "306 Python/build tests passed, 5 skipped, 9 subtests passed. Full suite also produced 175 engine setup errors because installed Chromium could not launch: missing libatk-1.0.so.0 (environment limitation, not evidence of 175 plugin defects). Desktop E2E skipped/unavailable; no live UI verification claimed.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 5,
      "best": "Measured pins and conservative fixed/gone/not-rechecked deltas close a useful visual repair loop inside Hermes Desktop rather than asking a model to guess from CSS.",
      "worst": "It serves Desktop web-building users only, and its own README discloses a failing desktop-only scenario plus missing session/window correlation; neither the full Desktop flow nor browser engine ran here.",
      "total": 26,
      "blocked": null
    },
    {
      "pr": 149244,
      "name": "lineage",
      "validate": "Passed catalog admission (exit 0, security safe, declarations and Desktop SDK checks passed); enable and doctor exit 0; doctor registered 10 observer hooks and no tools. Pinned evidence: https://raw.githubusercontent.com/P0GL0L/hermes-lineage/61de70980f4322838a9b53f4802623146f1046c2/README.md ; https://raw.githubusercontent.com/P0GL0L/hermes-lineage/61de70980f4322838a9b53f4802623146f1046c2/lineage_core/tracker.py ; https://raw.githubusercontent.com/P0GL0L/hermes-lineage/61de70980f4322838a9b53f4802623146f1046c2/lineage_core/receipt.py.",
      "inside": "Loaded. Empty plan added no model tools as intended. Four scripted Hermes calls wrote source.csv, read it, wrote note.md citing it, and returned a normal not_found error for a missing CSV. The plugin recorded two artifacts and an evidenced source edge. Native hermes lineage show, receipt --write, check and verify worked: signature/hash checks passed, verdict fresh became stale after editing the source; unknown artifact returned a clear exit-1 message. Desktop graph was read, not exercised.",
      "tests": "Passed: 19 pytest tests after installing test-only PyYAML (initial attempt: 18 passed and one missing-PyYAML failure). Additional real-Hermes receipt and staleness flow passed.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 4,
      "best": "Automatic provenance receipts distinguish observed, evidenced and inferred relationships, making the agent's actual work inspectable without another model call.",
      "worst": "URL refetch validates only the initial URL before urllib follows redirects, leaving a gap in the advertised public-host-only protection; provenance also cannot prove semantic dependence.",
      "total": 27,
      "blocked": null
    }
  ]
}