{
  "batch": "deep-39",
  "setup": "Hermes source cloned once from https://github.com/NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f, installed with uv sync --locked --python 3.14 (Python 3.14.10); pytest/pytest-asyncio/PyYAML added to the scratch venv. Used supplied RUN.md and tryit.py with Nous tests/fakes/fake_llm_provider.py against real AIAgent.run_conversation. Initial localhost proxy error was corrected with NO_PROXY; no plugin penalty. All homes, fixture data and downloads isolated under test/scratch. Runner adaptation retained full validator stdout, disabled progressive tool disclosure for planned direct calls, selected context.engine=cmi for CMI, and seeded PCC with a synthetic local Demo project. No entrant code was changed. Runtime observations and test counts below are this seat's observations, not independent certification; pinned source URLs are in sources.json. Desktop/browser UI and physical printer/GPU were not exercised.",
  "plugins": [
    {
      "pr": 153024,
      "name": "home-3d-workshop",
      "validate": "Passed catalog validate, enable and doctor (all exit 0), with no warnings; 14 tools and four hooks. Pinned 8068c2533ac03e9067448bf5c6433419f0ab316c.",
      "inside": "Loaded the Workshop tools, approval/access hooks and CLI. Five scripted Hermes calls ran: simulator status, model-card lookup, import and slicing succeeded; negative queue id in start was blocked cleanly. The requested slice id 1 referred to a simulator seed model, not the newly imported id 3, so it demonstrates slicing but not that imported part end-to-end. Separately, hermes home3d demo completed the simulator request/import/slice/queue/owner-approval/watch/eject/failure/reprint loop, explicitly staging approval and delivery without messages or real hardware. Safety code binds single-use passes to action/target/session and refuses voice/unattended starts. No actual Bambuddy, OpenSCAD binary or printer was tested.",
      "tests": "Python: 970 passed, 1 failed across the main run (955 passed/1 failed) and checks subset (15 passed); 1 slow-check test not completed after a first full-suite attempt triggered a blocked private Bambuddy test URL. Failure: test_plugin_keys_and_state expects UNKNOWN for plugins.enabled=[x], but pinned watcher returns DISABLED. Tests collected: 972 total, including 16 checks. Node Desktop: 31 passed. Simulator CLI demo also exited 0.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "A printer becomes a shared household agent with code-enforced owner gates, a zero-model watcher and persistent delivery tracking.",
      "worst": "It serves a hardware-specific audience, and the watcher-state test fails while physical printer behavior and one slow connectivity test remain unverified.",
      "blocked": null
    },
    {
      "pr": 151281,
      "name": "tironian",
      "validate": "Passed catalog validation (exit 0), enable and doctor; 11 hooks and llm_execution middleware. Loader warns that provides_middleware is an unknown manifest field, but registration succeeds. Pinned 9b590cc094f4b65cd70e979f632e1783424526a8, plugin/ subdirectory.",
      "inside": "Loaded; no agent tool is claimed, instead captures/grading hooks, routing middleware, /tironian, CLI and Desktop. Four scripted terminal calls through real Hermes: status and report succeeded, dataset no-such-family raised an uncaught traceback (exit 1), demo completed a dry-run training/evaluation/routing/fallback loop with scripted history and students. No GPU training, Modal upload, real learned model quality or Desktop verification.",
      "tests": "104 passed (Python pytest, 24.88 s).",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 25,
      "rank": 6,
      "best": "Capturing actual Hermes work, testing personal students on held-out sessions and routing through validated fallback materially changes the agent's model execution.",
      "worst": "An unknown family in the documented dataset CLI produces an uncaught exception, so Reliability is capped at 2; actual GPU training remains unverified.",
      "blocked": null
    },
    {
      "pr": 151887,
      "name": "cmi",
      "validate": "Passed catalog validate, enable and doctor (all exit 0). Rule 9 warning: undocumented core imports including model metadata, compression and pricing. Pinned 399468f0687c051476961e4bb045eb84263be6b1, cmi/ subdirectory.",
      "inside": "Loaded as the documented context.engine=cmi, adding cmi_recall, cmi_search, project-memory tools and /cmi. Initial three calls cleanly handled empty history, missing key and negative search limit. A further real Hermes turn generated a synthetic 30-line build log through terminal; after that turn finished, a new conversation searched the persisted index and recalled that log successfully, while a missing key still returned a clean error. Recall matched the stored original tool-result bytes exactly. Project topics without a bound project returned a clear not-enabled error. These short conversations do not measure the advertised percentage savings.",
      "tests": "498 passed, 9 skipped in 103.00 s with the Hermes source supplied via HERMES_AGENT_DIR; tests ran from the entrant repository rather than with Hermes tests on PYTHONPATH.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "A reversible, cache-aware context engine with searchable past tool outputs improves ordinary Hermes work without another model call.",
      "worst": "Undocumented compressor and pricing imports are a compatibility risk, and this seat did not reproduce the README's percentage savings on real user workloads.",
      "blocked": null
    },
    {
      "pr": 152190,
      "name": "hermes-tuneup",
      "validate": "Passed catalog validate, enable and doctor (all exit 0); registers tuneup_check and three hooks. Rule 9 warning lists undocumented Hermes resolver/compressor imports. Pinned e844617bb451fa6d19cd6162a25551599f60e5a9.",
      "inside": "Loaded. Three real Hermes calls worked: active report resolved the synthetic 128K custom model's compaction trigger to 96K with the small-window floor explained; all-profile digest worked; nonexistent named profile returned a clean read-only error. Agent-facing tool cannot apply fixes. Code review found user confirmation, redacted diff, pre-write signature checking and backups in the separate fix flow; no real user configuration was read or changed. Desktop and an interactive confirmed fix were not exercised by this seat.",
      "tests": "Initial full suite: 314 passed, 6 failed, caused by proxy interception in host-allowlist assertions. Reran all 49 adapter-contract tests with HTTP(S)/ALL proxies cleared: 49 passed, including those six. Other 271 tests passed in the full run; 320 distinct tests passed across these runs.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 4,
      "total": 25,
      "rank": 5,
      "best": "Makes Hermes's actual compaction rules understandable and keeps fixes behind explicit human confirmation with a preview and backup.",
      "worst": "It is an excellent advisor rather than a new agent capability, and dependence on many undocumented resolver functions makes upgrades fragile.",
      "blocked": null
    },
    {
      "pr": 152899,
      "name": "project-command-center",
      "validate": "Passed catalog validate, enable and doctor (all exit 0), with no validation warnings; three tools and two hooks. Pinned eaa3e2d3a046eb677f4c67aae98b78f1dcec4043.",
      "inside": "Loaded pcc_board, pcc_set_status, pcc_work_item and context/PR hooks. Seeded a synthetic Demo project through its board-store API to stand in for Desktop project synchronization. All five planned calls ran: read board, set active lane, attach issue 7, reread persisted lane/link, reject negative issue as bad_issue. No GitHub writes, live account use, full Desktop rendering or remote project fetch was attempted. Code uses profile-scoped get_secret for GITHUB_TOKEN and bounded fixed-argv Git operations.",
      "tests": "All 8 standalone Python suites passed (448 check assertions across final runs). Three initially failed because their supposed non-repo fixtures inherited the outer workspace Git repository; reruns with an invalid .git boundary in the scratch HOME prevented parent discovery and passed without entrant edits. JavaScript: 50 passed, 6 excluded using --test-skip-pattern for graph tests requiring an absent full-history Hermes fixture; the unfiltered run had those 6 fixture failures. pytest alone collected no tests because Python suites are standalone scripts.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 27,
      "rank": 2,
      "best": "A shared SQLite board, session-aware agent tools, automatic PR links and a substantial Desktop project view remove repeated project coordination work.",
      "worst": "Its strongest value is for Desktop users already organizing GitHub projects, and six real-history graph tests require fixtures absent from a normal clone.",
      "blocked": null
    },
    {
      "pr": 153900,
      "name": "annotary",
      "validate": "Automated catalog validate, enable and doctor passed (all exit 0); rule 9 warning for gateway.webapp_listener.webapp_settings. Manual review raises rule 11 concern: accounts.token_for reads per-profile token files and server._secrets reads owner credentials from its own secrets.json rather than Hermes secret routes. These are generated local-service credentials, not stolen provider keys; mode-0600 storage and disclosure mitigate exposure but do not satisfy the stated route-only rule. Pinned 253abf811b4be90f57834f8233730fbc688e7ced.",
      "inside": "Loaded ten native tools plus document review integration. Five real Hermes calls succeeded or failed cleanly as appropriate: share Markdown to its actual pinned local server, read v1, push v2 before any reviewer comments, list documents, reject giving both path and markdown. First-use setup correctly reported the missing browser page/wake permission/public URL. The binary download is version/checksum-pinned. Human review UI, phone access, cross-profile handoff and gateway wake behavior were not exercised interactively.",
      "tests": "331 passed, 3 skipped in 163.98 s (Python, using its real pinned server fixtures); 29 passed (Node Desktop tests).",
      "useful": 5,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "Persistent line comments, versions and consent-controlled agent wake-ups make reviewing agent documents a real cross-surface workflow.",
      "worst": "Direct file-based owner/profile credential reads conflict with catalog rule 11 even though automated validation passes, so I rate Reliability 2 pending a Hermes-managed secret route.",
      "blocked": null
    }
  ]
}