{
 "batch": "deep-37",
 "setup": "Hermes installed from source as RUN.md says: git clone --depth 1 of github.com/NousResearch/hermes-agent (main, commit fdde27e7, 2026-11-11), uv 0.13.7 installed into test/scratch/bin, uv sync --locked --python 3.14 (CPython 3.14.10, interpreter placed under test/scratch because the sandbox cannot read ~/.local). hermes plugins --help lists install, validate and doctor. Each plugin fetched at its pinned commit and run with tryit.py against Nous's fake_llm_provider: validate --install-deps --json, enable, doctor --ci, then an in-process AIAgent conversation. Plugin tools are deferred behind Hermes's tool_search bridge, so added_by_plugin is empty for every plugin; planned calls were routed through tool_call and all ran. Plugin test suites were run with the venv's pytest/unittest, Hermes off PYTHONPATH. No Claude Code or Codex CLI, no model keys and no other model providers exist on this machine, so paths that need them were judged by their failure messages and by reading.",
 "plugins": [
  {
   "pr": 153050,
   "name": "mandate",
   "validate": "passed (ok: true, 16 checks). Two warnings: security scan 'caution' for curl_pipe_shell in PROOF.md, live-run.md and scripts/integration_scenario.py plus ssh_dir_access in that script (docs and bench, not the plugin path); and catalog rule 9 private core imports in bench/replay.py, demo/run_demo.py, scripts/integration_scenario.py and mandate_core/cli.py (cron.jobs.get_job). Isolation verdict: host.",
   "inside": "loaded: enable exit 0, doctor OK with 1 tool and 2 hooks (pre_tool_call, post_tool_call), plus CLI 'hermes mandate' (watch, propose, approve, undo, drift, status, export, enforce, observe, widen, list, doctor) and /mandate. Calls: terminal echo ran normally under the hooks (observe mode); mandate_status {} and {job: 'nightly-triage'} both returned {mandate: null, 'No Mandate job is bound to this run. Nothing is enforced.'}; mandate_status {job: 12345} was rejected by schema validation (not invoked). CLI: 'hermes mandate list' and 'hermes mandate propose nope' gave clear errors naming the next command. The learn-then-enforce path needs a real cron job and was not driven here; its bench and integration tests cover it. desktop/plugin.js (331 lines) passes node --check.",
   "tests": "pytest: unit suite 156 passed, 1 skipped, 0 failed; the integration test tests/integration/test_real_hermes.py fails only because it asserts 'v0.23.0' in the Hermes version string and this checkout is main (fdde27e)",
   "useful": 4,
   "reliable": 4,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "rank": 2,
   "best": "A per-job, argument-level least-privilege policy learned from a cron job's own runs and vetoed in pre_tool_call before the tool runs, with observe/propose/approve/enforce, a drift inbox, a Desktop page and a replayable keyless benchmark.",
   "worst": "Its value shows only for users running unattended cron jobs, the status tool does nothing until a job is bound, and validate warns on private core imports (cron.jobs.get_job in cli.py) and curl-pipe-shell lines in its docs and scripts.",
   "blocked": null,
   "total": 25
  },
  {
   "pr": 150497,
   "name": "thoth",
   "validate": "passed (ok: true), no warnings; security scan safe; isolation verdict: host.",
   "inside": "loaded: enable exit 0, doctor OK with 19 tools and 3 hooks (pre_tool_call, post_tool_call, pre_llm_call). Calls in a fresh git repo: thoth_init created .thoth/BRAIN.md and brain.db; thoth_plan created T1 and warned 'T1 is for Claude Code, which is not installed on this machine'; thoth_assign T1 to claude failed cleanly: \"Claude Code isn't installed on this machine (no 'claude' on PATH). Install it and sign in, set claude_path, or give this task to another agent.\"; thoth_wait returned 'No open runs'; thoth_context returned a full brief; thoth_status and thoth_models worked; thoth_assign T99 returned 'Task T99 does not exist'. Caution: when thoth_init was given a sub-folder inside an existing git repository it chose the enclosing repository root as the project and wrote .thoth/ there, not in the named folder. desktop/plugin.js (3121 lines) passes node --check; a bundled MCP server (mcp_server.py) lets Claude Code and Codex write to the board.",
   "tests": "pytest: 75 passed, 2 failed, 6 skipped (test_core.py::test_gated_tools_need_the_hook and test_thoth_never_makes_a_project_of_a_folder_that_holds_much_more; the second expects wording the error no longer has)",
   "useful": 4,
   "reliable": 3,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "rank": 4,
   "best": "Turns Hermes into the coordinator of a real multi-agent team: the actual Claude Code and Codex CLIs work in per-task git worktrees from a shared brain, cross-review each other, and Hermes merges, with a large Desktop board.",
   "worst": "Two of its own tests fail at the pinned commit, the main path needs Claude Code or Codex installed and paid for, and thoth_init silently adopted the enclosing git repository root instead of the folder it was given.",
   "blocked": null,
   "total": 24
  },
  {
   "pr": 153606,
   "name": "habits",
   "validate": "passed (ok: true), 1 Python dependency (pyyaml) installed. One warning: catalog rule 9 private core imports at 7 sites in habits/plugin.py and habits/tools.py (tools.approval._presence, _unattended_contexts, detect_dangerous_command, hermes_cli.plugins.get_pre_tool_call_directive, tools.skill_usage.bump_use, agent.file_safety.get_read_block_error). Security scan safe; isolation: host.",
   "inside": "loaded: enable exit 0, doctor OK with 2 tools and 7 hooks, plus /habits, 'hermes habits' and a system-prompt section. Calls: habit_run with an inline reflex (terminal df, check, end) ran dry and for real, returning the disk output and a step report with no model turn; habit_run {file: '/etc/passwd'} refused ('is not a reflex: give a .yaml, .yml or .json file'); habit_run {} refused with a clear message; habit_save without reflex refused. 'hermes habits list' listed 7 built-in reflexes and flagged tidy-downloads as one that writes files. No Desktop UI; HTML gallery/replay pages are generated instead. The optional Jev key is read via agent.secret_scope.",
   "tests": "pytest: 4902 passed, 9 failed, 27 skipped (all 9 in tests/test_switches.py: a flagged command 'chmod 600 ~/x' is expected to be refused or listed and is not)",
   "useful": 4,
   "reliable": 3,
   "polish": 4,
   "creative": 4,
   "beyond": 4,
   "rank": 6,
   "best": "Reflexes run as written with no model turn between steps, can fire before the model in pre_llm_call or from cron, and a nightly 'dream' mines history for repeated routines; the refusal messages are the clearest in the batch.",
   "worst": "Nine of its own tests fail at the pinned commit around the permissions/flagged-command switch, it leans on seven private Hermes imports, and 20,000 lines with learning modes and a paid third-party decision model is a lot for a user to trust on every turn.",
   "blocked": null,
   "total": 23
  },
  {
   "pr": 151245,
   "name": "freeze",
   "validate": "passed (ok: true, 15 checks), no warnings, no Python dependencies, security scan safe; isolation: host.",
   "inside": "loaded: enable exit 0, doctor OK with 0 tools and 4 hooks (pre_llm_call, post_tool_call, post_api_request, post_llm_call), plus /freeze and 'hermes freeze'. A chat that ran terminal ls and read_file was recorded: plugin-data/freeze/data.db holds 1 turn and 2 calls. CLI: 'hermes freeze list' said 'No frozen workflows yet' with the next step; 'hermes freeze run no-such-workflow' and 'hermes freeze show ../bad name' returned clear errors (name rule enforced). The /freeze compile step calls the user's model through ctx.llm and could not be driven by the scripted model, so compile and replay were judged from code and its tests.",
   "tests": "unittest: 62 passed, 0 failed (python -m unittest discover -s tests)",
   "useful": 4,
   "reliable": 4,
   "polish": 3,
   "creative": 4,
   "beyond": 4,
   "rank": 5,
   "best": "Freezes a finished agent turn into a checked, model-free workflow that replays Hermes's own tools, stops on drift, sandboxes compiled steps in python -I with no environment, and never approves anything itself.",
   "worst": "No screenshots, demo or Desktop UI, the compile path could not be verified here, and 'hermes freeze' is unavailable under plugins.isolation: host, so /freeze run and scheduling stop working there.",
   "blocked": null,
   "total": 23
  },
  {
   "pr": 152544,
   "name": "crystallize",
   "validate": "passed (ok: true), no warnings, security scan safe. Isolation verdict: in_process only (live_notice.py uses ctx.post_message). Hermes main prints 'unknown manifest field(s) ignored: provides_middleware' on every load.",
   "inside": "loaded: enable exit 0, doctor OK with 4 tools and 8 hooks, plus llm_request and tool_execution middleware, /crystallize, 'hermes crystal' (many subcommands) and a Desktop pane. Calls: crystal_list returned {routines: []}; crystal_run {routine_id: 'does-not-exist'} returned 'No routine with id ... Do the task yourself.'; crystal_full_result with an unknown id returned a clear 'unavailable or expired' error; crystal_cheap {subtask: 'yes'} and crystal_run {} were rejected by schema validation (not invoked). The savings path (context rent, cron tune, helper model) needs recorded runs and a helper model and was judged by reading. It edits user config (auxiliary task, tools.tool_search.defer) and cron jobs through the hermes CLI. desktop/plugin.js (2479 lines) passes node --check.",
   "tests": "pytest: 1438 passed, 17 failed, 2 skipped in 39 s; all 17 failures trace to OSError read-only file system on /tmp in this sandbox (tests/test_routines.py, test_execution_features.py, test_prerun.py), not to plugin logic",
   "useful": 4,
   "reliable": 3,
   "polish": 4,
   "creative": 4,
   "beyond": 5,
   "rank": 3,
   "best": "Attacks the real bill with llm_request middleware that stops re-sending old tool results, cron snapshot scripts that skip unchanged runs, a helper-model slot and receipts, all with originals recoverable and a Desktop pane.",
   "worst": "Sixty-odd modules (auction, rent, rhythm, brake, blind) behind a dense, hedged README that retires its own earlier savings tables, a manifest field this Hermes rejects on every load, and it rewrites user config and cron jobs to work.",
   "blocked": null,
   "total": 25
  },
  {
   "pr": 153955,
   "name": "shadow",
   "validate": "passed (ok: true), security scan safe. One warning: catalog rule 9 private core imports at 42 sites (shadow/common/discovery.py, models.py: hermes_cli.auth, config_backend, inventory, agent.models_dev, agent.usage_pricing and more). Isolation verdict: in_process only (ctx.post_message in switch and trial).",
   "inside": "loaded: enable exit 0, but in a non-interactive session the declared capabilities llm.provider_override and llm.model_override were NOT granted (fail closed, as designed); doctor OK with 6 tools and 22 hooks plus tool_execution, llm_execution and llm_request middleware, /shadow, 'hermes shadow' and Desktop RPC. Calls: shadow_setup_check returned 'The setup check did not pass. Nothing was saved.'; shadow_clarify returned a fixed text; shadow_read {path: 123} and shadow_search {} were rejected by schema validation (additionalProperties false). CLI: 'hermes shadow doctor' listed each missing item with its fix (capability grant, a second model, a judge model from a third maker); 'hermes shadow status' said nothing is running; 'hermes shadow results' said no tests yet. A real trial needs two providers plus a judge and spends money up to a cap, so it was judged by reading. desktop/plugin.js (977 lines) passes node --check.",
   "tests": "pytest: 1458 passed, 0 failed, 436 skipped (tests/); acceptance/ suite not run",
   "useful": 4,
   "reliable": 4,
   "polish": 5,
   "creative": 5,
   "beyond": 5,
   "rank": 1,
   "best": "Replays your own recent turns on a candidate model and your current one in isolated children, has a third-maker judge compare them blind in both orders, then switches the default model gradually with monitoring, rollback and undo, all behind explicit capability grants and a spend cap.",
   "worst": "It imports 42 private Hermes internals that can break on any release, needs three providers before it does anything, and 436 of its tests are skipped.",
   "blocked": null,
   "total": 28
  }
 ]
}