{
 "batch": "deep-18",
 "setup": "Cloned NousResearch/hermes-agent, uv sync --locked --python 3.14 (Python reinstalled inside test/scratch because the sandbox could not read the first copy); ran tryit.py from RUN.md on each plugin's pinned commit; installed pytest, jsonschema and pyyaml into that venv to run plugin tests.",
 "plugins": [
  {
   "pr": 153549,
   "name": "flota",
   "validate": "failed: security scan (flota.yaml:15-16 flagged ssh_backdoor, sudo_usage in the catalog entry); all other checks passed; warns of private core imports (hermes_cli.profiles, cron.jobs, tools.environments.ssh)",
   "inside": "loaded; enable and doctor OK: 1 tool (flota), 2 hooks (subagent_start/stop), delegation provider, CLI and slash command. Calls worked: status with no machines -> {machines: []}; wake of unknown machine -> clean error naming configured machines; bad action rejected by schema. No real machine to SSH/wake, so the wake/sleep path was judged by reading.",
   "tests": "passed: 899 passed, 5 skipped",
   "useful": 4,
   "reliable": 2,
   "polish": 5,
   "creative": 5,
   "beyond": 5,
   "total": 26,
   "rank": 3,
   "best": "A delegation backend that wakes a sleeping home machine over SSH/Wake-on-LAN, runs the subagent there and suspends it afterwards, never while busy; only a plugin can do this, with 899 tests and a measured demo.",
   "worst": "The catalog security scan fails (sudo/ssh strings in plugin-catalog/flota.yaml) and it leans on private core imports, so it would be sent back until fixed; needs a Linux home lab with SSH.",
   "blocked": null
  },
  {
   "pr": 153960,
   "name": "on-air",
   "validate": "passed",
   "inside": "loaded; enable and doctor OK: 24 tools in 2 toolsets, 3 hooks, CLI `hermes onair`, Desktop and dashboard pages. Calls: setup/stage status refused cleanly ('No owner session could be established'), wrong-arg calls rejected by strict schemas (additionalProperties false, required text). No real OBS/Twitch/owner session, so the live show was judged by reading.",
   "tests": "mostly passed: 2330 passed, 114 skipped, 1 failed (tests/setup/test_stage_runtime.py::test_registered_agent_action_runs_the_prepared_stage_lifecycle); ran 7 min",
   "useful": 3,
   "reliable": 4,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 24,
   "rank": 3,
   "best": "A guarded AI co-host for streams with owner approval, safety filters, OBS control and a Live desk, polished with docs and strict schemas and a huge test suite.",
   "worst": "Niche audience (streamers), very large surface (24 tools) that needs OBS/Twitch/voice, and one of its own tests fails here.",
   "blocked": null
  },
  {
   "pr": 151448,
   "name": "muster",
   "validate": "passed",
   "inside": "doctor OK (2 tools, 14 hooks) but `hermes plugins enable` refused: uv lock could not build the editable muster distribution ('Distribution not found'), so inside Hermes muster_status and muster_fanout did not exist ('Tool does not exist'). Hooks/ledger judged by reading.",
   "tests": "passed: 34 passed",
   "useful": 4,
   "reliable": 2,
   "polish": 4,
   "creative": 3,
   "beyond": 4,
   "total": 21,
   "rank": 6,
   "best": "Durable sqlite ledger that tells you which delegated subagents are running, stuck, spinning or lost, even after a crash, with stop/steer for the ones it starts.",
   "worst": "Hermes refused to enable it here because its pyproject declares a build-system with no packages, so its tools never loaded; only 34 tests for a hook-heavy plugin.",
   "blocked": null
  },
  {
   "pr": 150644,
   "name": "hermes-canvas",
   "validate": "passed",
   "inside": "loaded; enable and doctor OK: 4 tools (canvas_draw/update/get/next), 7 hooks, Desktop pane. Calls worked: canvas_draw made 2 boxes bound to files/check and returned a card; canvas_next listed the ready box with a delegate_task goal; canvas_get returned state; bogus op rejected by the schema.",
   "tests": "passed: 306 passed",
   "useful": 4,
   "reliable": 5,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 26,
   "rank": 2,
   "best": "The agent's plan becomes a canvas whose box status comes from real file writes and check exit codes, the user can edit it mid-build, and hooks hold a finish while a check fails; 306 tests, replay GIF.",
   "worst": "Only helps multi-step coding tasks in Desktop, and box checks run model-supplied shell commands.",
   "blocked": null
  },
  {
   "pr": 153020,
   "name": "breadcrumbs",
   "validate": "passed",
   "inside": "loaded; enable and doctor OK: memory provider/context engine (kind exclusive), 0 register_tool tools, 3 hooks; crumb and crumb_find come from the memory provider so the test harness did not offer them ('Tool does not exist'); judged by reading and tests.",
   "tests": "passed: 669 passed, 25 skipped, 41 subtests",
   "useful": 4,
   "reliable": 4,
   "polish": 5,
   "creative": 5,
   "beyond": 5,
   "total": 28,
   "rank": 1,
   "best": "A content-addressed memory and context engine where every fact keeps its exact source and the model zooms with crumb(id, level), with no model calls, a receipt per read and 669 tests.",
   "worst": "Its tools could not be driven in the scripted harness, and replacing the memory/context engine is a bigger commitment (exclusive) than adding a tool.",
   "blocked": null
  },
  {
   "pr": 152159,
   "name": "genui",
   "validate": "passed",
   "inside": "loaded; enable and doctor OK: 3 tools (show_component, update_component, ask_user), 1 hook, Desktop cards. Calls: task_card with missing props returned precise schema errors; unknown type rejected by enum; update of unknown id gave a clear error; ask_user ok.",
   "tests": "partly run: 39 passed (test_adapter, test_front, test_bench need playwright/Anthropic and were skipped)",
   "useful": 3,
   "reliable": 4,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 24,
   "rank": 4,
   "best": "Live Desktop cards (PR, task, review list, study) updated in place with buttons that send the decision back to the agent, validated against JSON schemas.",
   "worst": "Four fixed card types and it needs Hermes Desktop, so command-line users gain little; some tests need playwright.",
   "blocked": null
  }
 ]
}