{
 "batch": "trial-2",
 "setup": "Hermes installed from source as RUN.md says: git clone --depth 1 of NousResearch/hermes-agent (HEAD 34ce08bb, 2026-11-11), uv 'sync --locked --python 3.14' with uv installed into test/scratch; 'hermes plugins --help' lists install, validate and doctor. tryit.py ran each plugin: validate, enable and doctor ran for real. Every 'hermes chat' exited 1 because this machine's sandbox forbids AF_UNIX sockets (socket.socket(AF_UNIX) -> EPERM), so the Hermes gateway control socket could not start (RuntimeError 'gateway session bootstrap control listener unavailable'), even with no plugin. No scripted-model turn ran. Instead, a scratch harness used the same tryit Hermes home: Hermes's own PluginManager discovered and loaded each plugin in-process, and the planned calls went through model_tools.handle_function_call, Hermes's real tool dispatcher. Separately, 'hermes plugins enable' was refused for both plugins that ship a buildable pyproject.toml (hermes-custodian, operate-hermes-plugin) with 'uv lock exited 2 ... Distribution not found at .../plugin-sources/...'. Both build cleanly with 'uv build' outside Hermes, so this was treated as an environment-manager problem on this machine and not held against them; they were marked enabled in the scratch config.yaml and loaded fine. The two hosted-MCP packages (dyslexiasteps, makeirl) add no Python tools; their servers were called directly over MCP JSON-RPC, because Hermes's MCP client needs the gateway session. understory's pinned repo returns 404 and was not judged.",
 "plugins": [
  {
   "pr": 151295,
   "name": "dyslexiasteps",
   "validate": "passed: all 15 checks ok, no warnings, isolation verdict 'portable' (Agent Plugins v1: plugin.json + mcp.json + skills/)",
   "inside": "enable exit 0; doctor OK: 0 tools, 0 hooks, one skill (phonics-word-lists) and one remote MCP server, no Python code. The tool list comes from the server, so Hermes's MCP client was not exercised (no gateway here). Called https://mcp.dyslexiasteps.com/mcp directly: initialize and tools/list worked (6 read-only tools); get_words_for_grade grade-1/sight/limit 8 returned 8 of 136 words with source lists; find_words with a wrong argument name, get_word_list id 'no-such-list' and grade 'grade-9' each returned a clean isError message saying what to fix.",
   "tests": "none in the package (the server's source is a separate repo, not judged)",
   "useful": 2,
   "reliable": 3,
   "polish": 3,
   "creative": 2,
   "beyond": 2,
   "total": 14,
   "best": "A no-key, read-only, well-described hosted word-list service with clean validation errors and honest disclosure that the tool set is not pinned.",
   "worst": "Configuration only: a static phonics dataset behind a remote MCP server whose tools can change under the pinned sha, serving a narrow audience; a skill with web access could cover most of it.",
   "blocked": null
  },
  {
   "pr": 152879,
   "name": "understory",
   "validate": "not run: the pinned repository could not be fetched",
   "inside": "not run: git fetch of https://github.com/projectmehari/understory at f22c33c3 failed ('could not read Username'); the GitHub API returns 404 Not Found for the repo, it is not among the owner's public repos, and the Hermes catalog at 34ce08bb has no understory.yaml",
   "tests": "not run",
   "useful": 0,
   "reliable": 0,
   "polish": 0,
   "creative": 0,
   "beyond": 0,
   "total": 0,
   "best": "Nothing could be judged.",
   "worst": "The pinned repository is private or deleted (GitHub 404), so no code exists to judge.",
   "blocked": "The repository named in field.json (projectmehari/understory) returns 404 on GitHub, so its pinned commit could not be read."
  },
  {
   "pr": 149965,
   "name": "agenda-guardian",
   "validate": "passed: all checks ok, security scan safe, declared tools/hooks match registrations, no warnings",
   "inside": "enable exit 0, doctor OK; loaded in Hermes with 15 tools (day_setup ... day_schedule), 2 hooks (pre_gateway_dispatch, pre_tool_call), the /day slash command and a system-prompt section. All 5 planned calls ran: day_setup {} returned onboarding questions and correctly saw the Google skill as not signed in; day_setup settings saved work_hours and timezone; day_add_task added #1 (90 min, priority 1, due today); day_plan explained there was no calendar and why the task did not fit before 17:00 (run late in the day), naming the rules used; day_done with an unknown task returned ok:false 'no open task matches'.",
   "tests": "passed: 449 passed, 1 skipped (pytest, Python 3.12, no Hermes needed)",
   "useful": 4,
   "reliable": 5,
   "polish": 4,
   "creative": 4,
   "beyond": 4,
   "total": 25,
   "best": "A deterministic planner that learns your own estimate error, explains every block by the rule behind it, and writes to Google Calendar only after a typed /day write on and a validated proposal, with marker-checked undo; 449 tests pass.",
   "worst": "Very large surface (15 tools, 2 hooks, cron scripts, a dashboard tab, a dense 2,000-word catalog description); its pre_tool_call guard is self-described as a convenience rather than a security boundary, and calendar writing depends on the separate Google Workspace skill.",
   "blocked": null
  },
  {
   "pr": 151267,
   "name": "makeirl",
   "validate": "passed: all checks ok, no warnings, isolation verdict 'portable' (plugin.json + mcp.json + skills/)",
   "inside": "enable exit 0; doctor OK: 0 tools, 0 hooks, one skill (makeirl-release-gate) and one remote MCP server marked untrusted. No tool could be called: a direct MCP initialize to https://makeirl.com/api/mcp returned HTTP 401 with an OAuth Bearer challenge (resource_metadata, scope 'read write'), as disclosed. Needs a MakeIRL account and 'hermes mcp add makeirl --auth oauth'.",
   "tests": "none in the package",
   "useful": 2,
   "reliable": 3,
   "polish": 2,
   "creative": 3,
   "beyond": 3,
   "total": 16,
   "best": "Puts real KiCad 9 ERC/DRC/BOM/parity manufacturing checks in front of an agent with a release-gate skill, and its disclosure is candid about the 401, OAuth via Hermes and browser-only payment.",
   "worst": "A thin config wrapper around a paid, account-gated hosted service whose tools are not pinned; the bundled mcp.json alone does not work, and by its own disclosure the OAuth setup step drops the package's untrusted setting unless the user sets it again.",
   "blocked": null
  },
  {
   "pr": 153944,
   "name": "hermes-custodian",
   "validate": "passed: all 15 checks ok, security scan safe, 1 pyproject requirement, isolation verdict 'host'",
   "inside": "doctor OK (4 tools, 0 hooks). 'hermes plugins enable' was refused here by Hermes's uv lock step ('Distribution not found' for the editable plugin source); the package builds fine with uv build, and operate-hermes-plugin failed the same way, so this was treated as the environment. With it marked enabled, Hermes loaded it and registered custodian_status, custodian_findings, custodian_changes and custodian_scan (toolset custodian). All 3 calls ran: status and scan returned a clean ok:false 'Not configured for this profile. Run custodian setup...'; findings with an argument was refused: 'This tool takes no arguments; change scope through trusted setup.'",
   "tests": "mixed: 382 passed, 71 failed, 9 skipped (pytest, Python 3.12). 55 failures are 'Fixture base directories are forbidden for privileged callers' because this machine runs as root, 8 lack the optional textual extra, and most of the rest are sandbox effects (read-only fs for systemd-analyze, EPERM, root-owned install).",
   "useful": 3,
   "reliable": 4,
   "polish": 4,
   "creative": 3,
   "beyond": 3,
   "total": 20,
   "best": "A carefully hardened, cross-platform (Linux/macOS/Windows) local integrity and permission monitor for a Hermes host, with consent-scoped setup, a loopback dashboard and TUI, screenshots and about 7,000 lines of tests.",
   "worst": "Its agent tools are four argument-less read-only wrappers that do nothing until a human finishes an interactive terminal setup; it shares the name and two tool names (custodian_status, custodian_scan) with the unrelated catalog plugin 'custodian'; and the README says the Pi 5 targets are unverified while docs/verification/pi5-monitor.json records them met.",
   "blocked": null
  },
  {
   "pr": 153405,
   "name": "operate-hermes-plugin",
   "validate": "passed: all checks ok, no warnings, isolation verdict 'host'",
   "inside": "doctor OK (11 tools). 'hermes plugins enable' was refused by the same uv lock 'Distribution not found' error as hermes-custodian (the package builds with uv build), so it was marked enabled in the scratch config; Hermes then loaded 11 tools (operate_create_gig ... operate_heartbeat) and the /operate command. 3 calls ran: operate_list_gigs (no key) returned live marketplace gigs from operate.center; operate_check_queue returned a clean 'Operate.Center API HTTP 401 Unauthorized'; operate_hire_gig with an empty operator_id returned \"Missing required parameter 'operator_id'\".",
   "tests": "none: no tests directory; CI only runs py_compile and an inline mock-registration script",
   "useful": 2,
   "reliable": 2,
   "polish": 3,
   "creative": 3,
   "beyond": 3,
   "total": 16,
   "best": "Gets the key from Hermes's secret scope or the environment (not a file) and lets an agent both sell and buy gigs on a live marketplace, with clean errors for a missing key or argument.",
   "worst": "Moves money without approval inside the plugin (operate_hire_gig places orders and manage_order 'approve' releases escrow; the 'respects the owner's spend policy' claim depends on the server); it sends the Bearer key to any URL that merely starts with the API URL string, e.g. operate.center.evil.com; it claims SHA-256 verification on uploads the code does not do; and /operate demo writes fake earnings that /operate status then shows.",
   "blocked": null
  }
 ]
}