{
  "batch": "deep-55",
  "setup": "Installed real Hermes from NousResearch/hermes-agent source commit fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14 (Python 3.14.10); verified plugins install/validate/doctor CLI. Used RUN.md and tryit.py with Nous FakeLLMServer and AIAgent.run_conversation, empty plans then user-shaped calls. All plugin repositories were fetched at the field.json pins, root folders. Supplemental harness preserves full validation JSON, disables tool-search deferral to expose schemas, and disables Syrinx night cutoff for daytime-flow tests; Lethe is activated through its actual slash-command handler and Retouch gets a synthetic local image/mark through its real Workspace API. No live LLM, provider key, Desktop application, Chrome companion, ROS installation or physical robot was used. Runtime/test results below are direct local observations; linked pinned source supports implementation claims. Scores judge only this batch, not overall winners. Harness source: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/tryit.py Ranking tie-break: Retouch and Roomsmith both score 28; repeat image/screenshot work is more likely to keep Retouch installed than episodic room planning. Syrinx ranks behind both because of its outward-action classification gap. Look ranks last on this batch evidence because its supported browser/hardware path could not be exercised and its dependency-complete suite retained failures.",
  "plugins": [
    {
      "pr": 152918,
      "name": "look",
      "validate": "Passed: pinned source catalog validate --install-deps --json exit 0 with all checks passing and security scan safe; doctor/enable exit 0 (14 tools, 7 hooks, 2 middleware declarations). Warning for private core imports in benchmark/probe_profile.py and runtime/native_input.py. First empty validation also emitted a dependency-publication retry warning despite exit 0; subsequent full validation passed.",
      "inside": "Loaded in real Hermes and exposed all 14 schemas when progressive disclosure was disabled. Four planned calls completed in both default and direct-schema runs: gaze_calibrate(mode=hover,reset_drift=true) returned ok=true, chrome_connected=false; gaze_read cleanly returned No active browser task; invalid gaze_choose(candidate_index=0) was rejected by default Hermes argument validation (direct-schema run returned simulated-only error); gaze_benchmark(trials_per_sigma=10) returned synthetic resolution metrics. No call crashed, but the benchmark is not evidence of real eye-tracking accuracy. Could not exercise pointing, voice, browser approvals or live Chrome on the supported Apple Silicon platform. Sources: https://raw.githubusercontent.com/SilentHacks/look/7c01a94aa651adedaf27e5216598c6deef293050/tools.py and https://raw.githubusercontent.com/SilentHacks/look/7c01a94aa651adedaf27e5216598c6deef293050/README.md",
      "tests": "Final dependency-complete pytest run: 365 passed, 16 failed, 393 subtests passed. Fourteen failures directly raised PermissionError when the sandbox denied AF_UNIX sockets; another setup assertion and a private-directory assertion failed. Earlier collection lacked numpy and later failures lacked huggingface_hub; both were installed before the final count. Read README, manifest, tools, browser ownership/actions/extension code, speech/download handling, platform limitations and tests. No physical browser, microphone or eye-tracking acceptance test.",
      "useful": 3,
      "reliable": 3,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 25,
      "rank": 6,
      "best": "Point-and-ask browser context, local speech and guarded page actions create a distinctive way to interact with Hermes that a text skill cannot reproduce.",
      "worst": "The supported product is an Apple Silicon source beta with substantial browser/native setup, while this run could verify only offline logic and clean missing-browser errors, with 16 remaining test failures.",
      "blocked": null
    },
    {
      "pr": 153724,
      "name": "roomsmith",
      "validate": "Passed: hermes plugins validate --install-deps --json returned exit 0 and ok=true; all 16 checks passed, no warnings; doctor and enable also exit 0 (11 tools, 1 pre_llm_call hook).",
      "inside": "Loaded in real Hermes. Empty plan completed; default progressive disclosure exposed no extra direct schemas, but all five planned plugin calls dispatched successfully: created a measured 4 x 3 x 2.5 m room (revision 1), atomically added a 1.2 x 0.75 x 0.6 m desk and USD 800 budget (revision 2), inspected matching state, rejected width_m=-1 with out_of_range and field name, and exported a 711-byte summary. No crash. Desktop editor, photo inference and live shop fetch were not exercised. Source: https://raw.githubusercontent.com/devSebb/roomsmith/421e6f30fca43066f75ed7258281315c7eeb2fde/rs_core/service.py Supplemental direct-schema run exposed all 11 tools and repeated the five-call workflow successfully.",
      "tests": "Passed: Python pytest tests: 254 passed, 1 skipped, 2 warnings; npm ci succeeded, editor tsc --noEmit passed, node editor tests 13 passed/0 failed. Inspected engine, tools, geometry/fit checks, revision storage, product-fetch code and Desktop/dashboard source; did not launch Desktop.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "A shared, revisioned room engine and finished Desktop editor let the human and Hermes manipulate the same geometry, with fit checks and exact budgets rather than prose guesses.",
      "worst": "Its benefit is episodic room planning, and photo dimensions, retailer extraction and product-shaped stand-ins still require human measurement and review (README limitations: https://raw.githubusercontent.com/devSebb/roomsmith/421e6f30fca43066f75ed7258281315c7eeb2fde/README.md).",
      "blocked": null
    },
    {
      "pr": 153302,
      "name": "lethe",
      "validate": "Passed: catalog validate --install-deps --json exit 0, ok=true, all 16 checks passed, no warnings; enable/doctor exit 0, registering 0 tools and 5 hooks.",
      "inside": "Loaded; intentionally adds no tools. Baseline without activation allowed a memory write. In the supplemental real AIAgent session, called the real /lethe handler before the conversation, then planned memory(add), skill_manage(create), and malformed memory(action=invalid,target=invalid). All three were intercepted before persistence with the explicit Not saved explanation, not a crash. /lethe nonsense returned help; status/where counted 3 vetoes; /lethe-end reported transcript deletion. Independently queried the scratch state.db afterward: zero session rows. Did not test a live external memory provider or gateway/desktop exit flow. Sources: https://raw.githubusercontent.com/itsflownium/hermes-lethe/8b93e2dce9402648c7e238c1d8c83dc649f2f204/guard.py and https://raw.githubusercontent.com/itsflownium/hermes-lethe/8b93e2dce9402648c7e238c1d8c83dc649f2f204/shred.py",
      "tests": "Passed: 67 Python tests, 0 failures. Read tests for the review-fork session id, per-session guards, receipts, arming expiry and deletion failures, plus manifest, README, Desktop/backend, storage and finalization code. Background review coverage is tested through callbacks, not a live model review in this run.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "Per-chat persistence vetoes and transcript deletion change Hermes memory behavior at runtime, giving a broadly useful private-chat mode a prompt cannot enforce.",
      "worst": "External memory-provider per-turn synchronization remains outside its protection, explicitly disclosed in the pinned README (https://raw.githubusercontent.com/itsflownium/hermes-lethe/8b93e2dce9402648c7e238c1d8c83dc649f2f204/README.md), so this is scoped privacy rather than universal erasure.",
      "blocked": null
    },
    {
      "pr": 153938,
      "name": "syrinx",
      "validate": "Passed: catalog validate --install-deps --json exit 0, ok=true, all 16 checks passed; warning for private core import runtime.py:470 agent.pet.store.register_local_pet. Enable and doctor exit 0: 4 tools, 6 hooks.",
      "inside": "Loaded. Default empty and planned conversations were cleanly blocked before the model by the active night cutoff, returning /late guidance and queuing the request. Set night_enabled=false only in the scratch profile for the flow exercise and exposed all 4 schemas. Five calls ran: syrinx_flow(15), syrinx_park(question/default pnpm), syrinx_status (active, parked=1), syrinx_flow(minutes=bad) (structured error), syrinx_flow(0) (ended=true, parked=1). No thrown exception. Did not wait for the focus timer or morning delivery; tests use private clocks. Source: https://raw.githubusercontent.com/cstljuan/syrinx/444db83b1a4fe902916cec25b2de27c389075538/core.py",
      "tests": "Passed: 64 Python tests, 0 failures, including core, runtime, presence, backend, receipt card and Argus. Read the hooks, classifier, focus/return state machine, storage, Desktop source and manifest. No live macOS auto-focus or Desktop run. A separate non-executing classifier probe confirmed git push is held, but rsync and browser_exec with a POST body are not; no network action was executed.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "It gives the agent a useful focus-and-return lifecycle, batching questions and counting actual work through hooks while leaving approval prompts alone.",
      "worst": "Its promise that nothing leaves the machine outruns a regex classifier: core.py explicitly exempts scp/rsync-labelled transfers and treats browser tools as web reads, leaving outward paths unheld.",
      "blocked": null
    },
    {
      "pr": 153090,
      "name": "retouch",
      "validate": "Passed: catalog validate --install-deps --json exit 0, ok=true, all 16 checks passed, no warnings; enable and doctor exit 0 (4 tools, no hooks).",
      "inside": "Loaded and exposed all 4 tools when progressive disclosure was disabled. Initial unseeded plan listed an empty workspace and returned structured errors for an invalid project. For the full workflow, created a 128 x 96 synthetic PNG and user-style rectangular mark through the real Workspace backend, then ran five calls inside AIAgent: retouch_get; retouch_edit(adjust, exposure +1); retouch_reply(needs_review, explicitly noting no visual assessment); retouch_export(png); retouch_get(project_id=../../invalid). All returned cleanly. Edit produced version 2, reported 813 pixels changed inside and 0 outside; an independent comparison of saved version images confirmed both counts and the PNG export existed. No live generative provider, actual image aesthetics or Desktop UI was tested. Sources: https://raw.githubusercontent.com/Lyfae/hermes-retouch/1a5ac4346b991754a25a5dc14745f533341c8c95/retouch/service.py and https://raw.githubusercontent.com/Lyfae/hermes-retouch/1a5ac4346b991754a25a5dc14745f533341c8c95/retouch/composite.py",
      "tests": "Passed: 266 unit/integration tests, 0 failures; separately 6 property/state-machine/concurrency tests passed (80 warnings). Read README, manifest, handlers, region compositing, version store, image ingest/export, provider/path restrictions, backend and Desktop hidden-session orchestration. Fixture seeding initially required correcting the judging harness to exchange its generated project id through a file because Hermes redirected subprocess stdout; that was a harness issue, not a plugin call failure.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Human-drawn regions become separate agent work threads, and the saved edit actually preserved every outside pixel, combining a new Desktop workflow with a testable editing guarantee.",
      "worst": "Desktop is required to create marks and generative removal still depends on a configured image provider and human aesthetic review (https://raw.githubusercontent.com/Lyfae/hermes-retouch/1a5ac4346b991754a25a5dc14745f533341c8c95/README.md), although local adjustments work without that provider.",
      "blocked": null
    },
    {
      "pr": 149313,
      "name": "hermes-ros",
      "validate": "Passed: catalog validate --install-deps --json exit 0, ok=true, all 15 checks passed and no warnings; enable/doctor exit 0 with 9 tools and 1 pre_tool_call safety hook.",
      "inside": "Loaded in real Hermes with the shipped mock robot, not a substitute backend written by the judge. All five planned calls returned: ros_status showed default 0.5 m/s limit; ros_camera_snapshot wrote a 320 x 240 PNG; ros_drive(linear=0.2,duration_s=1) moved x from 0 to 0.2 with 10 publishes then zero velocity; ros_drive(linear=3) was blocked by over_speed_linear before motion; ros_stop(estop=true) latched the stop and required human /ros clear. Default progressive disclosure hid direct schemas but dispatch worked. Real rosbridge, ROS2, cameras and hardware were unavailable and not verified locally. Source: https://raw.githubusercontent.com/arcabotai/hermes-ros/57a67b89142a6b7659a2b40c4796f8e1cad5e542/safety.py Supplemental direct-schema run exposed all 9 tools and reproduced the same results.",
      "tests": "Passed after installing the declared development dependency pyyaml: 70 passed, 11 skipped, 1 warning. Initial collection failed because pyyaml was absent; rerun passed. ROS2/real-backend integration tests skipped; read tool handlers, rosbridge backend, safety interlock, nonce verification, settings, mock, dashboard and tests.",
      "useful": 3,
      "reliable": 5,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "A working robot toolchain with mock parity, bounded motion, fail-closed interlock checks and a human-cleared stop is a concrete extension from agent software into hardware.",
      "worst": "The install audience is mostly robot owners, and the documented limits include no actions/Nav2 interface and safety bypass through non-ROS tools (https://raw.githubusercontent.com/arcabotai/hermes-ros/57a67b89142a6b7659a2b40c4796f8e1cad5e542/README.md).",
      "blocked": null
    }
  ]
}