{
  "batch": "deep-04",
  "setup": "Installed real Hermes once from NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14 (CPython 3.14.10); installed pytest test dependencies in that scratch environment. Read field.json and checked out each exact SHA/subdir. Used supplied tryit.py for all six empty-plan loads, catalog admission with --install-deps, enable, doctor and AIAgent.run_conversation against Nous FakeLLMServer. Plugin tools are deferred in this Hermes, so scenarios use tool_describe followed by four tool_call invocations. Reused the successfully enabled scratch homes for those conversations, changing only the fake-model endpoint: repeated fresh-home dependency preparation was obstructed by the evaluation isolation guard, not plugin code. Tironian has no tools, so tested its real Hermes CLI four times instead, alongside its hook-driven chat. All state and fixtures stayed in test/scratch; optional host/device probing was blocked and was not evidence. Local test servers bypassed proxy routing. A neutral synthetic home name and a differently named Wear OS test checkout fixed harness artifacts; no plugin implementation was edited. Tests ran without an explicit Hermes-source PYTHONPATH; Wear OS used its documented HERMES_AGENT_ROOT bootstrap. No real SSH fleet, AWS account, GPU training, watch or native Desktop renderer was exercised; Drawbridge smoke uses simulated UI/transport with the real native file API. Runtime outcomes below are my observations, not upstream claims; sources.json supplies pinned source evidence. Equal totals are ordered by expected lasting usefulness: personal model learning, general compute, then specialized paper visualization.",
  "plugins": [
    {
      "pr": 154010,
      "name": "own-compute",
      "validate": "Passed catalog admission (--install-deps, exit 0), enable and doctor. Pinned repository Siioh/hermes-own-compute@97522db41e7f6352089bd350df45bc64267a71e6, root; manifest id is cloud-compute. Source evidence: README.md, core/instances_api.py and core/agents.py in sources.json.",
      "inside": "Loaded: 15 tools, 2 hooks, /compute, a guide skill and system-prompt context; Desktop compute UI and backend are present in the source. In real Hermes, compute_status returned configured=false, empty hosts/jobs/environments, setup guidance and six templates; compute_host(action=types) described SSH, Slurm, EC2 and local hosts; env_exec on missing-env returned structured not_found; invalid compute_host action was rejected by Hermes schema validation before invocation. These are setup/error-path checks, not evidence that a remote workload or EC2 billing controls worked live. Source review found approval, TTL/reaping, secret-redaction and save-before-delete implementations, not just README promises.",
      "tests": "Latest bounded offline-fixture run: 3172 passed, 11 failed, 30 skipped, 4 deselected. Nine failures concern stopping/cleanup of synthetic runs: new commit-fetch protection raises ApiError(not_fetched) before older expectations reach destroy; two repository-detection assertions are affected by temp folders being inside a Git workspace (the plugin strips GIT_CEILING_DIRECTORIES). Four exclusions covered PID inspection, a system-service script and two harness home/path assumptions. Initial full run was 3170 passed/17 failed/30 skipped; corrected proxy routing removed two failures. Live host tests stayed skipped. The remaining stop-path test mismatch limits reliability to 3 despite broad coverage.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 3,
      "best": "A substantial, well-documented compute control plane brings sandboxed jobs, Slurm and git-returning agents into Hermes with approvals and a Desktop interface.",
      "worst": "Nine stop/cleanup tests disagree with the pinned not_fetched safeguard, leaving the suite and that important lifecycle behavior insufficiently reconciled.",
      "blocked": null
    },
    {
      "pr": 151546,
      "name": "wearos",
      "validate": "Passed automated catalog admission (--install-deps, exit 0), enable and doctor; security scan reported caution, not a dangerous finding. Pinned borealBytes/hermes-wearos-plugin@2991d28fa7aeeb86d52be77680f4be7c1d80a41a, root. Manual review nevertheless found a catalog rule 11 violation in tts_air.py: _hf_token reads plugin-data/hf_token after the Hermes secret lookup; store_token writes that file. Catalog policy and pinned code are both sourced.",
      "inside": "Loaded: 3 watch tools, 1 session-end hook, /pair, hermes watch, a TTS provider and a Desktop backend/page. Real Hermes watch_pair opened a roughly five-minute pairing window without exposing or approving a code; watch_send with Build complete. returned no paired watch and a setup hint; empty text returned text is required; passing code=123456 to watch_pair was explicitly refused as a human-only operation. Pairing crypto/grants/revocation were read and fixture-tested, not demonstrated with physical hardware. Host/Tailscale readiness diagnostics were blocked by the evaluation guard, not scored as plugin failures.",
      "tests": "177 passed plus 3 subtests passed in the exact pinned code copied to a differently named test checkout. Initial proxy routing caused local HTTP fixture failures; a remaining bare wearos module assertion was caused by pytest importing the checkout directory named wearos, and disappeared with the neutral checkout name. No physical watch, tailnet enrollment, gated TTS weights or voice cloning was tested.",
      "useful": 3,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 23,
      "rank": 6,
      "best": "It adds a real wrist interface, human-only pairing and scoped delivery rather than merely forwarding a prompt to a watch.",
      "worst": "The Hugging Face token fallback reads a plugin-owned credential file instead of using Hermes exclusively, violating catalog rule 11 despite the automated check passing.",
      "blocked": null
    },
    {
      "pr": 153549,
      "name": "flota",
      "validate": "Failed automated catalog admission (exit 1): security scan marked dangerous ssh_backdoor at plugin-catalog/flota.yaml:16 and sudo_usage at :15. Those lines are disclosures about authorized_keys and a restricted sudo command, so the scanner result is not evidence of a malicious backdoor, but the brief still caps Reliable at 2. Enable and doctor passed. Separately, ssh_guard.install assigns cls.__init__ on the imported core SSHEnvironment, a manual rule 9 conflict that the automated no-core-override check missed. Pinned Bto-03/hermes-flota@0b480a64f2e7229169dc3003d029f251d803a5da, root.",
      "inside": "Loaded: flota tool, 2 subagent lifecycle hooks, delegation provider, /flota and CLI, with fleet Desktop/dashboard source. In real Hermes status returned an empty fleet; wake and sleep of missing-machine returned clear Unknown machine errors; an invalid action was rejected by Hermes schema validation. Read provider routing, wake methods, busy protection, idle sleep and pairing code. No configured machine was contacted, awakened or suspended. Its own documentation acknowledges goal-based child matching and a shared busy-state file that concurrent gateway/chat processes can overwrite.",
      "tests": "899 passed, 5 skipped, 1 warning in the pinned offline suite. Tests cover routing, wake/power rules, busy protection, SSH sync guard, pairing, UI/backend and CLI fixtures; they do not replace real-hardware verification here.",
      "useful": 4,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 5,
      "best": "The delegation backend and subagent lifecycle hooks make owned machines wake for actual agent work and sleep afterward, a useful extension a skill alone cannot supply.",
      "worst": "Admission fails, and the SSH constructor patch plus documented concurrent busy-state collisions undermine the otherwise strong safe fleet lifecycle.",
      "blocked": null
    },
    {
      "pr": 152755,
      "name": "hermes-drawbridge",
      "validate": "Passed catalog admission (--install-deps, exit 0), enable and doctor. Pinned sohomsahaun/hermes-drawbridge@84c5c7cf959c75037d1ea88ead599ba639368185, root. Source evidence includes README.md, dashboard/plugin_api.py and tests/smoke.mjs.",
      "inside": "Loaded: drawbridge_folder, 2 approval hooks and the Desktop folder-share/backend integration. In real Hermes list_folders returned an empty list; request_access(read,300 seconds) returned a request id and transcript directive; reading hello.txt without a share returned offline; ../outside.txt returned invalid-request. The separate smoke suite exercised actual Hermes native file operations through a simulated Desktop client, including read/write approvals and profile/session boundaries. No requester files were shared. Review found scoped grants, heartbeat expiry, strict relative paths and version-checked edit proposals.",
      "tests": "Passed the complete repository smoke suite: node --experimental-transform-types tests/smoke.mjs HERMES_SOURCE, using the Hermes environment Python, on Linux. One end-to-end suite passed with many assertions (no numeric test-case count emitted), against the real native file API with simulated UI/transport. No pytest suite is present. Visual Desktop rendering and the advertised Windows target were not independently run.",
      "useful": 5,
      "reliable": 5,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "It solves a common remote-agent problem with explicit, per-chat local folder access and reviewed edits, and its native-file smoke suite substantiates the security boundaries.",
      "worst": "The product depends on matching Desktop/backend capabilities, lacks a supplied visual demo in this pin, and explicitly acknowledges the editor-save race between final comparison and write.",
      "blocked": null
    },
    {
      "pr": 152533,
      "name": "vellum",
      "validate": "Passed catalog admission (--install-deps, exit 0), enable and doctor. Pinned 0xsaltylollipop/vellum@286b44d587500bd4792decdc73370791262aec0b, root. Pinned README.md, paper_anim/turns.py and paper_anim/canvas.py are sourced.",
      "inside": "Loaded: 4 canvas/paper tools, 5 hooks, the paper-canvas skill and Desktop/browser/backend source. First real-Hermes scenario correctly blocked read/draw/look/clear outside a registered paper turn. Then used the plugin storage helpers to register a synthetic clicked card and cached paper: real Hermes vellum_read returned that fixture text; a string offset was rejected by Hermes schema validation; clearing another card was blocked by the plugin ownership hook; drawing on the right card returned its bounded no-Desktop-answer error after about 30 seconds. The whole conversation completed. No figure pixels, audio synchronization or browser sandbox were run live; sandbox/CSP and sentence/canvas integration were read in source.",
      "tests": "73 passed in the pinned pytest suite, covering inputs, paper-turn authorization, context defusing, bounded results and voice/canvas state. Two real-Hermes conversations additionally covered ordinary-chat refusal and a seeded paper workflow; the seed was explicitly synthetic, not fetched research content. There is no screenshot/demo asset in the pinned tree, so visual finish remains less substantiated than the concept.",
      "useful": 3,
      "reliable": 4,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "A persistent interactive figure inside the paper, synchronized with the user’s own Hermes conversation, is a notably original use of Desktop and streaming hooks.",
      "worst": "Its audience is mainly arXiv readers, and the central rendered drawing/voice experience remains unverified here because no Desktop renderer was attached.",
      "blocked": null
    },
    {
      "pr": 151281,
      "name": "tironian",
      "validate": "Passed catalog admission (--install-deps, exit 0), enable and doctor. Pinned angelraph/tironian@9b590cc094f4b65cd70e979f632e1783424526a8, plugin/ subdirectory. README.md, plugin/core/engine.py and plugin/core/cli.py are sourced. Reliable is capped at 2 by the actually reproduced invalid-family CLI crash, despite the passing catalog check.",
      "inside": "Loaded with no model tools: 11 hooks, llm_execution middleware, /tironian, hermes tironian CLI, auxiliary judge slot and Desktop source. The empty-plan real AIAgent conversation completed and its turn appeared in Tironian status. Four Hermes CLI calls: status succeeded; families returned no families; demo completed its synthetic 93-session/183-request history import, dry-run training, weak-model rejection, good-model 30/30 first and 30/30 follow-step gates, 20 shadow comparisons, live routing and malformed-output fallback; dataset missing-family exited 1 with an uncaught ValueError: unknown family missing-family. The demo explicitly uses scripted history/models and dry-run training; it is not a real fine-tuning benchmark. No GPU job or claimed laptop accuracy was independently reproduced.",
      "tests": "104 passed on the exact pinned pytest suite after correcting the evaluation harness’s synthetic home name. Earlier three pipeline failures were caused by the artificial home basename user: its username redactor then changed message role user, so they are not counted as an ordinary-install regression. The separate actual CLI invalid-family crash remains a product defect not covered by those passing tests.",
      "useful": 4,
      "reliable": 2,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 2,
      "best": "It closes a real self-improvement loop from Hermes sessions to held-out-gated personal students, with model-execution routing and silent fallback demonstrated by its honest offline demo.",
      "worst": "hermes tironian dataset missing-family produces an uncaught ValueError traceback rather than a clean user error, forcing Reliable to 2 under the brief’s crash cap.",
      "blocked": null
    }
  ]
}