{
  "batch": "deep-x-02",
  "setup": "Hermes source a84025ade22d2a4085f575d4353072e5b702635d cloned once under test/scratch; uv sync --locked --python 3.14 succeeded with workspace-local Python/cache. Hermes plugins --help listed install, validate and doctor. Used the supplied tryit.py and Nous tests/fakes/fake_llm_provider.py against AIAgent.run_conversation, fresh synthetic homes and no live accounts. Initial concurrent bare-home collision and loopback proxy failures were discarded; later runs used distinct homes and loopback proxy bypass. Standard catalog/doctor checks were run; CLI enable attempts exceeded 120 seconds during dependency publication. Supplementary probes use explicit plugins.enabled and toolsets with dependencies preinstalled, and Tool Search disabled so deferred plugin tools are visible; no entrant source edited. Runner forces process exit only after writing the conversation result and closing the agent, to bound teardown. Unit suites also ran in a separate Python 3.14 environment without Hermes imports; Ark and Hypnos were additionally rerun completely in the Hermes environment. Differences and skips are reported per entry. Local execution observations below are our own tests; source URLs support the pinned implementation, not independent verification of author benchmark claims. Desktop UIs, remote storage and real LoRA training were not exercised.",
  "plugins": [
    {
      "pr": 150563,
      "name": "hermes-roam",
      "validate": "Passed catalog validation and runtime doctor at 8b9fba41fea77f2fe2d6d572cffd43039d2da11f; declarations matched registrations and security/core-override checks passed. Standard --install-deps validation also returned exit 0. CLI enable exceeded 120 seconds in dependency publication; subsequent explicit fixture activation loaded cleanly. Source: https://raw.githubusercontent.com/vcruz305/hermes-roam/8b9fba41fea77f2fe2d6d572cffd43039d2da11f/plugin.yaml",
      "inside": "Loaded three tools, three hooks, CLI/slash commands and a setup skill. In the five-call real-Hermes flow, local-backend initialization worked; roam_status returned generation 0, dry-run push planned three files, actual encrypted push created generation 1, and read-only pull preview succeeded. Bad/unauthorized apply=true returned PermissionError with the user-operated restore command. The dry-run response misleadingly used bundle_key=\"invalid-path\" for an empty key. This was a single-device local-store test, not live S3/R2/git or a second-device restore. Implementation: https://raw.githubusercontent.com/vcruz305/hermes-roam/8b9fba41fea77f2fe2d6d572cffd43039d2da11f/README.md At equal total, ranks above Augur because portable agent state is more foundational than a model-dependent focus assistant.",
      "tests": "Complete remaining unit suite: 604 passed, 2 failed, 1 skipped; six tests in test_backend_git_timing.py excluded because its private blackhole endpoint is blocked by the sandbox (initial full-suite attempts stopped there). Failures: 20k-path walk exceeded its 2-second timing bound (3.14 seconds), and human-status output exceeded 80 columns for a long workspace path; optional realistic-home performance fixture skipped. Crypto, archive traversal, merging, rollback, profile and consent tests are among the passing suite. A final targeted rerun passed the 20k-path timing test and reproduced only the 80-column formatting failure (1 passed, 1 failed); the full earlier counts are retained above.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 25,
      "rank": 5,
      "best": "Encrypted, versioned sync with local config boundaries and user-controlled installs makes a Hermes setup portable between machines.",
      "worst": "Multi-device and remote-backend operation remain unverified here, and the observed dry-run placeholder and long status lines detract from the otherwise careful implementation.",
      "blocked": null
    },
    {
      "pr": 154055,
      "name": "augur",
      "validate": "Passed catalog validation and runtime doctor at 6e775b23e2a322c8eee32db4074a44cb85e86e89: declarations matched registration, security scan safe, no core override, Desktop SDK check passed. Both standard --install-deps validation and a direct gate rerun passed. Source: https://raw.githubusercontent.com/Bruno-coder21/augur/6e775b23e2a322c8eee32db4074a44cb85e86e89/plugin.yaml",
      "inside": "Loaded in real Hermes with four tools and six hooks, plus commands, classifier/insight model slots, Friday blueprint and Desktop RPC. In a five-call fake-model conversation, augur_park created item 1; augur_queue listed it; augur_done marked it needs_you; id 999 returned a clear not-found error; augur_report generated a real PNG whose contents I inspected. CLI enable timed out in dependency publication; explicit enabled configuration with preinstalled dependencies worked. The real-model classifier quality, Telegram buttons, Friday cron execution and Desktop chip were not exercised. Implementation and disclosures: https://raw.githubusercontent.com/Bruno-coder21/augur/6e775b23e2a322c8eee32db4074a44cb85e86e89/README.md",
      "tests": "101 passed, 0 failed in the complete isolated Python 3.14 unit suite; the separately run 12 classifier tests also passed (included in the 101). Tests use fake Hermes contexts/models; this is not a reproduction of the author's real-model accuracy evaluation.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "total": 25,
      "rank": 6,
      "best": "Persistent goals, bounded hook-based nudges and a tightly guarded Friday queue turn focus tracking into a usable part of everyday Hermes.",
      "worst": "Its central classification depends on an auxiliary model, while real-model accuracy, Telegram interactions and scheduled execution remain unverified here.",
      "blocked": null
    },
    {
      "pr": 150644,
      "name": "hermes-canvas",
      "validate": "Passed catalog validation and runtime doctor at d6eeff0d6f7ec653262537bdd7bb001cc6d3c7f8: four tools and seven hooks matched, security and core-override checks passed, and Desktop surface stayed within the SDK. Standard --install-deps validation also passed. Source: https://raw.githubusercontent.com/0xbyt4/hermes-canvas/d6eeff0d6f7ec653262537bdd7bb001cc6d3c7f8/plugin.yaml",
      "inside": "Loaded four tools and seven hooks. Five-call fresh-file conversation: canvas_draw created a bound box; real write_file wrote 15 bytes; the exact terminal check exited 0; canvas_get showed done, writes=1 and check_exit=0; an invalid canvas_update attempting to set done manually returned a clear error because the box owns a check. An earlier attempt reused a fixture file and Hermes correctly refused the unread overwrite; it was rerun on a fresh path rather than attributed to the plugin. No Desktop window or concurrent subagent session was exercised. Implementation: https://raw.githubusercontent.com/0xbyt4/hermes-canvas/d6eeff0d6f7ec653262537bdd7bb001cc6d3c7f8/README.md",
      "tests": "Final complete suite: 306 passed, 0 failed in 3.15 seconds, with temporary paths bounded by a synthetic non-repository directory. Earlier runs had 304 passes and 2 failures: the non-repository test inherited the surrounding git checkout, and a 0.5-second stress timing assertion failed under concurrent work; after isolating the directory, one intermediate run still exceeded timing (305 passed, 1 failed), but the final unmodified suite passed completely. Tests cover binding, steering, persistence, APIs, shell parsing and diffs.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 2,
      "best": "A plan whose state follows real writes and checks, with mid-turn user steering and finish gates, substantially improves how Hermes builds software.",
      "worst": "Write attribution depends on recognizable tool and shell patterns, and the Desktop/subagent experience was inspected in source rather than run end to end.",
      "blocked": null
    },
    {
      "pr": 149640,
      "name": "hermes-ark",
      "validate": "Passed catalog validation and runtime doctor at a23e529c1f636aa49d974576cdc5cb15fc8242d3; tools/hooks/middleware declarations, security scan, core-override and Desktop SDK checks passed. Runtime activation after the CLI dependency-publication timeout succeeded with installed dependencies. Source: https://raw.githubusercontent.com/medomohamed22/hermes-ark/a23e529c1f636aa49d974576cdc5cb15fc8242d3/plugin.yaml",
      "inside": "Loaded ark_checkpoint, a session hook, tool-execution middleware, CLI/slash commands; Desktop backend code was inspected separately. Initialized an explicitly unencrypted synthetic local repository; auto-snapshots were disabled only in this fixture. Four real-Hermes calls produced a checkpoint, saved a memory preference, produced another checkpoint, then blocked an agent terminal restore attempt with an explicit user-only administration message. The invalid restore selector never reached the restore engine because middleware blocked it. Encryption, automatic guards, restore and Desktop API paths were exercised in the separate suite; no live S3 or Desktop window. Implementation: https://raw.githubusercontent.com/medomohamed22/hermes-ark/a23e529c1f636aa49d974576cdc5cb15fc8242d3/README.md",
      "tests": "174 passed, 1 skipped in the full Hermes-environment suite after copying identical source to a valid Python package directory hermes_ark and bypassing the proxy for loopback. The skipped test needs live S3 credentials. Earlier run: 171 passed, 3 failed, 1 skipped (two loopback proxy errors and one busy-repository timing assertion); all three passed in the corrected complete rerun. A separate no-Hermes environment passed 76 with 96 integration skips.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "Automatic pre-write checkpoints, item-level recovery and a human-only administration boundary protect the agent memories and skills users accumulate over time.",
      "worst": "Reliable off-site recovery still depends on user-managed storage and passphrase custody, and the live S3 path was not exercised here.",
      "blocked": null
    },
    {
      "pr": 150800,
      "name": "hypnos",
      "validate": "Passed catalog validation and runtime doctor at 0a809b9085fdeaed63fd909b927afcc41de512de; hook/middleware declarations matched registration and the external trainer runtime was declared separately. Security/core-override/Desktop SDK gates passed. Source: https://raw.githubusercontent.com/0xNoramiya/hypnos/0a809b9085fdeaed63fd909b927afcc41de512de/plugin.yaml",
      "inside": "Loaded with no agent-facing tools, as declared: capture hooks, llm_execution routing middleware, commands/CLI and a teacher model slot. In a four-call real-Hermes conversation, memory saved a synthetic pnpm preference; hermes hypnos status reported one captured turn and no brain; habits refresh extracted one procedural habit with ten probes; malformed brain diff with one version returned usage and exit 2. The normal model conversation completed through passthrough routing. No trainer/model was installed and no LoRA was trained or promoted; live/shadow routing gains and the author GPU timelapse are not independently reproduced. Implementation: https://raw.githubusercontent.com/0xNoramiya/hypnos/0a809b9085fdeaed63fd909b927afcc41de512de/README.md",
      "tests": "229 passed, 2 skipped in the complete Hermes Python 3.14 suite with a JUnit report; isolated no-Hermes suite also completed. The suite exercises capture/scrubbing, habit extraction, fake-trainer nights, promotion/rollback, routing/fallback and lineage/unlearning. Real external-trainer integration is opt-in and skipped; synthetic trainer tests are not evidence of actual weight-learning quality.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "Learning procedural habits into a personal LoRA and routing mastered tasks through Hermes middleware is the batch's most ambitious change to the agent itself.",
      "worst": "Its main promise needs a separate training stack and sustained representative data, while actual training, promotion quality and long-term drift remain unverified here.",
      "blocked": null
    },
    {
      "pr": 152691,
      "name": "grimoire",
      "validate": "Passed catalog validation and runtime doctor at 4824773040b131f3179e5896d71d1435908758a7 with DuckDB installed; eight tool and six hook declarations matched, and security/core-override/Desktop SDK gates passed. CLI enable timed out during dependency publication, but explicit fixture activation loaded without an error. Source: https://raw.githubusercontent.com/bettersandeep/grimoire/4824773040b131f3179e5896d71d1435908758a7/plugin.yaml",
      "inside": "Loaded eight tools, six hooks, slash commands and the learning/check machinery. Five real-Hermes calls attached a four-row CSV, profiled a 25% missing amount column, returned South=250 and North=150 with query receipt q1, kept a proposed spelling merge pending human approval, and rejected DELETE as non-read-only SQL. A separate two-call receipt probe made the fake model claim an invented total of 999: the real transform hook appended '2 of 3 numbers trace', identified 999 as unsourced and gave verdict needs revision. No external account required. Desktop charts were read in source/docs, not run in a Desktop window; author large-file speed/token benchmarks were not reproduced. Implementation: https://raw.githubusercontent.com/bettersandeep/grimoire/4824773040b131f3179e5896d71d1435908758a7/README.md At equal total, rank 1 over Canvas reflects broader everyday retention for file analysis and answer verification.",
      "tests": "python -I selftest.py passed with DuckDB 2.0.0: its 21 named test groups plus main integration assertions executed, covering file formats, SQL/file-access guards, receipts and arithmetic, audit checks, merge approval, learning, drivers, comparisons, charts and restart behavior. This custom self-test is not a pytest case count. Both the five-call tool flow and the separate numerical-provenance probe completed inside Hermes.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "Read-only data tools, persistent receipts and a real post-answer number check give everyday Hermes analysis a safety net that a prompt cannot enforce.",
      "worst": "Its checker verifies digit-shaped numerical claims rather than the truth of an entire analysis, and the UI and published scale benchmarks remain unverified here.",
      "blocked": null
    }
  ]
}