{ "batch": "deep-30", "setup": "Installed real Hermes from a shallow Git source clone at d949b43e92232bc76d239e0864c3b898aa2bb826 with uv sync --locked --python 3.14 (CPython 3.14.10); plugins help listed install, validate and doctor. Used RUN.md/tryit.py and Nous tests/fakes/fake_llm_provider.py with isolated scratch Hermes homes, no external model/account/key. The catalog manager provisioned its pinned Python 3.14.9 environment. Empty plans ran before planned calls for all six. Later runner copies preserved full validator output; Retouch used a seeded sample image and mark because those are normally created in Desktop. Hermes lazily exposes plugin tools through tool_call, so the empty-plan added_by_plugin list was empty even where doctor registered tools. Unit suites ran with Hermes source absent from PYTHONPATH; standalone reruns distinguish tests that assume Hermes is absent. No Desktop window or live model provider was used; browser/WebGL rendering remains unverified, and runtime findings below are local observations, with pinned source URLs in sources.json. An initial archive-based installation was superseded by the completed source clone; scores use the clone.", "plugins": [ { "pr": 154067, "name": "hermes-design-studio", "validate": "Passed: validate --install-deps --json exited 0; 16/16 checks passed, including registrations, Desktop SDK surface and no core override. Security scan CAUTION points to exec-string/security-test and passwd-path strings in committed evidence, not an admission failure.", "inside": "Loaded: enable and doctor exited 0; doctor registered 7 tools and the usage skill. All 5 scripted tool requests returned through real Hermes: catalog found halftone; new created a revision-0 project; create_tool saved a shader draft requiring human approval and returned compilation pending after 10 seconds; preview returned an actionable no-renderer error after 10 seconds. The malformed apply project id was rejected by Hermes schema validation before invoking the handler. Rendering, approval UI and PNG export were not exercised in Desktop.", "tests": "Passed 355 Python tests (8 multiprocessing/fork deprecation warnings) and 55 Node tests after installing scratch-only React/esbuild/jsdom test dependencies; no test failures remained. Browser/WebGL and real Desktop tests not run.", "useful": 3, "reliable": 4, "polish": 4, "creative": 4, "beyond": 5, "rank": 5, "best": "A real revision-checked Desktop image/shader workshop shares state with the agent and enforces human-only approval of generated tools.", "worst": "Its core visual result still needs a connected WebGL2 Desktop renderer, and the source-install documentation remains burdened by local unpublished/fork-specific delivery history.", "blocked": null, "total": 25 }, { "pr": 153090, "name": "retouch", "validate": "Passed: validate --install-deps --json exited 0, all 16 checks passed with no warnings; dependency pillow>=12.3,<14 resolved. Enable and doctor exited 0.", "inside": "Loaded: 4 tools plus usage skill and Desktop pane/API. On a scratch sample photo/mark seeded with the same Workspace API used by the UI, five scripted requests completed: get returned project and image envelope; adjust exposure saved a new version with outside_changed_px=0 and inside_changed_px=2051; reply preserved a needs_review thread and explicit review limitation; export wrote a new PNG. An invalid operation was rejected by Hermes schema validation before invoking edit. The original stayed a separate version. Real Desktop sending/accepting marks and live image-provider generation were not run. Independent decoding confirmed a 128x96 PNG with exactly 2,051 changed pixels, all within the marked rectangle.", "tests": "Passed: 266 Python unit/integration tests, 6 Hypothesis/property checks (including concurrency/state/mask invariants), and the Node scheduler check covering 400 random schedules and 3,337 started sessions; 83 Pillow getdata deprecation warnings in property tests.", "useful": 4, "reliable": 5, "polish": 5, "creative": 4, "beyond": 5, "rank": 3, "best": "Mark-specific agent sessions, pixel-checked regional edits, threaded human review and immutable image versions form a coherent Desktop product.", "worst": "Object replacement depends on an external image provider and sends picture content to configured models, while the end-to-end Desktop interaction remains unverified in this seat.", "blocked": null, "total": 28 }, { "pr": 153809, "name": "living-spaces", "validate": "Passed: all 16 catalog checks, enable and doctor exited 0. Warnings identify private session_key/cron.jobs/cron.executions imports and a raw window.hermesDesktop bridge call; the gate does not fail these, but they are compatibility/reviewer concerns.", "inside": "Loaded: 15 tools, build-space skill, Desktop Spaces route and backend. After correcting my initial plan to supply the required description and integer base_rev=0, all five requests returned: create produced reading-desk revision 1; write_records stored a note; query returned it; edit_file advanced code to revision 2; ../config.yaml read was refused as Invalid project path. Creation accurately warned that canvas bot actions lack native maintaining-session binding in this CLI session. Actual Desktop guest rendering, native bot buttons and cron execution were not run.", "tests": "Passed 395 Python tests plus 1,017 subtests and 261 JavaScript tests. JS required scratch React/jsdom/esbuild/query dependencies, the provided vendored-Preact resolver, experimental VM modules and a generated renderer metafile; after those harness prerequisites no failures remained.", "useful": 5, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "rank": 2, "best": "Persistent Desktop mini-apps share revisioned files and records with chat, with native cron upkeep and recovery tools rather than disposable generated pages.", "worst": "Private cron/session APIs and a raw Desktop bridge add upgrade risk, and the documented Internet Off switch does not promise to prevent WebRTC outbound traffic.", "blocked": null, "total": 28 }, { "pr": 152256, "name": "nvidia-switchyard", "validate": "Passed: validate --install-deps --json exited 0, all 16 checks passed, enable/doctor exited 0. Numerous private-core import warnings include resolve_runtime_provider, provider registry/inventory and Codex header helpers. Manual rule-11 review: providers.py obtains raw provider API/OAuth credentials and builds bearer headers rather than using the catalog-required credentialed_provider_request/requires_auth route for subscription sign-ins.", "inside": "Loaded: 5 hooks, llm_request middleware, switchyard CLI, provider presets and dashboard/Desktop UI; no agent tools. Three scripted terminal calls through Hermes completed: status failed cleanly naming missing SWITCHYARD_CLOUD_API_KEY; list showed vgr preset and tiers; pin sideways returned argparse invalid-choice usage. No personal credentials or live local/cloud models were used. The plugin's own mock-tier tests separately exercised real native sidecar routing in the documented source layout.", "tests": "Passed all 174 distinct Python tests in the documented Hermes/plugins/switchyard layout with pinned nemo-switchyard==0.4.0a1, including real native sidecar routing against scripted local/cloud HTTP tiers. Completed across split runs after a 180-second run limit; all remaining cases passed (last segment: 26 passed). An initial alternate namespace harness caused provider/configuration failures; the documented source layout resolved those. Desktop UI not exercised.", "useful": 4, "reliable": 2, "polish": 5, "creative": 4, "beyond": 5, "rank": 4, "best": "Verification-gated local/cloud routing, provider presets and visible escalation traces change how Hermes uses the user's own models rather than merely adding a prompt.", "worst": "Subscription-token handling conflicts with catalog rule 11, and the default local decision budget can delay a request for 30 minutes while cloud failure may return an unchecked local answer.", "blocked": null, "total": 25 }, { "pr": 153714, "name": "tenure", "validate": "Passed: validate --install-deps --json, enable and doctor exited 0; 16/16 checks passed. Scanner CAUTION matches classifier/UI strings; review warnings flag inventory, plugins_settings.save_plugin_settings and agent.i18n private imports.", "inside": "Loaded: 7 hooks, tenure CLI/slash command and dashboard, no new tools. Four scripted core-tool requests exercised the hooks: a local note write succeeded with verification; reading it succeeded; a later malformed-JSON write was blocked for human confirmation after outside-content suspension, with no unattended hang; reading a missing file returned a normal not-found error. The bundled real-Hermes integration also passed its trusted-start approval/undo/probation/passport checks, reporting an intact 32-entry evidence chain.", "tests": "Hermes environment: 2,402 passed, 2 skipped, 5 failed (4 tests assume Hermes APIs are absent; one cold-passport timing assertion measured about 1.15s against 0.30s). All 4 host-absence assertions passed in isolation; the full standalone suite passed 2,407 with 2 skipped using the documented TENURE_PERF_SLACK=5 for cold-passport timing. Bundled real-Hermes integration: ALL CHECKS PASSED. Integration recorded 30 passing assertions.", "useful": 5, "reliable": 4, "polish": 5, "creative": 5, "beyond": 5, "rank": 1, "best": "Revocable per-action trust, real undo snapshots, evidence-based probation and human approval hooks improve the user's own agent across everyday tasks.", "worst": "Its large static shell classifier is explicitly not a sandbox, and private API dependencies plus the failed default cold-passport latency budget limit reliability confidence.", "blocked": null, "total": 29 }, { "pr": 152243, "name": "morph", "validate": "Passed: full validate --install-deps --json exited 0, all 16 checks passed without admission warnings. Doctor consistently registered 7 tools and 3 hooks. Initial empty-plan enable succeeded; later fresh-home enable repeatedly failed in Hermes PM with missing editable plugin-sources distribution metadata, including at the canonical morph directory name.", "inside": "Initial pinned install loaded. Later fresh homes did not enable: direct planned tool requests were unavailable and the scripted chat ended on an invalid tool call. Reusing the original successfully enabled home with three shipped offline HTTP demo services, five real-Hermes requests succeeded: discover found service schemas; plan_workspace made revision 1 using an explicitly reported deterministic fallback when the scripted model supplied no usable plan JSON; inspect read the source order; propose created a pending, applied=false review action; an unknown operation returned permission_denied. No approval token or approve tool was exposed. Fresh setup failure caps Reliable at 2 despite the successful reused installation.", "tests": "281 Python unit/integration/security tests passed in a standalone scratch environment, including real local HTTP apps and approval/read-back tests. In the Hermes Python environment 280 passed and 1 outside-Hermes credential-helper assertion failed; that assertion passed in isolation. Browser suite not run.", "useful": 4, "reliable": 2, "polish": 5, "creative": 4, "beyond": 5, "rank": 6, "best": "Schema-derived cross-application workspaces combine useful reads with proposals, human-only approvals and verified read-back instead of handing the agent unrestricted writes.", "worst": "Fresh enable repeatedly failed in the supplied Hermes dependency-publication workflow despite a passing catalog check, so the working reused installation does not establish reliable setup.", "blocked": null, "total": 25 } ] }