{ "batch": "deep-43", "setup": "Installed real Hermes from source commit fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14; verified plugins install/validate/doctor commands. Used the supplied tryit.py and Nous's FakeLLMServer with empty plans and scripted calls, then continued conversations for fixture-backed paths. Each plugin was fetched at its field.json commit. Scratch-only virtualenv, isolated Hermes homes and loopback proxy bypass; discarded an initial harness run that shared baseline state. Added declared audio libraries and test dependencies. CMI was explicitly selected as context.engine=cmi. Smart Dictation was also exercised through Hermes's actual transcription-provider registry because it intentionally exposes no model tools. AgentLine's own integration script used a local REST/WebSocket stand-in, never the paid service. No actual Desktop application, microphone, phone service or paid image/model provider was used. Tests and calls below are local observations; pinned source URLs in sources.json support code/doc claims. Ranking breaks the three-way 28-point tie by breadth of ongoing Hermes use: CMI, Sheets, then Retouch.", "plugins": [ { "pr": 152421, "name": "sheets", "validate": "Passed catalog admission (16 successful checks), enable and doctor; no reported warnings; four registered tools and bundled grids skill.", "inside": "Loaded in real Hermes. Empty plan completed; tools are deferred rather than included in the first model offer. Created a one-row Task/Owner grid, read it by name, and proposed a Priority column: response correctly said waiting for review, with no applied changes. Duplicate column names and a nonexistent row were rejected cleanly. Explicit sharing OFF blocked list/create. Fresh profile with no sharing.json nevertheless accepted create/read; this is a real discrepancy with the documented default-off gate. Earlier malformed proposals were rejected by Hermes schema validation; corrected calls succeeded. Desktop approval/restore UI, enrichment quality and exports were read/tested, not manually operated. Pinned source: https://github.com/Adolanium/hermes-sheets/tree/65888b7c6a0304d61ac5f1d060a5d4bd95ee73eb.", "tests": "Passed: 14 Python tool tests; 161 Node workspace tests. Covers typed values, proposals, filtering, undo/writeback, formulas and totals. No live research model or Desktop E2E run.", "useful": 5, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 28, "rank": 2, "best": "A local agent spreadsheet makes research and classification useful through typed columns, reviewable proposals and reversible Desktop workflows.", "worst": "A fresh profile allowed chat create/read before any sharing preference existed, although the README says chat access starts off.", "blocked": null }, { "pr": 152976, "name": "agentline", "validate": "Passed catalog admission, enable and doctor; nine tools, one pre_tool_call hook and a platform adapter registered; no reported warnings.", "inside": "Loaded in real Hermes. Empty plan completed; without an API key its nine account tools are intentionally unavailable. Scripted status/messages attempts therefore did not exercise account handlers; the bad 911 call was blocked by the hook. Then ran the pinned integration_hermes.py against actual Hermes and its own local REST/WebSocket stand-in: account status, emergency refusal, unattended call/SMS approval refusal, approved-path call and transcript, allowed inbound SMS, stranger rejection, live caller context, slow-answer SMS, reconnect and disconnect all passed. Approval-off existed only inside that local test. No real number was purchased, texted or called. Pinned source: https://github.com/prakhar-raj-031104/hermes-agentline/tree/6e4c34e61084e5db154a0159bc57440b6ffbceeb.", "tests": "Passed: 38 standard-library unit tests and 26 real-Hermes integration assertions (26 passed, 0 failed); external AgentLine service/voice quality untested.", "useful": 3, "reliable": 5, "polish": 4, "creative": 4, "beyond": 5, "total": 26, "rank": 4, "best": "A real bidirectional phone/SMS platform extends where Hermes can be reached, with demonstrated fail-closed approval and sender controls.", "worst": "A paid external service limited to US and Canadian numbers narrows day-one adoption, and its actual voice quality remains unverified.", "blocked": null }, { "pr": 151371, "name": "smart-dictation", "validate": "Passed catalog admission, enable and doctor with its three declared audio dependencies; no reported warnings; registers a transcription provider and auxiliary correction task, not agent tools.", "inside": "Loaded and completed empty-plan Hermes chat. Tool-search exploration confirmed this is not a callable agent-tool surface; no fabricated dictation tool is counted as a success. With stt.provider=smart_dictation, Hermes registry returned the real provider. Availability returned true after libraries/config were present, but a valid short WAV without cached weights returned stt_unavailable; nonexistent WAV returned transcription_failed; call after provider.close returned invalid_owner. All returned bounded error envelopes. Successful ASR, microphone capture and remote correction were not run; ASR in duration tests is explicitly stubbed. Native/contextual composer, consent and draft-owner fences were inspected in source and covered by supplied UI tests. Pinned source: https://github.com/metamindedu/hermes-smart-dictation/tree/e62b98787d4d056b8d7c333e05a80c6779f9ba2a.", "tests": "Passed after installing declared faster-whisper/av/numpy: 88 Python tests. npm test: 87 passed, 6 skipped. Initial Python run without audio libraries had 3 failures; all disappeared with declared dependencies. No inference or paid correction test.", "useful": 4, "reliable": 4, "polish": 4, "creative": 3, "beyond": 5, "total": 25, "rank": 5, "best": "Native transcription registration plus consent-aware correction and fenced composer insertion improve everyday voice input beyond what a prompt can implement.", "worst": "Fresh installation needs pre-cached model weights and profile preparation, while provider availability can be true even though transcription still reports stt_unavailable.", "blocked": null }, { "pr": 153090, "name": "retouch", "validate": "Passed catalog admission, enable and doctor; four tools and Desktop surface accepted; no reported warnings.", "inside": "Loaded and completed empty-plan Hermes chat. Empty project list and invalid project IDs returned clean results. Seeded a synthetic 100x100 image and ellipse through the plugin Workspace API, then used a real Hermes scripted conversation for retouch_get, exposure +1 adjustment, needs_review reply and PNG export: all succeeded. Edit reported 1893 inside pixels changed and 0 outside; a new version/export was written. Invalid operation explode was rejected by Hermes schema validation before dispatch. The scripted model is nonvision; it did not judge aesthetics. Generative removals and actual Desktop drawing were not exercised. A discarded fixture run used an unresolved test ID; corrected calls above used the real generated project ID. Pinned source: https://github.com/Lyfae/hermes-retouch/tree/1a5ac4346b991754a25a5dc14745f533341c8c95.", "tests": "Passed: 272 tests across tests/ and verify/, including region properties, concurrent edits, version/state transitions, IO hardening and review behavior; 83 Pillow deprecation warnings, no failures.", "useful": 4, "reliable": 5, "polish": 5, "creative": 4, "beyond": 5, "total": 28, "rank": 3, "best": "User-drawn regions, exact outside-mask preservation, per-mark review threads and versioned exports make agent image editing controllable.", "worst": "The verified guarantee covers unchanged outside pixels, while convincing removals and seam-free generated edits still depend on an untested image provider and visual review.", "blocked": null }, { "pr": 151887, "name": "cmi", "validate": "Passed catalog admission, enable and doctor, but validator warns about private core imports including agent.context_compressor._summarize_tool_result, model metadata and pricing helpers; no failed checks.", "inside": "Loaded as a context-engine plugin; default compressor selection alone does not expose its tools. After explicit context.engine=cmi, real Hermes offered cmi_recall and cmi_search. Three scripted calls returned an empty search and clear missing-key errors for unknown/empty recall keys. Additional probe used the engine registered by the real Hermes PluginManager on 83 synthetic history messages: JSON representation shrank from 154231 to 4651 characters, and stored original recall was byte-exact; path-like invalid key was refused without a crash. This synthetic character reduction is not a token, billing or answer-quality benchmark. No claimed README savings percentage was independently established; experimental cross-session project memory was not proven. Pinned source: https://github.com/jeremiebergel/context-manager-intelligence/tree/399468f0687c051476961e4bb045eb84263be6b1/cmi.", "tests": "Passed from the plugin repository: 501 tests, 6 skipped, both standalone-compatible and with HERMES_AGENT_DIR pointing at real Hermes. One host-suite invocation from the workspace root produced 2 subprocess import failures; rerunning from the documented repository directory passed all 501. Also passed real-Hermes engine compression/recall probe.", "useful": 5, "reliable": 4, "polish": 4, "creative": 5, "beyond": 5, "total": 28, "rank": 1, "best": "Deterministic context selection with exact recall changes Hermes itself and can help nearly every long-running session without extra model calls.", "worst": "Catalog rule-9 warnings identify private Hermes imports that create upgrade fragility despite passing admission and current runtime checks.", "blocked": null }, { "pr": 153647, "name": "riah-swipe", "validate": "Passed catalog admission, enable and doctor; four tools and pre_llm_call/pre_gateway_dispatch hooks; validator warns about private tools.interrupt.is_interrupted and tools.transcription_tools.transcribe_audio imports.", "inside": "Loaded and completed empty-plan Hermes chat. Real Hermes published two coffee-cart names, status showed both undecided, and results returned the same two cards; an empty deck was rejected by schema validation. Synthetic reviewer decisions were subsequently persisted through the core API; Hermes reread one revise and one discard with notes, then published round 2 linked by redo_of/replaces (not a manual browser or Telegram swipe). Inspected bundled board, one-time pairing, local-media restrictions and gateway feedback ownership. No actual phone/LAN or Telegram account test. A deterministic fixed-timestamp core probe reproduced reversed note order: decisions on dk-02 then dk-01 came back dk-01 then dk-02 because ties sort by card number. Pinned source: https://github.com/RiahStudio/riah-swipe/tree/513609d47b27a2703a2fd5bcb2fa54ce35d3d966/hermes-plugin.", "tests": "Initial documented unittest runner: 307 tests, 249 passed, 1 failed, 57 skipped without optional Telegram dependency; failure was test_notes_come_back_in_swipe_order_with_how_to_read_them. Focused daily-use rerun passed 29 with 15 skips, so the clock-sensitive defect is intermittent; fixed-timestamp reproduction confirms it. With Hermes-pinned python-telegram-bot installed: 307 tests, 305 passed, 2 failed, no skips; note-order failure recurred, plus test_publish_refuses_a_file_outside expected media_dirs in an error truncated by the long scratch path (the unsafe path was still refused). Initial pytest collection error was a hyphenated-folder import issue; judged using documented unittest runner.", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 4, "total": 25, "rank": 6, "best": "A bundled swipe board turns generated batches into structured keep/cut/revise feedback with notes and repeatable revision rounds.", "worst": "Same-timestamp decisions can reorder feedback notes, and private core imports add upgrade risk to an otherwise polished review workflow.", "blocked": null } ] }