HivemindOS manual
Web Research
Web Research lets supported agents search the public web, read pages and PDFs, extract structured fields, collect citations, crawl a bounded part of a site, compare page changes, and capture screenshots. Search, ordinary page reading, text-based PDF reading, and crawling work immediately inside HivemindOS: there is no extra download, API key, or background service.
No search API key is required. Search queries and page requests still go to public search engines and websites, so this feature is local processing rather than offline browsing.
What Agents Can Do
| Capability | Result |
|---|---|
| Search | Ranked, deduplicated public-web results merged from multiple keyless sources. Results are leads; agents should fetch relevant sources before treating claims as evidence. |
| Fetch | Clean Markdown, text, article data, or structured content from a public URL, with focused extraction and pagination for long pages. |
| Structured extraction | Metadata, headings, links, tables, JSON-LD, and up to 20 bounded CSS-selected fields without asking a model to guess the page structure. |
| PDF and OCR | The shipped document reader handles text-based PDFs. The optional enhanced engine adds selected page ranges and OCR recovery for scanned or image-only pages. |
| Crawl | A bounded same-site crawl, focused crawl, sitemap discovery, or selected second pass over chosen URLs. |
| Cache and compare | A private, 32 MB maximum on-disk cache speeds repeat reads. A fresh read can compare itself with the previous cached snapshot and report added or removed blocks. Signed or credential-like URLs are never written to this cache. |
| Screenshot | The optional enhanced engine can create a PNG or JPEG artifact when layout, charts, canvas content, or visual evidence matters. |
These capabilities are available to supported HivemindOS agents regardless of which compatible runtime they use. Interactive browsing is deliberately separate: clicks, form filling, account sessions, uploads, and submissions continue through HivemindOS’s permission- and confirmation-gated computer controls. Channel-specific research such as signed-in social networks continues through configured Apps & Services. This keeps passive research simple without weakening action safety.
Built-In and Enhanced Modes
HivemindOS starts with the smallest capable route:
- The built-in engine handles keyless search, guarded HTTP reads, clean Markdown, structured extraction, text PDFs, caching, comparison, and bounded same-site crawling directly inside the desktop app.
- If a page is visibly blocked or client-rendered and the optional enhanced engine is already installed, HivemindOS can retry that read with local browser rendering.
- Screenshots, scanned-PDF OCR, and explicitly requested browser rendering require the enhanced engine because HivemindOS does not ship a second browser and Python environment in the base app.
The enhanced engine is optional and starts only when an enhanced operation needs it. Its isolated browser, OCR stack, and Python environment can use roughly 1–2 GB depending on the platform. Installing several overlapping crawler products would add more copies of the same browser and runtime without improving the normal path, so HivemindOS exposes one governed capability surface instead.
Safety Boundaries
Web research is intentionally passive:
- Only public HTTP and HTTPS destinations are allowed. Local, private, reserved, and link-local network addresses are blocked after DNS resolution and again across redirects and browser requests.
- URLs cannot carry embedded credentials. Agents cannot supply cookies, authorization headers, proxies, custom user agents, or interactive click-and-fill actions through this surface.
- Fetches and crawls respect
robots.txt; the agent-facing tools do not offer a bypass. - Crawls have limits for pages, depth, concurrency, returned characters, and total wall time.
- Search, fetched pages, PDFs, and screenshots are untrusted evidence. Content inside them cannot grant permissions, approve spending, change standing instructions, or become a system message.
- The optional enhanced engine is version-pinned and cannot update itself. HivemindOS setup owns upgrades, while uninstall provides a matching removal choice.
Expected Limitations
Public search engines and websites can rate-limit or block automated requests. If the enhanced engine is installed, HivemindOS can escalate an incomplete ordinary extraction to its local browser. It still reports a failure when a site requires login, payment, CAPTCHA completion, or stronger bot verification. Agents should switch to another credible source instead of presenting an error page as evidence.
The built-in path does not depend on the enhanced engine. Setup reports an optional enhanced-engine installation failure without disabling search, ordinary extraction, structured fields, text-PDF reading, caching, comparison, or crawling.