feat: add web_browse tool for navigating and interacting with web pages using Bun's WebView
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# WebView Tool — Spec
|
||||
|
||||
## TL;DR
|
||||
A `web_browse` tool that uses Bun's built-in WebView to navigate, interact with, and
|
||||
screenshot web pages — no Puppeteer, no Playwright, no separate browser download.
|
||||
|
||||
## Problem
|
||||
The existing `web_fetch` tool only extracts static HTML. SPAs, pages requiring JS
|
||||
execution, form submissions, and interactive testing are impossible. The agent needs
|
||||
a real browser to handle modern web apps.
|
||||
|
||||
## Goal
|
||||
- Navigate to any URL and extract text/HTML after JS execution
|
||||
- Click buttons, fill forms, scroll pages
|
||||
- Take screenshots (save to file, return path)
|
||||
- Evaluate arbitrary JavaScript in the page context
|
||||
- Generate PDFs from pages
|
||||
|
||||
## Architecture
|
||||
|
||||
### Tool: `web_browse`
|
||||
|
||||
Input schema:
|
||||
```ts
|
||||
{
|
||||
url: string; // URL to navigate to
|
||||
action?: 'text' | 'html' | 'screenshot' | 'evaluate' | 'click' | 'type' | 'scroll' | 'pdf';
|
||||
selector?: string; // CSS selector for click/type/scroll
|
||||
text?: string; // text to type (for 'type' action)
|
||||
script?: string; // JS to evaluate (for 'evaluate' action)
|
||||
wait?: number; // ms to wait after navigation (default: 1000)
|
||||
}
|
||||
```
|
||||
|
||||
Default action: `text` — navigate and return page text content.
|
||||
|
||||
### Implementation (src/tools-webview.ts)
|
||||
|
||||
- Uses `Bun.WebView` with Chrome backend (headless)
|
||||
- One view per call (create → navigate → act → close)
|
||||
- Screenshot saved to `os.tmpdir()/shiro-webview-{timestamp}.png`
|
||||
- PDF saved to `os.tmpdir()/shiro-webview-{timestamp}.pdf`
|
||||
- Timeout: 15s per navigation, 5s per action
|
||||
- Error handling: browser not found → suggest install; page error → return error text
|
||||
|
||||
### Registration
|
||||
|
||||
- Add to `tools.ts` as a `net` tool (network access)
|
||||
- Add to `TOOL_META` as 'net'
|
||||
- Add to `TOOL_SETS.net` set
|
||||
- NOT in default tool set (opt-in via toolSets config)
|
||||
|
||||
### Tests
|
||||
|
||||
- Unit: tool schema validation
|
||||
- Integration: navigate to data: URL, verify text extraction
|
||||
- Integration: screenshot saves file
|
||||
- Integration: evaluate JS returns result
|
||||
- Edge case: invalid URL returns error
|
||||
- Edge case: browser not available returns helpful message
|
||||
|
||||
## Files touched
|
||||
|
||||
- **NEW**: `src/tools-webview.ts`
|
||||
- **EDIT**: `src/tools.ts` — register tool, add to TOOL_META and TOOL_SETS
|
||||
- **NEW**: `test/webview.test.ts`
|
||||
- **NEW**: `docs/webview.md`
|
||||
@@ -0,0 +1,83 @@
|
||||
# WebView Tool (web_browse)
|
||||
|
||||
A built-in browser tool powered by Bun's native WebView. No Puppeteer, no Playwright,
|
||||
no separate browser download — just the runtime's own headless browser.
|
||||
|
||||
## What it does
|
||||
|
||||
- **Navigate** to any URL and extract text/HTML after JS execution
|
||||
- **Click** buttons, links, and interactive elements
|
||||
- **Type** into input fields and forms
|
||||
- **Scroll** to elements on the page
|
||||
- **Screenshot** pages as PNG files
|
||||
- **Evaluate** arbitrary JavaScript in the page context
|
||||
|
||||
## When to use it
|
||||
|
||||
Use `web_browse` instead of `web_fetch` when:
|
||||
|
||||
- The page is a SPA (Single Page Application) that needs JavaScript
|
||||
- You need to interact with the page (click, type, scroll)
|
||||
- You need a visual screenshot of the page
|
||||
- `web_fetch` returns incomplete or empty content
|
||||
- You need to fill forms or submit data
|
||||
|
||||
Use `web_fetch` when:
|
||||
|
||||
- The page is static HTML
|
||||
- You just need the text content quickly
|
||||
- No JS execution is needed
|
||||
|
||||
## Actions
|
||||
|
||||
| Action | Description | Requires |
|
||||
|--------|-------------|----------|
|
||||
| `text` | Page text content after JS execution | — |
|
||||
| `html` | Raw HTML of the page | — |
|
||||
| `screenshot` | Save page as PNG, return file path | — |
|
||||
| `evaluate` | Run JavaScript, return result | `script` |
|
||||
| `click` | Click an element | `selector` |
|
||||
| `type` | Type text into an input | `selector` + `text` |
|
||||
| `scroll` | Scroll to an element | `selector` |
|
||||
|
||||
## Examples
|
||||
|
||||
```json
|
||||
// Get page text
|
||||
{ "url": "https://example.com", "action": "text" }
|
||||
|
||||
// Take a screenshot
|
||||
{ "url": "https://example.com", "action": "screenshot" }
|
||||
|
||||
// Click a button
|
||||
{ "url": "https://example.com", "action": "click", "selector": "button.submit" }
|
||||
|
||||
// Fill a form
|
||||
{ "url": "https://example.com", "action": "type", "selector": "#email", "text": "user@example.com" }
|
||||
|
||||
// Run custom JS
|
||||
{ "url": "https://example.com", "action": "evaluate", "script": "document.querySelectorAll('a').length" }
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- **macOS**: Uses system WKWebView (no install needed)
|
||||
- **Linux/Windows**: Requires Chrome, Chromium, Edge, or Brave installed
|
||||
- Bun 1.2+ for WebView support
|
||||
|
||||
## Tool set
|
||||
|
||||
`web_browse` is in the `net` tool set. Enable it in config:
|
||||
|
||||
```json
|
||||
{ "toolSets": ["core", "git", "net"] }
|
||||
```
|
||||
|
||||
Or enable all sets with `toolSets: ["*"]`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Each call creates a fresh browser instance (no persistent sessions by default)
|
||||
- Maximum 15s navigation timeout, 5s action timeout
|
||||
- Screenshots are saved to the system temp directory
|
||||
- Not available in Bun versions before 1.2
|
||||
Reference in New Issue
Block a user