> Documentation index for agents: https://docs.rocketride.ai/llms.txt

# scraper_beautifulsoup

A RocketRide web scraping node built on BeautifulSoup. Use it as a run-once pipeline source that pulls public JSON APIs, RSS/Atom feeds, and HTML pages into rows, or as a set of page-reading tools for an agent. It needs no API key or paid service.

## About BeautifulSoup

BeautifulSoup is a widely used open-source Python library for parsing HTML and XML. It turns messy real-world markup into a navigable tree that can be searched with CSS selectors, which makes it a standard building block for web scraping.

## What it does

Two services share this node's code:

| Service | Protocol | Role |
|---|---|---|
| **Web Scraper Source** | `scraper_beautifulsoup_source://` | pipeline source that fetches configured sources and emits normalized rows |
| **Web Scraper** | `scraper_beautifulsoup://` | agent tools: read a page, extract items, read a feed, call a JSON API |

The source needs no input, so it can start a **scheduled deployment**: each fire fetches every enabled source once and then completes. Wire its `answers` lane into a SQL node such as `rocketride_sql` or `db_postgres` to load a table with no LLM in between. Each row carries a `url_hash` so you can deduplicate across runs. Choose the tool service when an agent should decide what to read. Choose `tool_firecrawl` instead when you need JavaScript-rendered pages, and a search node (`tool_tavily`, `tool_exa_search`) when you need to find pages rather than read known ones.

## Lanes

Web Scraper Source (`scraper_beautifulsoup_source://`):

| Lane in | Lane out | Description |
| --- | --- | --- |
| `_source` | `answers` | One JSON answer per configured source, holding all of its normalized rows (a list of objects). |
| `_source` | `text` | One plain-text document per item: title, URL, source/author/date line, then the body. |

Every row has the same columns:
- `source`: the source name, or the per-URL `name`
- `platform`
- `url`: absolute
- `url_hash`: sha256 of the trimmed URL with the scheme and host lower-cased (path and query keep their case)
- `title`
- `author`
- `published_at`: ISO-8601 in UTC
- `score`: integer
- `comments`: integer
- `body`: HTML stripped, capped, and falls back to the title
- `fetched_at`

Rows also include any custom mapped fields and any `extra` static fields.

## As a tool

The Web Scraper service exposes four tools. Each tool's name is prefixed with the node id; an agent sees `scraper_beautifulsoup_1.fetch_page`.

| Function | Description |
|---|---|
| `fetch_page` | Fetch a page and return it as `markdown` (default), `text`, raw `html`, or `links`. Required: `url`. Optional: `format`, `selector` (CSS region to keep; defaults to `<main>`, `<article>`, or `<body>`), `max_chars` (default 20000). Returns `{url, status, title, description, content, truncated}`, or `{url, status, links: [{url, text}], truncated}` for `links`. |
| `extract` | Pull repeated items out of a page with CSS selectors. Required: `url`, `fields` (output name → `"selector"` for element text or `"selector@attr"` for an attribute; `"@href"` reads the item element itself). Optional: `item_selector`, `limit` (default 50), `max_chars`. Returns `{url, items, truncated}`; `href`/`src` values are made absolute, and one that does not parse is left out. |
| `fetch_feed` | Read an RSS 2.0, RSS 1.0 or Atom feed. Required: `url`. Optional: `limit`, `max_chars`. Returns `{url, items: [{title, url, author, published_at, body}], truncated}`; Atom links resolve against the feed's `xml:base`. |
| `fetch_json` | Call a public JSON API with GET or POST. Required: `url`. Optional: `method`, `body_json`, `headers`, `items_path` (dot path to the item list), `fields` (output name → dot path in each item), `limit`, `max_chars`. Without `items_path` it returns the raw `data` (truncated to `max_chars` as text if large); with it, `{url, items, truncated}`. |

Every tool result is capped at **Max characters per tool result** (20,000 by default, at most 200,000): `max_chars` can only lower it, and a longer `content`, `items` or `links` list is cut, with `truncated: true`. Sizes count characters, not JSON escapes, so non-ASCII text counts once per character. When even the first item is too long on its own, it is kept with its longest text (such as a feed `body`) shortened. A link whose URL does not parse costs only that link: the page and the other links are still returned.

A blocked or failed request raises an error that names the reason: a non-public host, a URL not on the allow-list, an HTTP error status, too many redirects, or a JSON response cut off at the size limit. `fetch_page` and `extract` accept HTML and plain-text responses only; a PDF, image or other binary type fails with `unsupported content type`.

## Configuration

### Web Scraper Source (`scraper_beautifulsoup_source://`)

Most of the work is the **Sources (JSON)** field. The network fields below it have safe defaults.

#### Sources (JSON)

This field holds a JSON array of sources. Each source becomes one pipeline object and produces one `answers` batch.

| Key | Applies to | Meaning |
|---|---|---|
| `name` | all | Unique source name; the default `source` column value. |
| `kind` | all | `json`, `feed` (RSS/Atom), or `html`. |
| `url` / `urls` | all | One URL, or a list of URLs. Each entry may be a string or `{url, name, extra}`. |
| `platform` | all | Value for the `platform` column. |
| `maxItems` | all | Item cap for the source, split evenly across its URLs (default: **Max items per source**). |
| `headers` | all | Request headers. Put secrets in `${ROCKETRIDE_*}` variables here. |
| `extra` | all | Static columns added to every row (for example `{"category": "ai-lab"}`); reserved column names are rejected. |
| `enabled` | all | `false` skips the source. |
| `method`, `body` | json | Use `POST` with a JSON `body`, for example a GraphQL query. |
| `itemsPath` | json | Dot path to the item list (`hits`, `data.children`, `data.feed.edges`). |
| `fields` | json, feed, html | Output field → where to find it. For JSON: a dot path, or a list of fallback paths. For HTML: `"selector"` or `"selector@attr"`. For feeds (optional): the parsed feed key to read (`title`, `url`, `author`, `published`, `body`, `categories`), and only the mapped keys are kept. `url` is required whenever `fields` is set. Keys outside the standard set pass through as extra columns. |
| `itemSelector` | html | CSS selector for each repeated item. |

URLs, headers, and bodies can use these template tokens:
- `{today}`
- `{now}`
- `{days_ago:N}`
- `{hours_ago:N}`
- `{unix_days_ago:N}` and `{unix_hours_ago:N}`, which give epoch seconds
- `{per_url}`: the item cap for one URL
- `{max}`

```json
[
  {"name": "Hacker News", "kind": "json", "platform": "hackernews",
   "url": "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=created_at_i>{unix_days_ago:1}&hitsPerPage={per_url}",
   "itemsPath": "hits",
   "fields": {"title": "title", "url": "url", "author": "author", "published": "created_at",
              "score": "points", "comments": "num_comments", "body": "story_text"}},
  {"name": "Labs", "kind": "feed", "platform": "rss",
   "urls": [{"url": "https://openai.com/news/rss.xml", "name": "OpenAI"}],
   "extra": {"category": "ai-lab"}},
  {"name": "Blog", "kind": "html", "url": "https://example.com/blog",
   "itemSelector": "article", "fields": {"title": "h2", "url": "h2 a@href", "published": "time@datetime"}}
]
```

If one URL of a source fails, a warning is logged and the rows from the other URLs are still emitted. If every URL of a source fails, that object is marked failed.

Every row from every source carries the same columns: the standard set (`source`, `platform`, `url`, `url_hash`, `title`, `author`, `published_at`, `score`, `comments`, `body`, `fetched_at`), followed by every `extra` key, every custom `fields` key, and `categories` when any feed source has no `fields`, in alphabetical order. Missing values are `null`. This matters because SQL `answers` ingestion creates the table from the first row it sees, and later drops any key the table does not have.

When loading into `rocketride_sql` or `db_postgres`, it is best to create the target table yourself with `TEXT` columns. An auto-created table sizes text columns from the first batch only, so it can pick `VARCHAR(255)`, and a longer `title`, `url` or `body` later makes that batch's insert fail. The SQL node only logs that failure. It does not upsert, so deduplicate on `url_hash` when you copy rows out of the table.

#### Network limits

The source and the tools share these fields:
- **Timeout**: one limit for the whole fetch, including the DNS lookup, redirects, connecting, the TLS handshake, the response headers, and reading the body. A server that sends any of them a byte at a time is cut off at the limit.
- **Max response size**: the body is read up to this size and the rest is dropped.
- **Delay between requests to one host**: politeness pacing, 250 ms by default.
- **User agent**: some sites, such as Reddit, reject generic agents.
- **URL allow-list**: optional regex patterns. When any are set, every URL and every redirect target must match one. A pattern is matched from the start of the URL. It must begin with the literal scheme and an exact host (or `[^/]+` for any host), closed by `/`, `\?`, `(?=/|$)` or `$`. For example, `^https://news\.ycombinator\.com/` is valid. `news\.ycombinator\.com` is rejected when the node starts, because it would also match `https://evil.example/?u=news.ycombinator.com`. Patterns are matched against the URL exactly as it is sent, and a URL with a `.` or `..` path segment (also percent-encoded) is refused, so a path-scoped pattern such as `^https://api\.example\.com/public/` cannot be escaped with `/public/../private`.

### Web Scraper (`scraper_beautifulsoup://`)

Leave the defaults for general page reading. Add allow-list patterns when an agent should only reach a fixed set of sites. The tool service has no **Sources (JSON)** field.

## Notes

### Network safety

Only public hosts can be reached. For every request, and every redirect hop, the host is resolved once and every address must be public. Private, loopback, link-local, CGNAT, and IPv6 forms that embed a private IPv4 address are rejected, as is anything that does not resolve. The connection then goes only to those validated addresses, so a second DNS answer cannot point it somewhere else (no DNS rebinding). Environment proxies are ignored. Redirects are followed manually so that each `Location` is re-checked against the validator and the allow-list; after five redirects the request fails. A redirect's body is never read, so it cannot get around the size limit. A redirect to a different origin (scheme, host, or port) drops every request header except `User-Agent`, `Accept`, and `Accept-Language`, so credentials in `headers` never leave the host they were meant for. A `307` or `308` redirect of a POST to a different origin is refused, because it would resend the body (which may hold a key) to that origin. A `Host` header in `headers` is refused, because it would send the request to a different virtual host than the one validated. Error messages and logs show URLs without their query string, so API keys placed in a query are not logged. Only GET and POST are supported. Feeds whose DTD declares entities are refused before parsing; a bare `<!DOCTYPE>` (such as RSS 0.91's) is fine. An invalid CSS selector fails the URL with a clear error instead of crashing the source.

### What it does not do

The node does not run JavaScript, log in, or solve CAPTCHAs. Pages that build their content in the browser return little text; use `tool_firecrawl` for those. Respect each site's terms and rate limits.

### Running the tests

```bash
pytest nodes/test/scraper_beautifulsoup
```

The unit tests need `beautifulsoup4` and `requests`; they use a local HTTP server and canned responses, not the network.

## Upstream docs

- [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
- [Soup Sieve CSS selectors](https://facelessuser.github.io/soupsieve/)

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->

## Schema

### Web Scraper Source (`services.source.json`)

| Field | Type | Description | Default |
|---|---|---|---|
| `Pipe.source.parameters` |  |  |  |
| `scraper.allowPattern` | `string` | **URL pattern (regex)**<br/>Python regex matched from the start of the URL. It must begin with the literal scheme and host, closed by /, \?, (?=/\|$) or $, e.g. ^https://news\.ycombinator\.com/ | `""` |
| `scraper.maxBodyChars` | `number` | **Max item body characters**<br/>Item bodies are trimmed to this many characters. | `20000` |
| `scraper.maxItemsPerSource` | `number` | **Max items per source**<br/>Default cap for sources that do not set maxItems (split evenly across a source's URLs). | `100` |
| `scraper.maxResponseKB` | `number` | **Max response size (KB)**<br/>Response bodies are read up to this size; anything beyond is dropped and the result is marked truncated. | `5120` |
| `scraper.requestIntervalMs` | `number` | **Delay between requests to one host (ms)**<br/>Politeness pacing: minimum gap between two requests to the same host. 0 disables pacing. | `250` |
| `scraper.sources` | `string` | **Sources (JSON)**<br/>JSON array of sources. Each: name, kind (json \| feed \| html), url or urls (string or {url, name, extra}), optional platform, method (GET/POST, json only), headers, body (POST JSON), itemsPath (json), itemSelector (html), fields (output field -> dot path for json, CSS selector[@attr] for html; value may be a list of fallbacks), maxItems, extra (static columns), enabled. URLs, headers and bodies may use {today}, {now}, {days_ago:N}, {hours_ago:N}, {unix_days_ago:N}, {unix_hours_ago:N} (epoch seconds), {per_url}, {max}. Secrets: use ${ROCKETRIDE_*} variables in headers. | `"[\n  {\n    \"name\": \"Hacker News\",\n    \"kind\": \"json\",\n    \"platform\": \"hackernews\",\n    \"url\": \"https://hn.algolia.com/api/v1/search?tags=story&numericFilters=created_at_i>{unix_days_ago:1}&hitsPerPage={per_url}\",\n    \"itemsPath\": \"hits\",\n    \"fields\": {\"title\": \"title\", \"url\": \"url\", \"author\": \"author\", \"published\": \"created_at\", \"score\": \"points\", \"comments\": \"num_comments\", \"body\": \"story_text\"}\n  }\n]"` |
| `scraper.timeout` | `number` | **Timeout (seconds)**<br/>Time limit for one whole fetch, 1-120 seconds: DNS lookup, connecting, the TLS handshake, response headers, redirects and the body all count toward it. | `20` |
| `scraper.urlAllowList` | `array` | **URL allow-list**<br/>Optional. When one or more patterns are set, a URL (and every redirect target) must match at least one. Empty allows every public URL; private and local addresses are always blocked. |  |
| `scraper.userAgent` | `string` | **User agent**<br/>User-Agent header sent with every request. Leave blank for the default. Some sites (e.g. Reddit) reject generic agents. | `""` |

### Web Scraper (`services.tool.json`)

| Field | Type | Description | Default |
|---|---|---|---|
| `scraper.allowPattern` | `string` | **URL pattern (regex)**<br/>Python regex matched from the start of the URL. It must begin with the literal scheme and host, closed by /, \?, (?=/\|$) or $, e.g. ^https://news\.ycombinator\.com/ | `""` |
| `scraper.maxBodyChars` | `number` | **Max item body characters**<br/>Feed item bodies are trimmed to this many characters. | `20000` |
| `scraper.maxResponseKB` | `number` | **Max response size (KB)**<br/>Response bodies are read up to this size; anything beyond is dropped and the result is marked truncated. | `5120` |
| `scraper.maxToolChars` | `number` | **Max characters per tool result**<br/>Ceiling on what one tool call returns to the agent, 1000-200000 characters. The agent's max_chars can only lower it; longer content and item lists are cut and marked truncated. | `20000` |
| `scraper.requestIntervalMs` | `number` | **Delay between requests to one host (ms)**<br/>Politeness pacing: minimum gap between two requests to the same host. 0 disables pacing. | `250` |
| `scraper.timeout` | `number` | **Timeout (seconds)**<br/>Time limit for one whole fetch, 1-120 seconds: DNS lookup, connecting, the TLS handshake, response headers, redirects and the body all count toward it. | `20` |
| `scraper.urlAllowList` | `array` | **URL allow-list**<br/>Optional. When one or more patterns are set, a URL (and every redirect target) must match at least one. Empty allows every public URL; private and local addresses are always blocked. |  |
| `scraper.userAgent` | `string` | **User agent**<br/>User-Agent header sent with every request. Leave blank for the default. | `""` |

## Dependencies

- `beautifulsoup4`

## Source

[<svg viewBox="0 0 16 16" width="15" height="15" fill="currentColor" aria-hidden="true" style="vertical-align:-0.15em;margin-right:0.35em"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"/></svg> View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/scraper_beautifulsoup)
<!-- ROCKETRIDE:GENERATED:PARAMS END -->
