Skip to main content
View source

Web Scraper Source

View as Markdown

A RocketRide web scraping node built on BeautifulSoup. Use it as a run-once pipeline source that pulls public JSON APIs, RSS/Atom feeds, and HTML pages into rows, or as a set of page-reading tools for an agent. It needs no API key or paid service.

About BeautifulSoup​

BeautifulSoup is a widely used open-source Python library for parsing HTML and XML. It turns messy real-world markup into a navigable tree that can be searched with CSS selectors, which makes it a standard building block for web scraping.

What it does​

Two services share this node's code:

ServiceProtocolRole
Web Scraper Sourcescraper_beautifulsoup_source://pipeline source that fetches configured sources and emits normalized rows
Web Scraperscraper_beautifulsoup://agent tools: read a page, extract items, read a feed, call a JSON API

The source needs no input, so it can start a scheduled deployment: each fire fetches every enabled source once and then completes. Wire its answers lane into a SQL node such as rocketride_sql or db_postgres to load a table with no LLM in between. Each row carries a url_hash so you can deduplicate across runs. Choose the tool service when an agent should decide what to read. Choose tool_firecrawl instead when you need JavaScript-rendered pages, and a search node (tool_tavily, tool_exa_search) when you need to find pages rather than read known ones.

Lanes​

Web Scraper Source (scraper_beautifulsoup_source://):

Lane inLane outDescription
_sourceanswersOne JSON answer per configured source, holding all of its normalized rows (a list of objects).
_sourcetextOne plain-text document per item: title, URL, source/author/date line, then the body.

Every row has the same columns:

  • source: the source name, or the per-URL name
  • platform
  • url: absolute
  • url_hash: sha256 of the trimmed URL with the scheme and host lower-cased (path and query keep their case)
  • title
  • author
  • published_at: ISO-8601 in UTC
  • score: integer
  • comments: integer
  • body: HTML stripped, capped, and falls back to the title
  • fetched_at

Rows also include any custom mapped fields and any extra static fields.

As a tool​

The Web Scraper service exposes four tools. Each tool's name is prefixed with the node id; an agent sees scraper_beautifulsoup_1.fetch_page.

FunctionDescription
fetch_pageFetch a page and return it as markdown (default), text, raw html, or links. Required: url. Optional: format, selector (CSS region to keep; defaults to <main>, <article>, or <body>), max_chars (default 20000). Returns {url, status, title, description, content, truncated}, or {url, status, links: [{url, text}], truncated} for links.
extractPull repeated items out of a page with CSS selectors. Required: url, fields (output name → "selector" for element text or "selector@attr" for an attribute; "@href" reads the item element itself). Optional: item_selector, limit (default 50), max_chars. Returns {url, items, truncated}; href/src values are made absolute, and one that does not parse is left out.
fetch_feedRead an RSS 2.0, RSS 1.0 or Atom feed. Required: url. Optional: limit, max_chars. Returns {url, items: [{title, url, author, published_at, body}], truncated}; Atom links resolve against the feed's xml:base.
fetch_jsonCall a public JSON API with GET or POST. Required: url. Optional: method, body_json, headers, items_path (dot path to the item list), fields (output name → dot path in each item), limit, max_chars. Without items_path it returns the raw data (truncated to max_chars as text if large); with it, {url, items, truncated}.

Every tool result is capped at Max characters per tool result (20,000 by default, at most 200,000): max_chars can only lower it, and a longer content, items or links list is cut, with truncated: true. Sizes count characters, not JSON escapes, so non-ASCII text counts once per character. When even the first item is too long on its own, it is kept with its longest text (such as a feed body) shortened. A link whose URL does not parse costs only that link: the page and the other links are still returned.

A blocked or failed request raises an error that names the reason: a non-public host, a URL not on the allow-list, an HTTP error status, too many redirects, or a JSON response cut off at the size limit. fetch_page and extract accept HTML and plain-text responses only; a PDF, image or other binary type fails with unsupported content type.

Configuration​

Web Scraper Source (scraper_beautifulsoup_source://)​

Most of the work is the Sources (JSON) field. The network fields below it have safe defaults.

Sources (JSON)​

This field holds a JSON array of sources. Each source becomes one pipeline object and produces one answers batch.

KeyApplies toMeaning
nameallUnique source name; the default source column value.
kindalljson, feed (RSS/Atom), or html.
url / urlsallOne URL, or a list of URLs. Each entry may be a string or {url, name, extra}.
platformallValue for the platform column.
maxItemsallItem cap for the source, split evenly across its URLs (default: Max items per source).
headersallRequest headers. Put secrets in ${ROCKETRIDE_*} variables here.
extraallStatic columns added to every row (for example {"category": "ai-lab"}); reserved column names are rejected.
enabledallfalse skips the source.
method, bodyjsonUse POST with a JSON body, for example a GraphQL query.
itemsPathjsonDot path to the item list (hits, data.children, data.feed.edges).
fieldsjson, feed, htmlOutput field → where to find it. For JSON: a dot path, or a list of fallback paths. For HTML: "selector" or "selector@attr". For feeds (optional): the parsed feed key to read (title, url, author, published, body, categories), and only the mapped keys are kept. url is required whenever fields is set. Keys outside the standard set pass through as extra columns.
itemSelectorhtmlCSS selector for each repeated item.

URLs, headers, and bodies can use these template tokens:

  • {today}
  • {now}
  • {days_ago:N}
  • {hours_ago:N}
  • {unix_days_ago:N} and {unix_hours_ago:N}, which give epoch seconds
  • {per_url}: the item cap for one URL
  • {max}
[
{"name": "Hacker News", "kind": "json", "platform": "hackernews",
"url": "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=created_at_i>{unix_days_ago:1}&hitsPerPage={per_url}",
"itemsPath": "hits",
"fields": {"title": "title", "url": "url", "author": "author", "published": "created_at",
"score": "points", "comments": "num_comments", "body": "story_text"}},
{"name": "Labs", "kind": "feed", "platform": "rss",
"urls": [{"url": "https://openai.com/news/rss.xml", "name": "OpenAI"}],
"extra": {"category": "ai-lab"}},
{"name": "Blog", "kind": "html", "url": "https://example.com/blog",
"itemSelector": "article", "fields": {"title": "h2", "url": "h2 a@href", "published": "time@datetime"}}
]

If one URL of a source fails, a warning is logged and the rows from the other URLs are still emitted. If every URL of a source fails, that object is marked failed.

Every row from every source carries the same columns: the standard set (source, platform, url, url_hash, title, author, published_at, score, comments, body, fetched_at), followed by every extra key, every custom fields key, and categories when any feed source has no fields, in alphabetical order. Missing values are null. This matters because SQL answers ingestion creates the table from the first row it sees, and later drops any key the table does not have.

When loading into rocketride_sql or db_postgres, it is best to create the target table yourself with TEXT columns. An auto-created table sizes text columns from the first batch only, so it can pick VARCHAR(255), and a longer title, url or body later makes that batch's insert fail. The SQL node only logs that failure. It does not upsert, so deduplicate on url_hash when you copy rows out of the table.

Network limits​

The source and the tools share these fields:

  • Timeout: one limit for the whole fetch, including the DNS lookup, redirects, connecting, the TLS handshake, the response headers, and reading the body. A server that sends any of them a byte at a time is cut off at the limit.
  • Max response size: the body is read up to this size and the rest is dropped.
  • Delay between requests to one host: politeness pacing, 250 ms by default.
  • User agent: some sites, such as Reddit, reject generic agents.
  • URL allow-list: optional regex patterns. When any are set, every URL and every redirect target must match one. A pattern is matched from the start of the URL. It must begin with the literal scheme and an exact host (or [^/]+ for any host), closed by /, \?, (?=/|$) or $. For example, ^https://news\.ycombinator\.com/ is valid. news\.ycombinator\.com is rejected when the node starts, because it would also match https://evil.example/?u=news.ycombinator.com. Patterns are matched against the URL exactly as it is sent, and a URL with a . or .. path segment (also percent-encoded) is refused, so a path-scoped pattern such as ^https://api\.example\.com/public/ cannot be escaped with /public/../private.

Web Scraper (scraper_beautifulsoup://)​

Leave the defaults for general page reading. Add allow-list patterns when an agent should only reach a fixed set of sites. The tool service has no Sources (JSON) field.

Notes​

Network safety​

Only public hosts can be reached. For every request, and every redirect hop, the host is resolved once and every address must be public. Private, loopback, link-local, CGNAT, and IPv6 forms that embed a private IPv4 address are rejected, as is anything that does not resolve. The connection then goes only to those validated addresses, so a second DNS answer cannot point it somewhere else (no DNS rebinding). Environment proxies are ignored. Redirects are followed manually so that each Location is re-checked against the validator and the allow-list; after five redirects the request fails. A redirect's body is never read, so it cannot get around the size limit. A redirect to a different origin (scheme, host, or port) drops every request header except User-Agent, Accept, and Accept-Language, so credentials in headers never leave the host they were meant for. A 307 or 308 redirect of a POST to a different origin is refused, because it would resend the body (which may hold a key) to that origin. A Host header in headers is refused, because it would send the request to a different virtual host than the one validated. Error messages and logs show URLs without their query string, so API keys placed in a query are not logged. Only GET and POST are supported. Feeds whose DTD declares entities are refused before parsing; a bare <!DOCTYPE> (such as RSS 0.91's) is fine. An invalid CSS selector fails the URL with a clear error instead of crashing the source.

What it does not do​

The node does not run JavaScript, log in, or solve CAPTCHAs. Pages that build their content in the browser return little text; use tool_firecrawl for those. Respect each site's terms and rate limits.

Running the tests​

pytest nodes/test/scraper_beautifulsoup

The unit tests need beautifulsoup4 and requests; they use a local HTTP server and canned responses, not the network.

Upstream docs​

Schema​

Web Scraper Source (services.source.json)​

FieldTypeDescriptionDefault
Pipe.source.parameters
scraper.allowPatternstringURL pattern (regex)
Python regex matched from the start of the URL. It must begin with the literal scheme and host, closed by /, ?, (?=/|$) or $, e.g. ^https://news\.ycombinator\.com/
""
scraper.maxBodyCharsnumberMax item body characters
Item bodies are trimmed to this many characters.
20000
scraper.maxItemsPerSourcenumberMax items per source
Default cap for sources that do not set maxItems (split evenly across a source's URLs).
100
scraper.maxResponseKBnumberMax response size (KB)
Response bodies are read up to this size; anything beyond is dropped and the result is marked truncated.
5120
scraper.requestIntervalMsnumberDelay between requests to one host (ms)
Politeness pacing: minimum gap between two requests to the same host. 0 disables pacing.
250
scraper.sourcesstringSources (JSON)
JSON array of sources. Each: name, kind (json | feed | html), url or urls (string or {url, name, extra}), optional platform, method (GET/POST, json only), headers, body (POST JSON), itemsPath (json), itemSelector (html), fields (output field -> dot path for json, CSS selector[@attr] for html; value may be a list of fallbacks), maxItems, extra (static columns), enabled. URLs, headers and bodies may use {today}, {now}, {days_ago:N}, {hours_ago:N}, {unix_days_ago:N}, {unix_hours_ago:N} (epoch seconds), {per_url}, {max}. Secrets: use ${ROCKETRIDE_*} variables in headers.
"[\n {\n \"name\": \"Hacker News\",\n \"kind\": \"json\",\n \"platform\": \"hackernews\",\n \"url\": \"https://hn.algolia.com/api/v1/search?tags=story&numericFilters=created_at_i>{unix_days_ago:1}&hitsPerPage={per_url}\",\n \"itemsPath\": \"hits\",\n \"fields\": {\"title\": \"title\", \"url\": \"url\", \"author\": \"author\", \"published\": \"created_at\", \"score\": \"points\", \"comments\": \"num_comments\", \"body\": \"story_text\"}\n }\n]"
scraper.timeoutnumberTimeout (seconds)
Time limit for one whole fetch, 1-120 seconds: DNS lookup, connecting, the TLS handshake, response headers, redirects and the body all count toward it.
20
scraper.urlAllowListarrayURL allow-list
Optional. When one or more patterns are set, a URL (and every redirect target) must match at least one. Empty allows every public URL; private and local addresses are always blocked.
scraper.userAgentstringUser agent
User-Agent header sent with every request. Leave blank for the default. Some sites (e.g. Reddit) reject generic agents.
""

Web Scraper (services.tool.json)​

FieldTypeDescriptionDefault
scraper.allowPatternstringURL pattern (regex)
Python regex matched from the start of the URL. It must begin with the literal scheme and host, closed by /, ?, (?=/|$) or $, e.g. ^https://news\.ycombinator\.com/
""
scraper.maxBodyCharsnumberMax item body characters
Feed item bodies are trimmed to this many characters.
20000
scraper.maxResponseKBnumberMax response size (KB)
Response bodies are read up to this size; anything beyond is dropped and the result is marked truncated.
5120
scraper.maxToolCharsnumberMax characters per tool result
Ceiling on what one tool call returns to the agent, 1000-200000 characters. The agent's max_chars can only lower it; longer content and item lists are cut and marked truncated.
20000
scraper.requestIntervalMsnumberDelay between requests to one host (ms)
Politeness pacing: minimum gap between two requests to the same host. 0 disables pacing.
250
scraper.timeoutnumberTimeout (seconds)
Time limit for one whole fetch, 1-120 seconds: DNS lookup, connecting, the TLS handshake, response headers, redirects and the body all count toward it.
20
scraper.urlAllowListarrayURL allow-list
Optional. When one or more patterns are set, a URL (and every redirect target) must match at least one. Empty allows every public URL; private and local addresses are always blocked.
scraper.userAgentstringUser agent
User-Agent header sent with every request. Leave blank for the default.
""

Dependencies​

  • beautifulsoup4