Web scrape

Fetch a page through the proxy and extract text, rows or links with a CSS selector; scripts never run.

Web
Since 0.2.0
web
scrape
html
css
selector
extract
network
data

Web scrape as the Patch draws it, at default values: inlets on the left, outlets on the right.

Web scrape fetches a web page through the Tune.Toys proxy and pulls out what a CSS selector matches: text, HTML, an attribute or a table, plus the page's links.

What it does#

With trigger cabled, each trigger scrapes once; otherwise a new url does. The page is always fetched by the proxy, so you must be signed in, and the host must be in your network allowlist. The proxy honours the site's robots.txt (a disallowed page fails with an error) and returns HTML as text; nothing on the page runs. The browser then parses it with a bounded HTML parser and sends:

text: the text of every matched element, joined;
items: one table row per matched element;
links: a table of the page's links, with javascript: and other non-web schemes dropped;
error: why the scrape failed.

When to use it#

Use it for public pages that have no API. When a site offers an API, a Web request or an addon node is more reliable and kinder to the site.

Ports#

Port	Direction	Signal	Lanes
`trigger`	Input	Data (trigger)	1
`url`	Input	Data (string)	1
`text`	Output	Data (string)	1
`items`	Output	Data (table)	1
`links`	Output	Data (table)	1
`error`	Output	Data (string)	1

Parameters#

Parameter	Label	Range	Default	Unit	Scale	Notes
`extract`	Extract	0 to 3	0		linear	What each selected element contributes. Choices: 0 Text, 1 HTML, 2 Attribute, 3 Table.
`limit`	Limit	1 to 1000	100		linear	Most rows extracted per page.

url: url.
selector: text.
attribute: text.

Extract (`extract`)#

What each matched element contributes: its Text, its HTML, the value of the node's attribute field (such as href), or, for table elements, its cells as rows.

Limit (`limit`)#

The most rows taken from one page, 1 to 1 000.

Selector and attribute#

The selector field is a CSS selector such as h1, .headline a or table.prices; the attribute field names the attribute for Extract Attribute.

Tips#

Check the site's terms before scraping it, and scrape rarely.
Pages that build their content with scripts look empty to a scraper; look for the site's API instead.

Help patches

Each help patch is an ordinary Toy that teaches one idea. The diagram shows the Patch; the preview shows its Frame. Open it in TuneToys to play and edit it.

1. Read a heading

beginner Teaches: extract, limit

Web scrape fetches a page through the proxy and extracts the text of its first-level headings.

Patch: Value value output connected to Convert value input. Web scrape text output connected to Console log “log” value input. Web scrape error output connected to Console log “oops” value input. Convert value output connected to Web scrape url input.

Connections as text

Value value output connected to Convert value input
Web scrape text output connected to Console log “log” value input
Web scrape error output connected to Console log “oops” value input
Convert value output connected to Web scrape url input

Extract Text joins the text of every element the selector matches, up to Limit.
The proxy fetches HTML, honours robots.txt and never runs scripts.

Frame preview (static). Controls show their default values.

Open in TuneToys

Web request Fetch a URL on each trigger under the network allowlist; secrets come from a vault slot, never the Toy.
Convert Convert a value to another datatype, optionally picking a field or formatting a string; failures go to error.
Console log Write each incoming value to the Console panel with a level and label, rate-limited.