# Web scrape

> Fetch a page through the proxy and extract text, rows or links with a CSS selector; scripts never run.

Source: https://docs.tune.toys/nodes/web.scrape/

Web scrape fetches a web page through the Tune.Toys proxy and pulls out what a CSS selector matches: text, HTML, an attribute or a table, plus the page's links.

## What it does

With `trigger` cabled, each trigger scrapes once; otherwise a new `url` does. The page is always fetched by the proxy, so you must be signed in, and the host must be in your network allowlist. The proxy honours the site's `robots.txt` (a disallowed page fails with an error) and returns HTML as text; nothing on the page runs. The browser then parses it with a bounded HTML parser and sends:

- `text`: the text of every matched element, joined;
- `items`: one table row per matched element;
- `links`: a table of the page's links, with `javascript:` and other non-web schemes dropped;
- `error`: why the scrape failed.

## When to use it

Use it for public pages that have no API. When a site offers an API, a [Web request](https://docs.tune.toys/nodes/web.request/) or an addon node is more reliable and kinder to the site.

## Ports

<!-- ports:start -->

| Port      | Direction | Signal         | Lanes |
| --------- | --------- | -------------- | ----- |
| `trigger` | Input     | Data (trigger) | 1     |
| `url`     | Input     | Data (string)  | 1     |
| `text`    | Output    | Data (string)  | 1     |
| `items`   | Output    | Data (table)   | 1     |
| `links`   | Output    | Data (table)   | 1     |
| `error`   | Output    | Data (string)  | 1     |

<!-- ports:end -->

## Parameters

<!-- parameters:start -->

| Parameter | Label   | Range     | Default | Unit | Scale  | Notes                                                                                  |
| --------- | ------- | --------- | ------- | ---- | ------ | -------------------------------------------------------------------------------------- |
| `extract` | Extract | 0 to 3    | 0       |      | linear | What each selected element contributes. Choices: 0 Text, 1 HTML, 2 Attribute, 3 Table. |
| `limit`   | Limit   | 1 to 1000 | 100     |      | linear | Most rows extracted per page.                                                          |

<!-- parameters:end -->

<!-- data:start -->

- `url`: url.
- `selector`: text.
- `attribute`: text.

<!-- data:end -->

### Extract (`extract`)

What each matched element contributes: its Text, its HTML, the value of the node's `attribute` field (such as `href`), or, for `table` elements, its cells as rows.

### Limit (`limit`)

The most rows taken from one page, 1 to 1 000.

### Selector and attribute

The `selector` field is a CSS selector such as `h1`, `.headline a` or `table.prices`; the `attribute` field names the attribute for Extract Attribute.

## Tips

- Check the site's terms before scraping it, and scrape rarely.
- Pages that build their content with scripts look empty to a scraper; look for the site's API instead.

## See also

[Web request](https://docs.tune.toys/nodes/web.request/), [Convert](https://docs.tune.toys/nodes/data.convert/), [Console log](https://docs.tune.toys/nodes/log.console/), [Data cables](https://docs.tune.toys/concepts/data-cables/).