Core concepts
The objects you work with in Firehose — organizations, taps, rules, matches, keys, and URL Watch — and how each one relates to the others.
A few objects make up everything in Firehose.
Organization
The billing and ownership boundary. Taps, rules, filters, URL watches, and saved matches all belong to an organization.
Tap
A tap is an API token (prefixed fh_) scoped to an organization. It owns a set of rules and is
the thing you stream from. Create one tap per use case — e.g. one for brand mentions, one for
competitor news — so each has its own rule set and its own stream.
Rule
A rule is a query attached to a tap. A page is delivered to your stream if it matches any rule on the tap. Rules also carry optional flags — see Rules for the full rule object.
Match
When a crawled page satisfies a rule, that's a match. It arrives on the stream as an update
event carrying the page's metadata and the content diff that matched. See
Match payload for every field.
Keys
| Key | Prefix | Used for |
|---|---|---|
| Management key | fhm_ | Create, list, update, and revoke taps |
| Tap token | fh_ | Manage rules on a tap and open the stream |
See Authentication for details.
URL Watch
URL Watch is the other half of Firehose, and it has no rules. Instead of matching the live crawl against queries, it re-crawls a specific list of URLs you choose on a set cadence and records the diff between crawls. See Stream vs URL Watch to choose between them.
Next steps
Quickstart
Create a tap and a rule and open your first stream.
Stream overview
How rules match crawled pages and deliver them live.