Data feeds

A data feed is a saved job search plus a destination. JobsPipe delivers the postings it first discovered since the last delivery that match the search. Set one up in the dashboard under Developers → Data feeds. Feeds are part of the Builder package and above; workspace owners and admins create and change them, and every member can see them.

What a row is

Each row is a job exactly as POST /v1/jobs/search returns it (see the job schema).

• JSONL files hold one job object per line, gzipped.
• Parquet files and warehouse tables have these columns: id, job_title, company, company_domain, country_code, location, url, date_posted, discovered_at, job (the whole job object: a JSON column in Parquet and BigQuery, a VARIANT in Snowflake) and delivered_at.

A posting delivered twice to a warehouse updates its row rather than adding a second one. A day's bucket files are split every 10,000 rows. _SUCCESS is written last and is a small JSON file with the row count, the files, the window and the search, so a reader that waits for it never reads half a day. A day with nothing new still gets a _SUCCESS with row_count: 0.

Each feed in a workspace needs its own bucket and prefix.

Timing

A delivery covers postings discovered from where the previous one stopped up to four hours before it runs: a posting can take that long to become searchable after it is first seen. The first delivery starts with the day before the feed was created. Deliveries start within 15 minutes of the scheduled hour. One delivery carries at most 50,000 postings. When a window holds more, the delivery covers the earlier part of it, the feed's row says up to when, and the rest comes in the next delivery; no posting is skipped.

Credits

A posting in a feed costs what a job result from the search API costs, and a job your workspace already paid for this calendar month is free again, in a feed or a search. Credits are charged only after the files are written or the merge succeeds. Nothing is charged for:

• a delivery with no new postings,
• a delivery the destination refused or that could not be reached,
• a test delivery.

When the balance cannot pay for a whole delivery, it is skipped and the feed's row says it is out of credits, with the number it needed; the postings are not lost, and the next run after a top-up delivers them. A delivery is never charged in part.

Health

The feed's row shows what the last delivery did. After five failed deliveries in a row the feed switches itself off and shows Needs attention; fix the cause, send a test delivery, and switch it back on. Send a test delivery writes a small file under prefix/_jobspipe_test/, or one row that is removed straight away, so you can check permissions before the first scheduled run.

Credentials are stored encrypted and are never shown again after you save them; the dashboard shows only their last characters.

Permissions to grant

AWS S3

Create an IAM user (or role with an access key) limited to the feed's prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::your-bucket/your/prefix/*"
    }
  ]
}

Leave Endpoint empty and enter the bucket's region.

Cloudflare R2

Create an R2 API token with Object Read & Write on the bucket. Use https://<account id>.r2.cloudflarestorage.com as the endpoint and auto as the region.

Google Cloud Storage

GCS is written through its S3-compatible XML API. Create an HMAC key for a service account that has Storage Object User on the bucket (Cloud Storage → Settings → Interoperability). Use https://storage.googleapis.com as the endpoint and auto as the region.

BigQuery

1. Create the dataset.
2. Create a service account with BigQuery Data Editor on the dataset and BigQuery Job User on the project, and download a JSON key.
3. Paste the key file. The table is created on the first delivery. Short-lived stage tables named <table>_jobspipe_stage_* appear in the dataset during a delivery and are dropped after it; they expire on their own after a day if a delivery is interrupted.

Snowflake

1. Create a user for JobsPipe with key-pair authentication: generate an unencrypted PKCS#8 private key, then ALTER USER <user> SET RSA_PUBLIC_KEY = '...'.
2. Grant its role USAGE on the warehouse, database and schema, and CREATE TABLE on the schema: the target table is created on the first delivery, and each delivery uses a temporary stage table. If you create the target table yourself, also grant SELECT, INSERT, UPDATE, DELETE on it.
3. Enter the account identifier (for example myorg-myaccount), user, role, warehouse, database, schema and table, and paste the private key.