Endpoints
Ingest a website
Hand the pipeline a URL — it crawls, extracts, chunks, embeds, and stores the whole site as public knowledge. Async, with a pollable job.
POST/v1/sources/website
The frontend hands the ingestion pipeline just a URL. The whole pipeline runs from it — crawl (sitemap-aware) → extract readable text → chunk → embed → upsert as the tenant's public knowledge. After it finishes, the widget can answer from the site.
This is asynchronous: the call returns a jobId immediately, and you poll a
second endpoint for progress. Requires a key with an internal_agent surface.
Request body
Prop
Type
{ "url": "https://acme.example.com", "maxPages": 200 }Response — 202 Accepted
Prop
Type
{ "jobId": "job_7f31c8", "status": "queued", "url": "https://acme.example.com" }GET/v1/sources/website/:jobId
Poll for progress. Returns the job record; 404 if the id is unknown.
Prop
Type
{ "jobId": "job_7f31c8", "status": "crawling", "pagesCrawled": 42 }{
"jobId": "job_7f31c8",
"status": "done",
"pagesCrawled": 118,
"chunksIngested": 964,
"finishedAt": 1752412800000
}Poll loop
JOB=$(curl -s -X POST http://localhost:8080/v1/sources/website \
-H "Authorization: Bearer sa_demo_internal_key" \
-H "Content-Type: application/json" \
-d '{"url":"http://localhost:8080/sample/index.html"}' | jq -r .jobId)
until [ "$(curl -s http://localhost:8080/v1/sources/website/$JOB | jq -r .status)" = "done" ]; do
sleep 1.5
curl -s http://localhost:8080/v1/sources/website/$JOB | jq -c '{status, pagesCrawled}'
doneTry it
POST http://localhost:8080/v1/sources/website
Ingest requires an internal_agent key. The public demo key returns 401.