# Logmoth service API — agent reference

Source snapshot: 2026-10-20. Describes the Logmoth instance service's current HTTP API.
Use the instance's HTTPS origin as BASE_URL, for example https://YOUR-INSTANCE.logmoth.sh.
Do not send these requests to logmoth.com or control.logmoth.com; those are the frontend and control plane.
Obtain the instance URL and an appropriately scoped API key from the user/operator.
Never put a key in a URL, browser bundle, committed file, or log.

## Instructions for the implementing agent

Use this document to build an integration, not just individual HTTP calls. Inspect the user's
existing logging pipeline, runtime, and deployment lifecycle first. Reuse their HTTP client,
configuration, metrics, and background-job conventions. Establish expected event sizes and
volume, acceptable delivery delay, and whether losing queued logs during a crash is acceptable.
Explain the delivery policy you choose. The client defaults below are starting recommendations,
not server limits or a throughput guarantee; make them configurable and tune with measurements.

Deliver a small client with enqueue, flush, and close operations, bounded resource usage,
explicit failure reporting, a clear and human-readable code structure (see "Structure the
client code"), and tests for the failure paths below. Keep ingestion credentials
on a trusted server or collector. Use separate least-privilege ingest and read keys. Do not
automatically create keys, delete data, or trigger administrative merges to make a demo work.

### Help the user use Logmoth well

- Explain the model with one of their real event types: a namespace groups events with a
  shared attribute schema; attrs are small typed fields used to find events; payloads retain
  rich details to inspect after finding them. Do not put an entire trace in a string attribute.
- Propose stable names and types before the first write. Use namespaces for meaningful
  dataset/schema boundaries, not one namespace per request. Keep request, trace, run, tenant,
  and user identifiers as string attribute VALUES, never dynamically generated field names.
  Identifiers should be strings even when they look numeric; numbers are for measurements.
- For agent traces, consider one event per meaningful step with string trace_id, span_id,
  parent_span_id, service, environment, model, tool, and outcome attributes as appropriate;
  numeric duration_ms and token counts; and prompt, response, tool arguments/results, and
  error details in payloads. Select fields the user will actually query, not an exhaustive
  schema. Record event time once at creation so queueing delay does not change its meaning.
- Keep enough context to reconstruct what happened, including model/version and failures
  when available. Redact credentials and apply the user's policy for personal or sensitive
  content before enqueueing, including before writing a local spool. Logs are evidence of
  runtime behavior; they complement evals rather than proving correctness by themselves.
- Demonstrate two concrete workflows: find failed/slow steps in a time range, then find the
  related trace by its string ID and load selected payloads. Start with attributes only and
  retrieve large payloads on demand. Attribute indexing is automatic within the limits below;
  sending arbitrary payload JSON does not make its nested fields searchable.
- Explain that attribute types are fixed on first use. A type migration needs an explicitly
  chosen new attribute or namespace, not a client that alternates strings and numbers.
  Check /attributes for the actual types and indexed flags before building query controls.
  Ingest keys can call /attributes, so a producer can check types before sending.
- This API snapshot has no payload-promotion/backfill endpoint. Do not invent one or promise
  retroactive indexing through this API; extract needed attributes before ingesting new events.
  Design notes and marketing descriptions are not evidence that an endpoint is available.
- Object storage provides the storage foundation, but an instance still has finite ingest,
  memory, and query capacity. Explain 429 as backpressure. Measure representative workloads;
  do not promise constant latency, a fixed compression ratio, or a fixed total bill. /usage
  reports stored bytes; raw input volume, stored volume, and total operating cost differ.
- Call out current query boundaries: this is event retrieval plus simple aggregations
  (count, sum, avg, min, max grouped by attributes or time buckets) in restricted SQL, not
  joins, percentiles, or a general analytics database. An aggregation either completes within
  60 s or fails with a suggestion to narrow it; it never returns partial totals. Report coverage
  and pagination honestly. Do not interpret an empty first page as proof that nothing happened,
  or the recent view as the entire history. To count or group events, use an aggregate query
  instead of paging through events and counting client-side.

### Build a bounded, asynchronous producer

The steps below and the next two sections describe one client design end to end. Build all of
it: the pieces depend on each other, and partial versions fail under load, overload, or restarts.

1. Prepare each event once, inside the client's enqueue rather than at call sites, so every
   caller gets the same validation and limits. Assign a unique idempotency_key per logical event
   (for example a UUID, persisted if records are spooled) and record ts at creation. Do not
   generate a new key, timestamp, or payload for each attempt. A trace ID alone is not a unique
   event key when a trace has steps. In one pass over the flattened attrs (nested objects become
   dotted names), record each attribute's type (string, number, bool, or string[]; skip nulls)
   and enforce the limits in step 2. Serialize the event once and keep the exact bytes,
   namespace, key, and attribute types together in the queue. Retries and restarts resend
   those bytes unchanged.
2. Attribute limits are 256 UTF-8 bytes per string (including each string[] item), 64 items per
   array, and 200 flattened attributes per event. Never truncate attributes silently. If the
   client shortens strings, it must cut on a UTF-8 character boundary, add a string[] attribute
   such as truncated_attributes listing the affected names, count truncations by namespace and
   attribute name, and write a diagnostic record holding the original value and the truncated
   event's idempotency key (see the diagnostics namespace below). The marker counts toward the
   200: if there is no room for it, send the truncated event without the marker; the counter
   and the diagnostic record, which carries the event's key, still make the truncation visible.
   Write the truncated values into the serialized event in every case, and record truncations
   only for events that are actually queued truncated. Reject events with too many attributes
   or array items with a visible error rather than dropping fields. Tell the user the policy
   when you hand over the integration.
3. Enqueue into a queue bounded by BOTH event count and serialized UTF-8 bytes, counting queued
   and in-flight/retrying data. A reasonable initial in-memory budget is 10,000 events or
   32 MiB, whichever fills first. Enforce the byte budget while admitting an event, not after
   allocating an unbounded batch. Per-namespace queues are fine when this one global bound
   covers them all. Avoid unbounded background tasks.
4. Make queue-full behavior explicit: wait with a caller deadline, reject enqueue with a
   visible result, or drop according to a documented policy with a counter. Never silently
   drop or let logging exhaust the application's memory. Keep network I/O off the request
   path unless the user deliberately needs synchronous durability.
5. If crash survival matters, use a bounded durable spool or an existing durable queue.
   Persist the serialized event and key before reporting durable local acceptance; acknowledge
   or remove it only after validating the service's success response. Recover unacknowledged
   records after restart. An in-memory enqueue is not a durable service acknowledgement.
   Expose disk-full and spool corruption failures instead of silently skipping records.
6. Batch by instance and namespace. Send NDJSON, not a JSON array. Initially flush at 200
   events, 1 MiB including newlines, or 250 ms after the first waiting event, whichever comes
   first. The byte threshold is a batch target: send a larger valid event alone, still below
   event/request/proxy limits. Never split an event's JSON across requests. A batch rejects a
   key that appears twice, so collapse repeated queued keys with identical serialized content;
   report the same key with different content as a conflict instead of dropping either event.
7. Deliver through a write pump. When the client is created, start a fixed pool of long-lived
   writer goroutines, threads, or tasks sized to the maximum in-flight batches (initially 2 per
   instance) and sharing one HTTP connection pool. Each writer loops: take the next ready batch
   from the bounded queue, deliver it with retries inside that same writer, mark it done, and
   repeat. Each writer awaits its own request inline. Never spawn a detached goroutine, task,
   or promise per batch or per request, even when a counter or semaphore limits how many run
   at once. The pool is shared by all namespaces with round-robin scheduling between them, not
   2 writers per namespace. Each writer holds at most one batch. Concurrency on the wire is
   controlled in one place, the send gate: every HTTP attempt needs a send slot (see Retries),
   so a writer may take a batch while sending is paused and wait for a slot while holding it.
   The bound applies per client process: run one client per process, and remember that N
   application processes put up to 2N batches in flight against the same instance. Set
   connection and whole-request deadlines (for example 3 s and 30 s).
8. Classify every batch response as exactly one outcome:
   - Accepted: 200 whose accepted count and events array both equal the number of lines sent.
     events is in line order, and duplicate=true counts as delivered.
   - Lines rejected: 400 or 409 with line_errors. Handle as described under Retries.
   - Retryable: transport failures, timeouts, 408, 429, 500/502/503/504, an unreadable
     response body, or a 200 whose body is malformed or does not match the batch.
   - Rejected: any other status, or a 400/409 without line_errors. The whole batch fails; record
     the status and a short excerpt of the body.
   Also track, per batch, whether any attempt may have committed. A connection that was never
   established (DNS failure, refused connection, connect timeout) did not commit. Anything that
   fails after the request may have been sent (timeout, reset, read failure, malformed 200) is
   uncertain. If the batch finally fails, report its records as uncertain rather than failed
   when any attempt was uncertain.
9. flush makes partial batches ready immediately instead of waiting for the 250 ms timer, then
   waits for records accepted before that call to receive a terminal outcome, with a deadline
   and explicit delivered/failed/pending result. close stops intake, makes partial batches
   ready, lets the writers drain until its deadline, then cancels them (waking writers that
   wait on the queue, a send slot, or a backoff) and waits for the pool to exit before
   preserving pending durable records or reporting unsent memory records. A request cancelled
   mid-flight may have committed; report its records as uncertain, not failed. Cancel timers
   cleanly. Do not claim guaranteed flushing after SIGKILL.
10. Long-lived writers need a long-lived process. In pre-fork servers, create the client in each
    worker after the fork. In serverless or edge runtimes that freeze or end background work
    after the response, flush before the invocation finishes (for example with the platform's
    waitUntil) or hand events to a collector or sidecar that owns the queue and writers.

### Check attribute types before sending

Attribute types are fixed per namespace on first use, and the server reports only the first
type conflict in a batch. A client that relies on server errors alone therefore needs one round
trip per bad line. Check types locally instead:

- Ingest keys can call GET /v1/ns/{namespace}/attributes. Fetch it for a namespace before that
  namespace's first batch; 404 means the namespace has no attributes yet. Instances that predate
  this permission return 403. Treat that like any failed fetch (below): the check keeps working
  from cached types, and the periodic retry picks the permission up once the instance is
  upgraded.
- Keep one cached type map per namespace, shared by all writers. The server is authoritative: a
  successful fetch replaces the cached map. After a batch is delivered, add the types it
  introduced.
- Before each send, walk the batch in order. The first event to use a name fixes its type for
  the rest of the batch. Quarantine any event whose type for a name differs from the cache or
  from an earlier event in the batch, and record the reason.
- Refetch after every 400 line rejection. Before quarantining an event locally, refetch if the
  cache is more than a few seconds old (for example 10 s), then check again. Otherwise a stale
  cache (for example after the namespace was deleted and recreated with different types)
  rejects good events forever without ever asking the server. If a fetch fails, check against
  the cached types only and retry the fetch at most once a minute per namespace, logging each
  failure as a warning; that rate limit is what keeps the warnings bounded.

### Retries, 429, and overload control

Use one retry layer; avoid multiplying HTTP-library retries by producer retries. Retry only
operations whose semantics permit it. Reads can be retried with unchanged parameters; writes
require the preserved idempotency keys described above. Do not apply an ingest retry policy
blindly to key creation, deletion, or other administrative operations.

- Every HTTP attempt, the first and each retry, acquires a send slot. A slot is free when
  dispatch is not paused and active sends are below the shared in-flight limit; otherwise the
  writer waits. When a pause ends, wait an extra random delay (for example up to 500 ms) so
  paused writers do not resume at the same instant. Release the slot once the response has
  been read. Checking the limit only before taking a new batch is not enough, because writers
  already retrying would keep the old concurrency.
- On a retryable outcome, compute the batch's delay (below), extend the shared pause to at
  least that delay, and lower the in-flight limit to 1. Then acquire a send slot again; that
  acquisition is where the writer waits, so there is no second, separate retry sleep. It is
  simplest to do this for every retryable failure, not only 429. Both 429 causes (memory
  pressure and a full write queue) apply to the whole instance, not one namespace, and 5xx and
  transport failures usually do too. Raise the limit by one after sustained success (for
  example 20 consecutive deliveries), never above the pool size. Do not stop or start writers
  to change concurrency. Never fan a rejected batch out into individual requests.
- After failed attempt n (n=0 for the first failure), choose
  `jitter = uniform(0, min(30s, 500ms * 2^n))`.
  Parse Retry-After as either nonnegative seconds or an HTTP date; for valid values use
  `delay = max(jitter, retryAfterDelay)`. A server delay is a floor, not capped at 30 s.
  Invalid/missing headers fall back to jitter. The service currently sends `Retry-After: 1`
  only on the 429 for memory pressure; 429s for a full write or merge queue carry no header,
  so your own backoff applies.
- Bound retries with at most 8 attempts (including the first) and a 5-minute delivery deadline
  from the first attempt, measured with a monotonic clock, whichever ends first. Count only
  retryable failures as attempts. If the next delay would pass the deadline, report/defer the
  batch instead of retrying early or silently discarding it. With these delays 8 attempts end
  after about a minute, while an instance restart can take several minutes. A batch that
  fails during a long restart is therefore reported as failed (uncertain if any attempt may
  have committed), not held. Raise the cap if the user would rather hold batches longer.
  During a restart, connections are refused while storage loads; afterwards ingest requests
  are accepted but wait for WAL recovery, so they can hit the client timeout. Treat both as
  transient, and treat the timeouts as uncertain outcomes.
- Lines rejected is not a transport failure. line_errors numbers the batch's nonempty lines
  from 1, in the order sent. Remove those lines, quarantine them with their reasons, refetch
  types after a 400 and re-run the type check on the remainder, then resend at once. These
  rounds use no retry attempts and no backoff. Each round removes at least one event, so the
  loop ends. If the reported lines match nothing that was sent, fail the whole batch. Error
  lists stop at 100 entries, and a 409 names only the first conflicting key, so revalidate the
  remainder rather than assuming it is clean.
- In a batch, an event over 10 MiB is reported as a 400 line error; quarantine that line.
  A 413 on the batch endpoint means the whole body exceeded 64 MiB (plain-text body), or a proxy
  limit was lower: split the batch, and quarantine a single event that still gets 413. On the
  single-event endpoint, an event over 10 MiB returns 413 with a JSON error. Do not truncate
  payloads silently or retry an oversized event forever.
- Do not blindly retry 400, 401, 403, 404, 409, 413, 415, or 422. Surface
  validation/auth/configuration errors. Idempotency conflicts mean the same key was used for
  different content; never “fix” that automatically by giving the event a new key. A namespace
  being deleted needs operator attention, not automatic recreation through endless ingest
  retries. GET /v1/status lists deleting_namespaces; once cleanup finishes, the next successful
  ingest creates a new, empty namespace with that name.
- For prolonged outages, the pause and an in-flight limit of 1 already act as a single probe;
  do not dispatch continuously. Keep queue admission bounded while paused. Exhausted retries
  become an explicit failure or durable deferred/dead-letter record. A durable replay worker
  needs its own paced schedule and retry-age policy; it must not reset the budget in a tight
  loop.
- Idempotency retention is about 24 hours from server commit, managed in hourly buckets.
  It is not permanent deduplication. Replays beyond that window can create duplicates,
  even with the same key. Use a conservative retry-age cutoff from the first send attempt;
  explain and resolve older uncertain records explicitly. Delivery is at least once with
  bounded deduplication, not an exactly-once guarantee.

### Structure the client code

Clear, human-readable structure is a requirement, not a polish step. The user's team has to
debug this client during an incident, so someone new to it should be able to find each rule in
this document in one obvious place. Split the client into small components, one file or
module each, named for what they do:

| Component | Owns |
| --- | --- |
| Client | The public API (constructor, enqueue, flush, close). It wires the components together, starts the writer pool, and holds little logic of its own. |
| Event preparation | Flattening attrs, recording types, enforcing limits, and truncating with the marker. Pure functions with no I/O or locks. |
| Event queue | The count and byte bounds, per-namespace pending events, batch readiness, round-robin selection, and release of delivered batches. |
| Send gate | Active sends, the in-flight limit, the shared pause, resume jitter, and gradual recovery. |
| Delivery | The writer loop, the per-batch retry loop, building requests, and classifying responses. |
| Attribute types | The per-namespace type cache, fetching /attributes, and the pre-send conflict check. |
| Diagnostics | Diagnostic-namespace records, dropped-event counts by reason, and the periodic summary. |
| Metrics | Instrument definitions and the observer that reads queue and send-gate snapshots. |
| Integrations | Adapters such as a logging-framework writer, kept outside the core so the core stays generic. |

Rules that keep it readable:
- Each stateful component owns its state and its lock and exposes small methods (admit,
  takeReadyBatch, acquire, backOff, and so on). Other components call those methods and never
  reach into its fields. Never hold one component's lock while calling another component or
  doing I/O.
- Keep decisions in pure functions that tests can call directly: attribute preparation,
  response classification, Retry-After parsing, backoff, and batch selection.
- The writer loop and the per-batch retry loop should read top to bottom as the steps in this
  document: take a batch, check types, acquire a slot, send, classify, then deliver, quarantine
  lines, retry, or fail. Keep them short enough to read on one screen.
- Name outcomes and drop reasons explicitly (an enum or constants), not with booleans or
  magic strings scattered through the code.
- Put each component's defaults as named constants at the top of that component, so the tuning
  knobs are easy to find and make configurable.
- Follow the codebase's existing naming, error handling, logging, and metrics conventions.

### Query clients

Use bounded page iteration rather than loading every page into memory. Preserve the request's
filters, time bounds, ordering, and cursor for retries. Process a page successfully before
checkpointing its next_cursor; make downstream processing idempotent if replay matters.
Honor cancellation and an overall query deadline, not only the server's per-page time budget.
Set the HTTP timeout above time_budget_ms plus transport overhead. Show partial progress,
coverage, and work; continue across empty pages with a non-null cursor. A later fresh search
is needed to discover new events; paginating a snapshot is not a live subscription.

Aggregate queries (see POST /v1/sql) are one request with no cursor. Set the HTTP timeout
above the aggregation budget (60 s by default) plus transport overhead. Treat 422 as "query too
expensive", not as a transient failure: do not retry it unchanged. Show the error and hint, and
narrow the time range, add WHERE conditions, or split the range into shorter queries and add up
counts and sums (recompute avg from sum and count; min and max combine directly).

### Diagnostics, metrics, and tests

Write quarantined events and truncation records to a diagnostics namespace owned by the client
(for example logmoth_client). Each record holds the record type, source namespace, reason, and
the full original event or value in a payload. Never create diagnostic records about the
diagnostics namespace itself, or one bad record can generate new ones forever. Keep accepting
diagnostic records while close drains, so late rejections are still captured.

Count every event that will never be delivered by namespace and reason (for example queue_full,
closed, oversized, invalid_event, retries_exhausted, shutdown, with an _uncertain variant where
it applies). Log a summary at most once a minute per namespace and reason. If application logs
are shipped through this client, only rate-limited client logs (this summary and the attribute
fetch warnings, each at most once a minute per namespace) may pass through the same queue; never
log each dropped event or failed send individually.

Expose queue events/bytes, oldest queued age, in-flight batches and the current in-flight
limit, delivery latency, delivered and duplicate events, retries by reason, 429 count,
validation failures, truncated attributes, dropped/quarantined records, and pending records at
shutdown. Alert on growing queue age and sustained 429s. Avoid event/trace IDs as metric labels.
Include safe error context without keys or full payloads. Relate client metrics to /status and
/usage when diagnosing capacity.

Before handing the integration over, test with a fake transport and controlled clock/randomness:
- delivery: successful batch delivery; byte-based batch splitting; a single oversized event;
  queue-full behavior; a constant writer count across many batches and 429s;
- retries: timeout after server commit followed by duplicate success; 429 with and without
  Retry-After (including an HTTP date); 503 recovery; attempt-cap and deadline exhaustion;
  a 429 lowering concurrency for batches already retrying; paused writers not resuming at the
  same instant; a refused connection reported as not committed and a timeout reported as
  uncertain;
- rejections: malformed success response; invalid batch lines; a batch with many attribute-type
  conflicts quarantined without one round trip per line; a stale type cache corrected by a
  refetch; a 403 from /attributes falling back to cached types and retrying at most once a
  minute; 401/403 and idempotency conflict without retry storms;
- attributes: an over-limit value truncated with a marker attribute, counter, and diagnostic
  record; an event with exactly 200 attributes and one over-limit value; no diagnostic records
  generated about the diagnostics namespace;
- lifecycle: graceful shutdown with an in-flight request; durable-spool restart; empty query
  pages with next_cursor.
Use a test instance for end-to-end checks and ask before load-testing production. Give the user
a short example of enqueue/flush/close, two useful queries, the chosen loss/durability and
attribute-truncation policies, and tuning knobs.

## Authentication and conventions

Send `Authorization: Bearer <API_KEY>` on /v1 requests. The `Bearer ` prefix is matched exactly
(capital B, one space).
Keys have one scope: ingest (write events and read attribute types), read (read/query only), or admin
(all API operations). The operator key also has full access. A read key cannot ingest; an ingest key
can call GET /v1/ns/{namespace}/attributes but cannot read events or anything else.
Browser UI sessions are not a substitute for agent API keys.

Use `Content-Type: application/json` for JSON requests and `application/x-ndjson` for batches.
JSON responses use UTF-8. Treat event IDs, key IDs, and cursors as opaque strings.
Timestamps are RFC 3339; returned timestamps are UTC. Example IDs below are placeholders.
The service-wide request-body limit is 64 MiB; individual ingested events are limited to 10 MiB.
Proxies may impose smaller limits. Check status codes before parsing successful response shapes.

## Write events (ingest or admin)

### POST /v1/ns/{namespace}/events

Body:
```json
{
  "ts": "2026-10-20T12:00:00Z",
  "idempotency_key": "request-123-completed",
  "attrs": {
    "service": "agent",
    "status": "ok",
    "duration_ms": 125,
    "success": true,
    "tags": ["production", "tool-call"],
    "http": {"status": 200}
  },
  "payloads": {
    "trace": {"steps": [{"tool": "search", "result": "example"}]},
    "message": "Large or unstructured content goes here."
  }
}
```

All four top-level fields are optional; unknown top-level fields are rejected.
Omitting ts sets the event time to the commit time (committed_at); record ts at creation instead
if queueing delay matters.
Namespaces are created on first successful ingest; there is no create-namespace endpoint.

200 response:
```json
{"event_id":"<opaque-event-id>","committed_at":"2026-10-20T12:00:01Z","duplicate":false}
```

Retry an uncertain write with the same idempotency_key and unchanged event content.
Keys are scoped to the namespace. A matching duplicate returns the original event with duplicate=true;
reusing the key for different content returns 409. Without a key, retrying can create duplicates.
Payload JSON is compared byte-for-byte; ts and attrs are compared after parsing, so re-serializing
a payload with different whitespace or key order counts as different content (409).
Deduplication lasts about 24 hours from commit (hourly retention buckets), not indefinitely.
A successful response means the write has passed the service's WAL commit path; storage durability
depends on the instance's configured WAL/storage backend.

### POST /v1/ns/{namespace}/events:batch

Body is newline-delimited JSON (not a JSON array), one event per nonempty line.
Blank lines are ignored; an empty batch is rejected. Repeating an idempotency key within one batch
is a line error, even when both events are identical.
Validation errors reject the whole batch (400) without writing its events. Errors identify up to 100
offending lines by physical line number, starting at 1 (blank lines count toward the numbering).
An attribute-type conflict reports only the first conflicting line. An idempotency conflict with an
earlier batch returns 409 without writing any events; line_errors names only the first conflicting line.

200 response:
```json
{"accepted":2,"events":[{"event_id":"<id-1>","committed_at":"<timestamp>","duplicate":false},{"event_id":"<id-2>","committed_at":"<timestamp>","duplicate":true}]}
```
accepted counts returned accepted entries, including duplicates. events is in the same order as
the batch's nonempty lines.

### Event validation and data model

- Namespace: 1–64 bytes, lowercase a-z, digits, underscores, hyphens; cannot start with underscore.
- Attribute and payload names: 1–128 bytes; lowercase a-z, digits, underscores, hyphens and dots;
  cannot start with underscore. Attribute dot-separated segments cannot be empty.
- attrs is an object. Nested objects flatten into dotted attribute names (http.status above).
  Duplicate flattened names are rejected; null attribute values are omitted.
- Up to 200 flattened attributes per event.
- Values: string (up to 256 UTF-8 bytes), number, boolean, or string array (up to 64 items,
  each string at most 256 bytes). No mixed/numeric/object arrays in attrs.
- Numbers use floating-point storage. Integer magnitudes greater than 2^53 must be strings.
- Each attribute's type is fixed on first use within a namespace; conflicting writes fail.
- Up to 1,000 attributes per namespace are indexed automatically; additional names retain their
  fixed type but are registered with indexed=false. Check /attributes rather than assuming
  every field is indexed. payloads is an object mapping names to arbitrary JSON
  values; use it for large traces, bodies and nested data. Payloads are not query predicates.
- ts must be at least 1970-01-01 and before 2400-01-01.
- idempotency_key: nonempty string, up to 128 bytes.

## Query events (read or admin)

### POST /v1/ns/{namespace}/query

```json
{
  "where": [
    {"attr":"service","eq":"agent"},
    {"attr":"duration_ms","gte":100,"lt":1000}
  ],
  "time":{"from":"2026-10-20T00:00:00Z","to":"2026-10-21T00:00:00Z"},
  "order":"desc",
  "include":{"attrs":"*","payloads":["trace"]},
  "page_size":100,
  "max_bytes":4194304,
  "time_budget_ms":2000
}
```

All fields are optional. where defaults to [] and combines conditions with AND.
time is [from, to): inclusive from, exclusive to. If time is present, both from and to are required
and from must be before to. Without it, all supported times are considered.
order is asc (default) or desc by event time.
include.attrs is "*" (default) or an array of names. include.payloads is "*" or an array of names;
payloads are omitted by default. An empty selection array requests none.

Supported condition forms:
- eq: string, number, or boolean.
- in: 1–1000 strings.
- prefix: string prefix.
- exists: true only.
- contains: one string-array member.
- contains_any: 1–1000 string-array members.
- Numeric ranges: gt, gte, lt, lte, or between:[low,high] (inclusive).
  One lower and one upper bound can be combined. Do not mix range and equality operators
  in one condition. Attribute names and predicate types must match the namespace schema.

Page controls:
- page_size: 1–1000; default 100.
- max_bytes: 1–16777216; default 4194304.
- time_budget_ms: 1–10000; default 2000.
- cursor: pass the previous response's next_cursor unchanged.
  Keep where, time, and order identical while paging.

200 response:
```json
{
  "events":[{
    "event_id":"<opaque-event-id>",
    "ts":"2026-10-20T12:00:00Z",
    "committed_at":"2026-10-20T12:00:01Z",
    "idempotency_key":"request-123-completed",
    "attrs":{"service":"agent","duration_ms":125},
    "payloads":{"trace":{"steps":[]}}
  }],
  "next_cursor":null,
  "coverage":{"complete":true,"windows_not_yet_indexed":0},
  "work":{
    "windows_matched":1,"windows_read":1,"slices_read":1,"hot_windows_read":0,
    "windows_skipped_by_index":0,"windows_not_yet_indexed":0,"rows_examined":1,
    "events_returned":1,"bytes_read":1000,"elapsed_ms":1.5
  }
}
```

Returns 404 if the namespace has no events yet (treat it as "no data yet"), and 400 if a condition
names an attribute the namespace has never received.

idempotency_key is omitted if absent; payloads is omitted if empty/not requested.
Continue while next_cursor is non-null, even if a page has few or no events.
Page limits do not imply the whole result set has been returned.
Inspect coverage and work; windows_not_yet_indexed reports indexing work, not a row count.

### POST /v1/sql

A restricted SELECT language over one namespace, not a general SQL database:
```json
{
  "query":"SELECT * FROM agent_logs WHERE service = 'agent' ORDER BY ts DESC LIMIT 100",
  "max_bytes":4194304,
  "time_budget_ms":2000
}
```

For event queries, the optional cursor uses the same query text unchanged and the response is
the same as /query. LIMIT controls page size (1–1000, default 100). Query text is limited to
64 KiB. Aggregate queries return rows instead; see "Aggregations" below.

Shape: `SELECT <list> FROM <namespace> [WHERE …] [GROUP BY …] [ORDER BY …] [LIMIT n]`.
- Select list: `*` (all attributes), attribute names, `ts`, `payloads.<name>`, or `payloads.*`.
  Selecting only payloads returns no attributes; use `SELECT *, payloads.trace` for both.
  Every event includes ts, so naming it in the list is allowed and changes nothing.
- WHERE accepts conditions joined by AND, with parentheses:
  `=` on strings, numbers, or booleans; `<`, `<=`, `>`, `>=`, and `BETWEEN a AND b` (inclusive)
  on numbers; `IN ('a', 'b')` on strings (up to 1000 values); `LIKE 'abc%'` (case-sensitive
  prefix match); `IS NOT NULL`; `'x' = ANY(tags)` and `tags && ARRAY['a', 'b']` on string arrays.
- Bound time with ts: `ts >= now() - interval '1 hour'`, `current_timestamp`, or literals such as
  `'2026-10-19'`, `'2026-10-19 14:30'` (UTC) or RFC 3339, optionally cast with `::timestamptz`;
  `ts BETWEEN x AND y` includes both ends (unlike JSON time, which excludes `to`). Interval units
  go up to weeks; months and years are rejected. A continuation cursor keeps the first page's now().
- ORDER BY on event queries accepts only ts.
- Not supported: INSERT/UPDATE/DELETE, joins, HAVING, DISTINCT, OFFSET, UNION, OR, NOT,
  `!=`/`<>`, IS NULL, ILIKE, percentiles, and functions other than now(), the aggregates and
  the time buckets below.
  Errors return 400 with query_error and often a hint.
Use JSON /query for the explicit predicate contract above.

#### Aggregations

A query with an aggregate or GROUP BY returns one row per group instead of events:
```sql
SELECT model, date_trunc('hour', ts) AS hour, count(*), avg(duration_ms)
FROM agent_logs
WHERE succeeded = false AND ts > now() - interval '7 days'
GROUP BY model, hour
ORDER BY count(*) DESC
LIMIT 50
```

- Aggregates: `count(*)`; `count(attr)` (events that have attr, any type); `sum`, `avg`, `min`,
  `max` of number attributes; `min(ts)` and `max(ts)`. Optional `AS name` on any column.
- GROUP BY: string, number, and bool attributes; `date_trunc('second'|'minute'|'hour'|'day', ts)`;
  `time_bucket('5 minutes', ts)` with any interval, buckets aligned to the Unix epoch (UTC).
  A key can be written as the attribute, a select alias, or a 1-based column position. Events
  without the attribute form a null group. string[] attributes and raw ts cannot be grouped.
  Every non-aggregate column in the select list must appear in GROUP BY.
- ORDER BY: any output column, alias, 1-based position, or aggregate, ASC or DESC, comma-separated.
  Nulls sort last ascending and first descending. Without ORDER BY, rows follow the group keys
  ascending. LIMIT caps the rows (1–10000).
- Without GROUP BY, an aggregate query returns exactly one row (count 0, other aggregates null,
  if nothing matched). With GROUP BY and no matches, rows is empty.
- WHERE and ts bounds work as for event queries, and the global index skips hours without matches.

Request: `{"query": "...", "time_budget_ms": 60000}`. time_budget_ms is optional (1–60000,
default 60000). cursor and max_bytes are rejected for aggregate queries (400).

Response:
```json
{
  "columns": [{"name":"model","type":"string"}, {"name":"hour","type":"time"},
              {"name":"count(*)","type":"number"}, {"name":"avg(duration_ms)","type":"number"}],
  "rows": [["gpt", "2026-10-20T14:00:00Z", 412, 1830.5], ["claude", "2026-10-20T14:00:00Z", 398, null]],
  "groups": 2,
  "events_aggregated": 810,
  "coverage": {"complete": true, "windows_not_yet_indexed": 0},
  "work": {"windows_matched": 7, "windows_read": 7, "slices_read": 9, "hot_windows_read": 1,
           "windows_skipped_by_index": 0, "windows_not_yet_indexed": 0, "rows_examined": 810,
           "events_returned": 0, "bytes_read": 52311, "elapsed_ms": 4.2}
}
```
Each row is an array in column order. Column type is string, number, bool, or time; times are
RFC 3339 UTC strings; null means a missing group value or an aggregate over no values. Default
column names are the expression text, such as `count(*)` or `date_trunc('hour', ts)`.
groups counts every group before LIMIT; events_aggregated counts the events that matched.

Limits. An aggregation never returns partial results:
- If it does not finish within the time budget, it returns 422 with an error that says how many
  of the candidate hours it got through and suggests a time range that would fit, for example
  `WHERE ts > now() - interval '9 hours'`.
- More than 10,000 groups returns 422 with a query_error pointing at GROUP BY. Group by an
  attribute with fewer distinct values, add WHERE conditions, or narrow the ts range.
LIMIT does not avoid the group cap: every group is counted before sorting and truncating.

### POST /v1/sql/suggest

Body: {"query":"SELECT * FROM agent_logs WHERE ","position":31}.
position is an optional UTF-16 offset, defaulting to the end.
Response: {namespace, replace_from, replace_to, suggestions, error}.
Each suggestion has label, insert, kind, detail, and optional caret. Use for completion UI.

## Read and exploration endpoints (read or admin)

| Method and path | Parameters / result |
| --- | --- |
| GET /v1/namespaces | {namespaces:[...]} with namespace, event_count, ingested_bytes, first_event_time, last_event_time, last_committed_at, attribute_count, events_per_second_last_1m. |
| GET /v1/ns/{namespace} | Namespace summary plus ingest rate/lateness metrics. |
| GET /v1/ns/{namespace}/attributes | {attributes:[{name,type,indexed,first_received_event_id,earliest_event_time}]} with type string, number, bool, or string[]. Ingest keys can call this one too, to check types before sending. |
| GET /v1/ns/{namespace}/recent | limit=1–1000 (default 100); payloads=* or comma-separated names. Returns {events:[...]}. Recent-event convenience view, not full-history pagination. |
| GET /v1/ns/{namespace}/events/{event_id} | payloads=* or comma-separated names; returns one Event object. Payloads omitted by default. |
| GET /v1/ns/{namespace}/windows | from/to in UTC hour format YYYY-MM-DDTHH (inclusive bounds); limit=1–5000 (default 500). Returns {windows:[...],next_from}. Pass next_from as from to continue. |
| GET /v1/ns/{namespace}/windows/{window} | UTC hour YYYY-MM-DDTHH. Returns window state, merge statistics, slices, and unmerged statistics. |
| GET /v1/status | Instance resource, storage, ingest and merge status. |
| GET /v1/status/history | range=5m (default), 1h, or 24h. Returns {range,step_seconds,points:[...]}. |
| GET /v1/usage | {ingested_events_total,ingested_bytes_total,stored_bytes,stored_bytes_breakdown:{wal_bytes,slice_bytes,global_index_bytes},as_of}. |

## Administration (admin only)

Do not perform destructive or access-changing operations unless the user has requested them.

- GET /v1/keys -> {keys:[{key_id,name,scope,created_at,last_used_at}]}, oldest first.
- POST /v1/keys with {"name":"my-producer","scope":"ingest"} -> 201 and the same key metadata
  plus api_key. Store the secret securely when returned; listing keys does not reveal it.
  Name is trimmed, then must be nonblank, at most 100 characters, with no control characters.
  An invalid scope returns 400. An instance holds at most 1,000 keys; beyond that, 409.
- DELETE /v1/keys/{key_id} -> 204, revokes the key; 404 if there is no such key.
- POST /v1/ns/{namespace}/merge, optionally ?window=YYYY-MM-DDTHH; no body.
  Returns {merged:[...]} after the requested merge finishes; 409 if the namespace is deleted
  during the merge, 429 if the merge queue is full, 500 if the merge fails.
- DELETE /v1/ns/{namespace} -> 202:
  {namespace,state:"removing_objects",deleted_through_sequence,deleted_at}; 404 for an unknown
  namespace, 429 if the write queue is full.
  Destructive: logical removal is recorded and underlying object cleanup continues asynchronously.

## Health and errors

GET /healthz needs no key. Returns 200 {"status":"ok"} or 503 {"status":"starting"}.

Service errors normally use {"error":"human-readable message"}, with optional:
- line_errors:[{line,error}] for invalid/conflicting batches.
- query_error:{message,hint?,from,to,line,column} for SQL diagnostics.

POST /v1/sql returns a JSON 422 {"error", query_error?} when an aggregation is too expensive
(time budget exceeded, or more than 10,000 groups). Distinguish it from the framework's
plain-text 422 below by the JSON body.

The ingest endpoints parse events themselves and return JSON 400s for malformed events.
The other JSON endpoints (/query, /sql, /sql/suggest, POST /v1/keys) can return plain-text
400 (malformed JSON), 415 (missing or wrong Content-Type), or 422 (unknown field, wrong type, or
missing required field) from the HTTP framework. Bad query-string values (for example limit=abc)
and a body over 64 MiB (413) are also plain text. Do not assume the control plane's separate
{error,message} format.

| Status | Meaning / action |
| --- | --- |
| 400 | Invalid event, query, attribute type, or batch; fix the request. |
| 401 | Missing or invalid bearer key. |
| 403 | Key lacks the required scope. |
| 404 | Requested resource does not exist. |
| 409 | Idempotency conflict, ingest into a namespace whose deletion is in progress, namespace deleted during a merge, or the 1,000-key limit. Inspect error. |
| 413 | Request body over 64 MiB, or a single event over 10 MiB on POST …/events; reduce the size. |
| 415 | Missing or wrong Content-Type on a JSON endpoint (plain text). |
| 422 | JSON body with an unknown field, wrong type, or missing required field (plain text). |
| 429 | Write/merge capacity or memory pressure; back off with jitter and honor Retry-After if present. |
| 500 | Internal failure; report context without credentials. |
| 503 | Read/admin request (including /attributes) during startup, or WAL/idempotency storage unavailable; retry with backoff. Ingest during startup is accepted but waits for WAL recovery. |

## Minimal example

Set BASE_URL to the actual instance origin and API_KEY to an authorized key in your local
environment. An admin key can run both examples; otherwise use separate ingest and read keys.

```sh
curl --fail-with-body "$BASE_URL/v1/ns/agent_logs/events" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"idempotency_key":"example-1","attrs":{"service":"agent","status":"ok"},"payloads":{"trace":{"steps":[]}}}'

curl --fail-with-body "$BASE_URL/v1/ns/agent_logs/query" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"where":[{"attr":"service","eq":"agent"}],"order":"desc","include":{"payloads":["trace"]},"page_size":100}'
```