Skip to content

Scale out error ingestion for EF (3/3): liveness and readiness endpoints - #5803

Open
johnsimons wants to merge 1 commit into
john/scaleout_error2from
john/scale_error3
Open

Scale out error ingestion for EF (3/3): liveness and readiness endpoints#5803
johnsimons wants to merge 1 commit into
john/scaleout_error2from
john/scale_error3

Conversation

@johnsimons

Copy link
Copy Markdown
Member

Part 3 of 3 introducing scale-out of error ingestion for EF persistence. Depends on #<PR 2>.

The ingestion-only host from part 2 serves no HTTP, so the container health check had
nothing to probe. This adds health endpoints to both hosts so one image serves both modes.

Endpoints

  • GET /health runs no checks and answers "is this process still serving". This is what a
    container health check should restart on.
  • GET /health/ready runs ready tagged checks. Currently one, reading the state the
    ingestion watchdog publishes. That covers more than it looks: a batch that keeps failing,
    including because the database is unreachable, trips the fault policy's circuit breaker,
    which raises a critical error, which the watchdog records there.

Both are anonymous and mapped in the normal host via UseServiceControl and in the
ingestion host inside BuildHost.

Responses are JSON rather than the default plain text because HealthCheckApp explicitly
rejects anything that is not non-empty application/json.

Dockerfile

HEALTHCHECK now probes /health instead of /api/configuration.

Reviewers should weigh this: /api/configuration exercised the API pipeline and the
license, so /health is a weaker signal. I think that is the right direction for something
that restarts containers, since failing a probe on a transient database blip and triggering
a restart loop is worse than a shallow check. If you would rather it stayed strict, point it
at /health/ready instead.

Testing

When_requesting_health covers the normal host on every persister, and the ingestion-only
end-to-end test asserts both endpoints including the JSON content type.

Verified: full solution Release clean, and the full RavenDB (177), SQL Server (151),
PostgreSQL (151) and Audit (65) acceptance suites.

Known gap

/health/ready has no direct database connectivity check. Ingestion failures reach it
transitively via the circuit breaker, but a dedicated check needs a health check package
reference in the persistence projects and a per-persister implementation.

Introduces dedicated `/health` (liveness) and `/health/ready` (readiness) endpoints to replace the Dockerfile's reliance on the configuration API for health monitoring.

The readiness check incorporates a new error ingestion health check, which monitors the state of the ingestion watchdog. This is particularly relevant for instances running in error-ingestion-only mode. The health check response is explicitly formatted as JSON to maintain compatibility with the container health check binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants