Skip to main content

Health checks and probes

Load balancers, Kubernetes probes, uptime monitors and Prometheus need to ask "is background processing working?" without signing in to the dashboard. Masterly.NightShift answers in three ways:

  • ASP.NET Core health checks. AddNightShift() adds a check to AddHealthChecks(). It answers in plain words, never with exception text, so it is safe on an anonymous endpoint.
  • The dashboard's /health and /metrics routes. A detailed JSON report and Prometheus text, behind the dashboard's authorization.
  • MapNightShiftHealth and MapNightShiftMetrics. The same report and text as endpoints of their own, outside the dashboard's authorization, which you limit with endpoint conventions.

The health check and the endpoints are in the Masterly.NightShift.AspNetCore package, in the Masterly.NightShift namespace. The health check's AddNightShift() extends IHealthChecksBuilder (what AddHealthChecks() returns); it is not the services.AddNightShift(...) that configures the library, which you still call as well.

Quick example​

A web app that runs a server: Kubernetes restarts the pod when its server stops sending heartbeats, and takes it out of the load balancer when the storage doesn't answer or no server in the cluster is alive.

using Masterly.NightShift;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddNightShift(config => config
.UseSqlServerStorage(builder.Configuration.GetConnectionString("NightShift")!));

builder.Services.AddNightShiftServer();

builder.Services.AddHealthChecks()
.AddNightShift(options => options.Scope = NightShiftHealthCheckScope.Process, name: "nightshift-server", tags: ["live"])
.AddNightShift(name: "nightshift-cluster", tags: ["ready"]);

var app = builder.Build();

app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = check => check.Tags.Contains("live") });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = check => check.Tags.Contains("ready") });

app.Run();

samples/WebSample in the repository is a complete app with these probes, the dashboard behind an authorization policy, and antiforgery. Run it with dotnet run --project samples/WebSample: it starts in Development on http://localhost:5000, where /health/live and /health/ready answer and the Development-only /login signs you in for /nightshift. It uses the LocalDB database Masterly.NightShift.WebSample unless ConnectionStrings:NightShift is set, and creates the database on first run if it doesn't exist.

How it works​

The health check​

AddNightShift() registers a check named nightshift. Each run makes one storage query, the list of servers with their last heartbeat, and compares the heartbeats with HeartbeatTimeout (one minute by default; servers send one every 30 seconds). The heartbeats are compared with the storage's clock where the storage has one, as the dashboard's pages do (see Which clock): one more read per run.

ScopeHealthy whenUnhealthy (or failureStatus) when
Cluster (default)The storage answers and at least one server anywhere sent a heartbeat within HeartbeatTimeout.The storage doesn't answer, no server is registered, or no server sent a heartbeat recently.
ProcessEvery server running in this process sent a heartbeat within HeartbeatTimeout.No server runs in this process (not started yet, or stopped), or one of them stopped sending heartbeats (the watchdog of another server may have removed it). When the storage doesn't answer, the result is Degraded, not the failure status: restarting the process wouldn't bring the database back, so a liveness probe keeps the pod running during an outage.

The result's Data has the numbers: servers and alive for the cluster, servers and silent for the process.

How the process scope knows its servers. With Scope = Process, AddNightShift also registers a small IBackgroundProcess. AddNightShiftServer starts the IBackgroundProcess services of the container in every server it creates, so this process learns each server's id when the server starts, and forgets it when the server stops. It holds one thread per server, which only waits.

Errors stay in the log. Exception messages can name the database server, the database or the login. The check's description only says "The NightShift storage did not answer." and the result carries no exception. The error itself is logged as a warning by the logger Masterly.NightShift.HealthChecks.NightShiftHealthCheck.

Which storage. The one registered by AddNightShift (the JobStorage service), or JobStorage.Current without one. Set NightShiftHealthCheckOptions.Storage to check another one.

The dashboard's /health​

GET {dashboard path}/health answers a JSON report: 200 with "status": "healthy" or "degraded", 503 with "status": "unhealthy". For example:

{
"status": "degraded",
"checkedAt": 1793958087,
"checks": [
{ "name": "storage", "status": "healthy", "detail": "answered in 4 ms" },
{ "name": "servers", "status": "healthy", "detail": "2 alive, 40 workers" },
{ "name": "queues", "status": "degraded", "detail": "no live server listens to: reports" },
{ "name": "processing", "status": "healthy", "detail": "running" },
{ "name": "recurring", "status": "healthy", "detail": "12 on schedule" }
],
"servers": 2,
"paused": false,
"queues": [
{ "name": "default", "length": 3 },
{ "name": "reports", "length": 17 }
],
"enqueued": 20,
"scheduled": 4,
"processing": 5,
"failed": 1,
"awaiting": 0,
"recurring": 12,
"retries": 2
}

checkedAt is a Unix time in seconds, servers counts the servers that are alive, and the overall status is the worst of the checks. Fields without a value (for example the job counts when the storage didn't answer, or fetched when the storage doesn't report it) are left out. The checks:

CheckUnhealthyDegraded
storageThe storage didn't answer. The detail says so and points to the log, where the error is.
serversNo server is registered, or none sent a heartbeat within the dashboard's ServerPossiblyAbortedThreshold.Some servers missed their heartbeat.
queuesA queue holds jobs that no live server listens to.
processingProcessing, a queue or a server is paused from the dashboard.
recurringA recurring job is more than a minute late: its next run, or the retry the scheduler set after an error, was due more than a minute ago and nobody started it (the Recurring Jobs page marks the same jobs as late). Or a recurring job is disabled by errors (no next run, with an error). The detail lists up to five ids. When the recurring jobs can't be read, the detail says so and the error is logged.

Cost. The report needs the statistics, the servers, the queues and the drain-mode hash. Late recurring jobs are found from the scores of the recurring-jobs set, which hold each job's next run: one query for the jobs due more than a minute ago, one for the jobs without a next run, and one hash field per job without a next run, at most 500 of each. A storage that can't query a set by score (it lacks JobStorageFeatures.ExtendedApi or JobStorageFeatures.Connection.BatchedGetFirstByLowest) falls back to reading every recurring job. A report is kept for 5 seconds per storage and ServerPossiblyAbortedThreshold (the only option it depends on), so probes that ask every second, several Prometheus servers, /metrics and the dashboard share one set of queries, even when MapNightShiftHealth, MapNightShiftMetrics and the dashboard are mapped with different DashboardOptions objects. A request served from a kept report still reads the storage's clock once, where the storage has one.

/ping answers pong without touching the storage: use it for a probe that only needs to know the web app is up.

Which clock​

A storage with a clock of its own writes server heartbeats with it: SQL Server, Redis, MongoDB, and the Entity Framework Core storage on a database it reads the clock of. /health, /metrics and the health check compare heartbeats, and the next runs of recurring jobs, with that clock, read once per request or run. So a process whose clock is ahead of the storage's doesn't report that no server is alive, and one whose clock is behind notices a stopped server on time. On a storage without a clock they use the process's clock, without asking the storage. When the storage doesn't answer for its clock, /health and /metrics make the report with the process's clock, and the health check reports that the storage did not answer.

MapNightShiftHealth and MapNightShiftMetrics​

Dashboard routes go through the dashboard's authorization filters, which a probe or a scraper usually can't pass. These two methods map the same /health report and /metrics text as endpoints of their own. They use the storage and DashboardOptions registered in the container unless you pass others, and the dashboard's authorization filters don't apply: add endpoint conventions to limit access.

// Only requests to the internal host name reach them
app.MapNightShiftHealth("/health/nightshift").RequireHost("jobs.internal:8080");
app.MapNightShiftMetrics("/metrics/nightshift").RequireHost("jobs.internal:8080");

The report names queues, servers and late recurring job ids, and the metrics name job methods; when that matters, prefer the health check, which only says whether things work.

Options / API​

NightShiftHealthChecksBuilderExtensions​

MemberMeaning
AddNightShift(this IHealthChecksBuilder builder, Action<NightShiftHealthCheckOptions>? configure = null, string? name = null, HealthStatus? failureStatus = null, IEnumerable<string>? tags = null)Adds the check. name defaults to "nightshift"; failureStatus to Unhealthy. Add it more than once, with different names, for both scopes. Throws ArgumentException when HeartbeatTimeout isn't positive.
DefaultName"nightshift".

NightShiftHealthCheckOptions​

PropertyTypeDefaultMeaning
ScopeNightShiftHealthCheckScopeClusterCluster or Process, see above.
HeartbeatTimeoutTimeSpan1 minuteHow old a server's last heartbeat may be for the server to count as alive. Keep it above the servers' HeartbeatInterval (30 seconds by default).
StorageJobStorage?nullThe storage to check; null uses the registered one, or JobStorage.Current.

NightShiftMonitoringEndpointRouteBuilderExtensions​

MemberMeaning
MapNightShiftHealth(this IEndpointRouteBuilder endpoints, string pattern, DashboardOptions? options = null, JobStorage? storage = null)Maps GET {pattern} to the dashboard's health report.
MapNightShiftMetrics(this IEndpointRouteBuilder endpoints, string pattern, DashboardOptions? options = null, JobStorage? storage = null)Maps GET {pattern} to the dashboard's Prometheus text. See Insights for the series.

Both throw InvalidOperationException when AddNightShift wasn't called.

More examples​

A worker service without a web server​

A Worker Service has no HTTP endpoint for Kubernetes to call. Publish the health report another way, for example with an IHealthCheckPublisher that writes a file a probe can check, or add a small Kestrel endpoint. The check itself works the same: register it with AddHealthChecks().AddNightShift(...) in the worker's services.

A degraded readiness instead of an unhealthy one​

When jobs are only part of what the app does, report trouble with them as degraded, so the load balancer keeps sending requests:

using Microsoft.Extensions.Diagnostics.HealthChecks;

builder.Services.AddHealthChecks()
.AddNightShift(failureStatus: HealthStatus.Degraded, tags: ["ready"]);

MapHealthChecks answers 200 for degraded by default.

Things to know​

  • The process scope needs servers started by AddNightShiftServer. A BackgroundJobServer you create yourself doesn't run the container's IBackgroundProcess services, and neither does AddNightShiftServer when you pass it your own list of additional processes. The check then reports that no server runs in this process.
  • Before the server starts, the process check is unhealthy. Give the liveness probe an initial delay longer than the server's start, or use a startup probe.
  • A stopped server is not alive. During a graceful shutdown the server stops, so the process check turns unhealthy; that is what a liveness probe should see.
  • The checks run synchronously. Each run makes one storage query (the server list) and, where the storage has a clock, one read of it. Probes every few seconds cost little; the dashboard's /health report is heavier and is cached instead.

See also​