OpenTelemetry
UseTelemetry() makes Masterly.NightShift emit traces and metrics through the standard .NET ActivitySource and Meter APIs, so OpenTelemetry (or any other listener) can export them. Use it when you want to see a background job in the same trace as the web request that created it, and to chart job counts, durations and concurrency in your metrics backend.
Quick example
Turn telemetry on in the Masterly.NightShift configuration, then subscribe to the Masterly.NightShift source and meter in the OpenTelemetry SDK.
using Masterly.NightShift;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddNightShift(config => config
.UseSqlServerStorage(builder.Configuration.GetConnectionString("NightShift")!)
.UseTelemetry());
builder.Services.AddNightShiftServer();
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService("checkout-api"))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddSource(NightShiftTelemetry.ActivitySourceName)
.AddOtlpExporter())
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddMeter(NightShiftTelemetry.MeterName)
.AddOtlpExporter());
The OpenTelemetry part needs these packages in your app (Masterly.NightShift itself does not reference OpenTelemetry):
OpenTelemetry.Extensions.Hosting(AddOpenTelemetry,WithTracing,WithMetrics)OpenTelemetry.Exporter.OpenTelemetryProtocol(AddOtlpExporter), or any other exporterOpenTelemetry.Instrumentation.AspNetCore(AddAspNetCoreInstrumentation), optional, for the request spans that jobs join
A job enqueued while handling a request now shows up in that request's trace:
public class CheckoutController
{
private readonly IBackgroundJobClient _jobs;
public CheckoutController(IBackgroundJobClient jobs)
{
_jobs = jobs;
}
public void PlaceOrder(int orderId)
{
// Runs inside the ASP.NET Core request span: the "create" span and the job's
// "process" span join the request's trace.
_jobs.Enqueue<OrderService>(x => x.ConfirmOrder(orderId));
}
}
How it works
UseTelemetry() adds one global filter (Diagnostics/TelemetryFilter, internal) that is a client filter, a server filter and an apply-state filter:
- Creating a job. When a job is created (
Enqueue,Schedule,ContinueJobWith, a recurring job firing, and so on), the filter starts acreate {Type}.{Method}span of kindProducer. Then it stores the trace context in two job parameters:TraceParent(the W3Ctraceparentvalue) andTraceState(only when the trace has atracestate). The span ends when the job has been written to storage. - Performing a job. When a worker moves the job from Enqueued to Processing, the filter notes the queue of the Enqueued state, which is the queue the worker took the job from. When the worker then performs the job, the filter reads
TraceParent/TraceStateback and starts aprocess {Type}.{Method}span of kindConsumerwhose parent is the stored context. The span is the currentActivitywhile your job method runs, so spans your own code creates (HttpClient, EF Core, your ownActivitySource) become its children. It ends after the job method and the server filters inside it have finished. - Metrics. The same filter counts created jobs, measures how long each execution took, and tracks how many jobs are running right now.
The trace context is stored even when nobody listens to the Masterly.NightShift source: if there is a W3C Activity.Current when the job is created (for example the ASP.NET Core request span), its id is saved, so the job joins the caller's trace as soon as a listener is attached on the server side.
When nothing listens, the cost is close to zero. The filter checks ActivitySource.HasListeners() and each instrument's Enabled flag before doing any work, and builds no span names, job names or outcomes then. It reads the Enqueued state (one storage query per job start) and the job parameters on the server only when someone listens to the source.
Every attempt of a job is its own process span. A job that is retried three times produces three process spans, all children of the same create span.
Options / API
| Member | Type | Meaning |
|---|---|---|
IGlobalConfiguration.UseTelemetry() | extension method | Adds the telemetry filter to GlobalJobFilters. Returns the configuration for chaining. |
NightShiftTelemetry.ActivitySourceName | const string = "Masterly.NightShift" | Name of the ActivitySource. Pass it to AddSource. |
NightShiftTelemetry.MeterName | const string = "Masterly.NightShift" | Name of the Meter. Pass it to AddMeter. |
The source and the meter both report the version of the Masterly.NightShift.Core assembly as their version.
Spans
| Span name | Kind | Started when | Ends when |
|---|---|---|---|
create {Type}.{Method} | Producer | a client filter sees the job being created | the job is stored (or creation failed or was canceled) |
process {Type}.{Method} | Consumer | a worker starts performing the job | the job method and the inner server filters finished |
{Type} is the short name of the job's type (no namespace), {Method} the method name, for example process EmailService.SendWelcome.
Span attributes
| Attribute | On | Value |
|---|---|---|
messaging.system | both | always nightshift |
messaging.operation.name | both | create or process |
messaging.operation.type | both | create or process |
messaging.destination.name | both | the queue. On create: for a job created in the Enqueued state, the queue it is added to, after state election, so a queue chosen by [Queue] is reported; for other initial states (Scheduled, Awaiting, ...), the queue passed when creating the job (the queue argument of Schedule, ...). On process: the queue the worker took the job from. Not set when unknown. |
messaging.message.id | both | the background job id. On create it is added when the job has been stored. |
nightshift.recurring_job.id | process | the id of the recurring job that created the run (PerformContext.RecurringJobId); not set for other jobs |
nightshift.job.name | both | {Type}.{Method}, the same value as in the span name |
code.function.name | both | {Namespace.Type}.{Method}, the full type name and the method |
nightshift.job.state | create | the initial state: Enqueued, Scheduled, Awaiting, ... |
nightshift.job.outcome | process | succeeded, failed, canceled, retry_later or skipped (see below) |
error.type | both, on failure | the full type name of the exception |
On failure the span status is set to Error with the exception message, and an exception event is added (exception.type, exception.message, exception.stacktrace). Apps on .NET 10 get the package's net10.0 build, which adds it with Activity.AddException. Apps on .NET 8 and .NET 9 get the net8.0 build (there is no net9.0 build), which adds the same event by hand. Exceptions thrown by the job method reach server filters wrapped in a JobPerformanceException; the span reports the original exception instead.
The outcome of a process span is:
| Outcome | When |
|---|---|
succeeded | the job method returned, or a server filter marked its exception as handled |
canceled | a server filter canceled the execution, a state change aborted the job while it ran (it was deleted or requeued, for example), or the server shut down while it ran and the job stopped with an OperationCanceledException; the job is enqueued again after a shutdown. Also a job that threw TerminateJobException with TerminateJobOutcome.Canceled; the span status is not Error |
retry_later | the job threw RetryLaterException to ask to run again later; the span status is not Error |
skipped | the job threw TerminateJobException (outcome Skipped, the default), or [SkipIfLate] found it started too late; it is deleted without a retry, and the span status is not Error. A job terminated as Failed has the outcome failed |
failed | any other exception, including an OperationCanceledException the job threw while the server was not shutting down (a timeout of its own, for example); the span status is Error. That includes an attempt whose method never ran because [Throttle], [RateLimit] or [DisableConcurrentExecution] gave up waiting for a slot or a lock, because the worker records that attempt as failed too: error.type (Masterly.NightShift.ConcurrencyLimitException, Masterly.NightShift.Storage.DistributedLockTimeoutException) tells it apart, and its duration includes the wait. Insights leaves such an attempt out |
A create span is marked as failed when creating the job threw an exception that no client filter handled, for example because the storage was down.
Instruments
| Name | Type | Unit | Tags | Meaning |
|---|---|---|---|---|
nightshift.job.created | Counter<long> | {job} | nightshift.job.name, nightshift.job.state, error.type (has a value only when creation failed) | Background jobs created. Jobs whose creation was canceled by a client filter are not counted. |
nightshift.job.duration | Histogram<double> | s | nightshift.job.name, nightshift.job.outcome, error.type (has a value only when the outcome is failed) | Duration of each job execution, including the server filters that run inside the telemetry filter. |
nightshift.job.active | UpDownCounter<long> | {job} | nightshift.job.name | Jobs being performed right now. |
nightshift.job.completed | Counter<long> | {job} | nightshift.job.name, nightshift.job.outcome, error.type | Job executions that ended, by outcome: succeeded, failed, skipped, canceled, retry_later. |
nightshift.job.retried | Counter<long> | {job} | nightshift.job.name | Failed attempts that [AutomaticRetry] scheduled for a retry, counted once the retry is committed. |
nightshift.job.stale_restarts | Counter<long> | {job} | nightshift.job.name, nightshift.job.outcome = restarted or failed | Jobs picked up again after their server stopped while they ran, counted once the state change is committed: restarted when the job runs again, failed when the restart went past UseStaleRestartLimit and failed it (see Reliability). |
nightshift.job.lateness | Histogram<double> | s | nightshift.job.name | How late delayed and recurring jobs started after the moment they were scheduled for (PerformContext.Lateness), read from the parameters loaded with the job, as SQL Server loads them. Only a job's first run is recorded: retries and runs picked up again after their server stopped aren't, as their lateness includes the earlier attempts. Fire-and-forget jobs aren't recorded. The recurring job isn't a tag, so the number of series stays bounded; the process span has it in nightshift.recurring_job.id. |
A process span of a run that was skipped or canceled gets an event, nightshift.job.skipped or nightshift.job.canceled, with the reason in nightshift.job.reason.
In the package's net10.0 build, which apps on .NET 10 get, the histogram advises these bucket boundaries, in seconds, because background jobs run from milliseconds to hours: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 300, 900, 3600. Apps on .NET 8 and .NET 9 get the net8.0 build (there is no net9.0 build), which creates the histogram without advice, so the SDK uses its default boundaries; see Histogram buckets on .NET 8 and .NET 9.
nightshift.job.name has one value per job method, so the number of series stays bounded. The job id is only on spans, never on metrics.
More examples
A console app or Worker Service without the generic host's DI
The OpenTelemetry SDK can be built by hand. Keep the providers alive for as long as the server runs, and dispose them at the end so the last data is flushed. This needs OpenTelemetry and an exporter package (here OpenTelemetry.Exporter.Console).
using Masterly.NightShift;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using TracerProvider tracerProvider = Sdk.CreateTracerProviderBuilder()
.ConfigureResource(resource => resource.AddService("report-worker"))
.AddSource(NightShiftTelemetry.ActivitySourceName)
.AddConsoleExporter()
.Build();
using MeterProvider meterProvider = Sdk.CreateMeterProviderBuilder()
.ConfigureResource(resource => resource.AddService("report-worker"))
.AddMeter(NightShiftTelemetry.MeterName)
.AddConsoleExporter()
.Build();
GlobalConfiguration.Configuration
.UseSqlServerStorage(@"Server=.\sqlexpress;Database=NightShift;Integrated Security=True;TrustServerCertificate=True;")
.UseTelemetry();
using (var server = new BackgroundJobServer())
{
Console.WriteLine("Press Enter to stop.");
Console.ReadLine();
}
Separate web and worker processes
When the web app only creates jobs and a separate process performs them, call UseTelemetry() in both. The web app's filter writes TraceParent when the job is created; the worker's filter reads it and starts the process span. Register AddSource(NightShiftTelemetry.ActivitySourceName) in both if you want the create spans as well; the worker needs it to produce process spans at all. If only the worker calls UseTelemetry(), jobs created by the web app have no stored trace context and each execution starts a new trace.
Histogram buckets on .NET 8 and .NET 9
The package's net8.0 build, which apps on .NET 8 and .NET 9 use, creates the histogram without bucket advice: the advice API needs .NET 9, and the package has no net9.0 build. The SDK then uses its default boundaries, which top out at 10 seconds. Set the boundaries with a view. It does no harm on .NET 10, where the view replaces the advice:
builder.Services.AddOpenTelemetry()
.WithMetrics(metrics => metrics
.AddMeter(NightShiftTelemetry.MeterName)
.AddView("nightshift.job.duration", new ExplicitBucketHistogramConfiguration
{
Boundaries = new double[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 300, 900, 3600 }
})
.AddOtlpExporter());
Sampling
The process span's parent is the stored context, including its "sampled" flag. With a parent-based sampler, a job is traced exactly when the request that created it was traced:
builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.1)))
.AddAspNetCoreInstrumentation()
.AddSource(NightShiftTelemetry.ActivitySourceName)
.AddOtlpExporter());
Your own spans inside a job
Activity.Current is the process span while the job method runs, so a span you start becomes its child:
using System.Diagnostics;
public class ReportService
{
private static readonly ActivitySource Source = new ActivitySource("MyCompany.Reports");
public void BuildDaily()
{
// Activity.Current is the job's "process" span here, so this span becomes its child.
using (var activity = Source.StartActivity("load sales"))
{
activity?.SetTag("report.kind", "daily");
}
}
}
Add AddSource("MyCompany.Reports") to the tracing setup to export it.
Listening without OpenTelemetry
The source and the meter are plain .NET APIs. In a test or a quick diagnostic you can listen with ActivityListener and MeterListener directly, with no extra packages:
using System.Diagnostics;
using System.Diagnostics.Metrics;
using Masterly.NightShift;
public static class RawListeners
{
public static IDisposable ListenToSpans()
{
var listener = new ActivityListener
{
ShouldListenTo = source => source.Name == NightShiftTelemetry.ActivitySourceName,
Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
ActivityStopped = activity => Console.WriteLine(
$"{activity.DisplayName} {activity.Kind} {activity.Duration.TotalMilliseconds:N0} ms " +
$"trace={activity.TraceId} outcome={activity.GetTagItem("nightshift.job.outcome")}")
};
ActivitySource.AddActivityListener(listener);
return listener;
}
public static IDisposable ListenToMetrics()
{
var listener = new MeterListener
{
InstrumentPublished = (instrument, meterListener) =>
{
if (instrument.Meter.Name == NightShiftTelemetry.MeterName)
{
meterListener.EnableMeasurementEvents(instrument);
}
}
};
listener.SetMeasurementEventCallback<double>((instrument, value, tags, state) =>
Console.WriteLine($"{instrument.Name} = {value:N3} {instrument.Unit}"));
listener.SetMeasurementEventCallback<long>((instrument, value, tags, state) =>
Console.WriteLine($"{instrument.Name} += {value}"));
listener.Start();
return listener;
}
}
Things to know
- Nothing exported? Check, in this order:
UseTelemetry()is called in the process that performs the jobs (not only in the web app); the OpenTelemetry setup hasAddSource(NightShiftTelemetry.ActivitySourceName)for spans andAddMeter(NightShiftTelemetry.MeterName)for metrics; your sampler keeps the spans (with a parent-based sampler, a job is traced only when its creator was). - Call it once, early.
UseTelemetry()adds a global filter. Put it in theAddNightShift(...)callback or next to your otherGlobalConfigurationcalls, in every process that creates or performs jobs. - Jobs created before telemetry was on (or by a process without
UseTelemetry()) have noTraceParentparameter. Theirprocessspan starts a new trace. The same happens when there was noActivityat creation time, for example for jobs created by the recurring job scheduler, unless thecreatespan itself is listened to, in which case the job continues thecreatespan's trace. - Only W3C ids are propagated. An
Activityusing the legacy hierarchical id format is not stored. W3C is the default on modern .NET. - The span covers the server filters inside the telemetry filter, not only the job method.
- The queue on
processspans comes from the Enqueued state that the worker took the job from. If the filter was not listening when the job moved to Processing, or that state could not be read, the span falls back to the queue passed when creating the job, or has nomessaging.destination.name. A job created withSchedulegets its queue when it is enqueued later, so itscreatespan shows only a queue passed explicitly. - Retries: each attempt is a separate
processspan and a separatenightshift.job.durationmeasurement. A failed attempt that will be retried has the outcomefailed. An attempt that threwRetryLaterExceptionhas the outcomeretry_later, even when the job has already asked more thanMaxRetryLaterRequeststimes andAutomaticRetryhandles it as a regular failure. - Job parameters:
TraceParentandTraceStateare visible among the job's parameters on the dashboard's job details page. Do not use these names for your own parameters. - This is not the dashboard's
/metricsendpoint. The dashboard also serves Prometheus text at/metrics, built from storage queries (job counts by state, queue lengths, Insights data). See Insights. TheMeterdescribed here reports what the current process does;/metricsreports what is in the storage.
See also
- Monitoring overview
- Insights: per-method statistics stored for the dashboard and
/metrics - Logging
- Custom filters: how client and server filters wrap a job
- Global configuration
- Dependency injection:
AddNightShiftandAddNightShiftServer