Timing
These filters deal with time: how long a job may run (JobTimeout), when it should be done by (Deadline), how long it may wait before it is pointless (LatencyTimeout, SkipIfLate), what time of day it may start (RunWindow, Blackout), and how long its data is kept after it finishes (ExpireAfter).
Quick example
using Masterly.NightShift;
public class ReportService
{
// Canceled and failed if it runs longer than 10 minutes.
[JobTimeout(600)]
public async Task BuildDaily(CancellationToken cancellationToken)
{
await Task.Delay(TimeSpan.FromMinutes(1), cancellationToken);
}
}
jobs.Enqueue<ReportService>(x => x.BuildDaily(CancellationToken.None));
Pass CancellationToken.None when you create the job; the server substitutes a real token when it runs it (see Cancellation).
Which one should I use?
| Attribute | Counts from | What happens | Cancels the job? |
|---|---|---|---|
JobTimeout | The start of the run | The job's cancellation token fires; the job fails with TimeoutException and is retried | Yes, if the method observes its token |
Deadline | The job's creation | Nothing is stopped; late jobs are flagged in the dashboard and a late finish is recorded and alerted | No |
LatencyTimeout | The job's creation | A job picked up too late is deleted instead of run | It never starts |
SkipIfLate | The moment the job was meant to run (ScheduledFor), when a continuation became due, or its creation | A run that starts too late is deleted as skipped; checked once, so a retry or a requeue runs | It never starts |
RunWindow / Blackout | The clock | A job picked up at the wrong time of day is rescheduled for the next allowed moment | It never starts until allowed |
| Schedule policy (not an attribute) | The job's due time, or a recurring occurrence | A stored business calendar (days, hours, holidays) moves, skips or rejects a delayed job and leaves out blocked recurring occurrences | It never starts until allowed |
ExpireAfter | The job's end | Changes how long a finished job is kept in storage; with IncludeFailed, also a failed one | No |
JobTimeoutAttribute
Cancels a job that runs longer than the given number of seconds and fails it with a TimeoutException (message: Timed out after 10 minutes (JobTimeout)). AutomaticRetry then retries it like any other failure, unless OnTimeout says otherwise.
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
timeoutInSeconds (constructor) | int | required | Maximum run time in seconds. From 1 to JobTimeoutAttribute.MaxTimeoutInSeconds (2,147,483 seconds, about 24.8 days); other values throw ArgumentOutOfRangeException when the attribute is read. |
Timeout | TimeSpan | The timeout as a TimeSpan. | |
OnTimeout | JobTimeoutAction | Retry | What happens to a job that ran out of time: Retry fails it with the TimeoutException, and AutomaticRetry retries it. Fail fails it without retries. Cancel deletes it with the reason Canceled: Timed out after ..., without retries. An unknown value throws ArgumentOutOfRangeException. |
Order | int | 0 | Position among interceptors; lower runs outermost. |
Related members: JobTimeoutAttribute.TimeoutExceededParameter ("TimeoutExceeded") and MaxTimeoutInSeconds (2147483). The limit is Int32.MaxValue milliseconds, about 24.8 days, as in Hangfire. For longer work, use a Deadline, which cancels nothing.
The method must accept a cancellation parameter and observe it. A method without one can't be interrupted: it runs to the end and succeeds, and the overrun is recorded in the TimeoutExceeded job parameter. If the storage fails to record it, a warning is logged and the job still succeeds.
Fail and Cancel end the job with a TerminateJobException that keeps the TimeoutException as its inner exception. Use Cancel for work that is pointless once it ran too long, such as a preview that a user stopped waiting for:
public class PreviewService
{
[JobTimeout(30, OnTimeout = JobTimeoutAction.Cancel)]
public Task RenderPreview(int documentId, CancellationToken cancellationToken) => Task.CompletedTask;
}
With IJobCancellationToken
public class ReportService
{
[JobTimeout(1800)]
public void ExportLedger(int year, IJobCancellationToken cancellationToken)
{
for (var month = 1; month <= 12; month++)
{
cancellationToken.ThrowIfCancellationRequested();
ExportMonth(year, month);
}
}
private static void ExportMonth(int year, int month) { }
}
Pass the token down
using System.IO;
using System.Net.Http;
public class ImageService
{
private readonly HttpClient _http;
public ImageService(HttpClient http)
{
_http = http;
}
[JobTimeout(120)]
public async Task DownloadOriginal(string url, string path, CancellationToken cancellationToken)
{
var bytes = await _http.GetByteArrayAsync(url, cancellationToken);
await File.WriteAllBytesAsync(path, bytes, cancellationToken);
}
}
How JobTimeout works
JobTimeoutAttribute is an interceptor (AsyncJobInterceptorAttribute), so it runs inside the performer around the method call, without a storage round trip. It creates a token that is canceled after the timeout, when the server shuts down, or when the token already passed to the method is canceled (an outer JobTimeout, for example), and passes it to the method in place of every CancellationToken parameter. An IJobCancellationToken parameter is wrapped so that ThrowIfCancellationRequested() and its ShutdownToken also see the timeout. When the method throws OperationCanceledException because this timeout expired (not because of shutdown, because the job was deleted, or because another timeout expired first), the interceptor throws TimeoutException instead.
DeadlineAttribute
Gives a job a deadline counted from its creation. It is a service-level expectation, not a timeout: nothing is canceled. The dashboard's Processing page and job details page highlight jobs that are still waiting or running past their deadline, and a job that finishes late keeps a record of how late it was.
using Masterly.NightShift;
public class OrderService
{
// Order confirmations should be out within 10 minutes of the order.
[Deadline("00:10:00")]
public Task ConfirmOrder(int orderId) => Task.CompletedTask;
// The same, in seconds.
[Deadline(3600)]
public Task SendInvoice(int orderId) => Task.CompletedTask;
}
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
timeout (constructor) | string | required | Time after creation, parsed as a TimeSpan with the invariant culture: "00:10:00", "1.00:00:00". Text that isn't a time span throws FormatException; zero or less throws ArgumentOutOfRangeException. |
timeoutInSeconds (constructor) | int | required | The same, in seconds. Zero or less throws ArgumentOutOfRangeException. |
Timeout | TimeSpan | The parsed deadline. | |
Order | int | -1 | Filter order. |
Related members: DeadlineAttribute.DeadlineAtParameter ("DeadlineAt"), DeadlineMissedParameter ("DeadlineMissed") and ReadDeadline(connection, jobId), which returns a job's deadline or null.
How it works: a client filter and an election filter. OnCreating stores creation time plus Timeout (UTC) in the DeadlineAt job parameter. When Succeeded is elected after that moment, the filter stores how late the job was (a TimeSpan) in DeadlineMissed and raises a DeadlineMissed alert if alert sinks are configured (Alerts and webhooks). If the storage fails to store DeadlineMissed, a warning is logged and the job still succeeds; the alert is still raised. Jobs that end Failed or Deleted are not marked.
Deadline plus a hard limit
public class OrderService
{
// Should be done within 5 minutes of the order; a single run may take at most 4.
[Deadline("00:05:00")]
[JobTimeout(240)]
public Task ReserveStock(int orderId, CancellationToken cancellationToken) => Task.CompletedTask;
}
LatencyTimeoutAttribute
Deletes a job that waited too long before a worker picked it up. Use it for work that is useless when late: a flash-sale alert, a one-time code, a live notification.
using Masterly.NightShift;
public class NotificationService
{
// Not worth sending if it couldn't start within 5 minutes of being created.
[LatencyTimeout(300)]
public Task SendFlashSaleAlert(int userId, int saleId) => Task.CompletedTask;
}
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
timeoutInSeconds (constructor) | int | required | Maximum time between creation and start, in seconds. 0 allowed; negative values throw. |
TimeoutInSeconds | int | The value as given. | |
LogLevel | Masterly.NightShift.Logging.LogLevel | Debug | Level of the log message written when a job is deleted. |
Order | int | -1 | Filter order. |
How it works: an election filter. When a worker tries to move the job to Processing and more than TimeoutInSeconds have passed since the job's creation time, the candidate becomes Deleted with the reason Background job has exceeded latency timeout of 300 second(s), and the job never runs.
Log deletions as warnings
In ASP.NET Core, LogLevel clashes with Microsoft.Extensions.Logging.LogLevel, so use an alias:
using Masterly.NightShift;
using NightShiftLogLevel = Masterly.NightShift.Logging.LogLevel;
public class NotificationService
{
[LatencyTimeout(60, LogLevel = NightShiftLogLevel.Warn)]
[AutomaticRetry(Attempts = 0)]
public Task SendOneTimeCode(int userId) => Task.CompletedTask;
}
SkipIfLateAttribute
Skips a run that starts too late. A report due at 09:00 that only gets a worker at 11:30, after an outage, is usually better skipped than sent:
using Masterly.NightShift;
public class ReportService
{
[SkipIfLate("00:30:00")]
public Task SendMorningReport() => Task.CompletedTask;
}
RecurringJob.AddOrUpdate<ReportService>("morning-report", x => x.SendMorningReport(), "0 9 * * *");
When the run starts more than the window after the moment the job was meant to run, it is deleted with the reason Skipped: started 2.5 h late, more than the 30 min allowed, without running and without a retry. Continuations that wait for the Deleted state start, and telemetry reports the run as skipped.
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
window (constructor) | string | required | The longest lateness allowed, as a time span such as "00:30:00" or "1.00:00:00". Text that isn't a time span throws ArgumentException; zero or a negative span throws ArgumentOutOfRangeException. |
windowInSeconds (constructor) | int | required | The same in seconds; it must be positive. |
Window | TimeSpan | The window. |
Related members: SkipIfLateAttribute.CheckedParameter ("LateStartChecked") and SkipIfLateAttribute.WaitedForPolicyParameter ("WaitedForPolicy").
"Meant to run" is PerformContext.ScheduledFor: the time a delayed job was scheduled for, or the occurrence of a recurring job. A continuation (a fan-in continuation or a chain step too) is measured from the moment its parents let it leave the Awaiting state, not from its creation, so the time its parents waited and ran doesn't make it late. Any other job without it, such as a fire-and-forget job, is measured from its creation, so the attribute also skips a job that waited too long in its queue.
How it works: a server filter. Just before the job method would run, it compares PerformContext.Lateness (which leaves out the time a paused or fenced server held the job, see Drain mode) with the window and records the check in the LateStartChecked job parameter, so each job is checked once. A retry of a job that passed the check runs however late it is, and so does a job requeued by hand after it was skipped, from the dashboard or with Requeue. The check costs one parameter write per job that has the attribute; when that write fails, a warning is logged and the job still runs (a retry is then checked again). A job that also has a SkipAfter window (RecurringJobOptions.SkipAfter or ScheduleOptions.SkipAfter) is checked against the stricter of the two, and the check is recorded once.
A job that is late because it waited for a schedule policy that couldn't be evaluated is skipped all the same: the window is the job's own rule, whatever made the job late. Its reason says what did, Skipped: started 3 h late: waited for policy 'business-hours', which couldn't be evaluated, and one warning with the policy's name is logged. The reason has at most 99 characters, so a long policy name is shortened in it. The same goes for a run of a recurring job that was due while the recurring job waited for its policy: the scheduler gives such a run the WaitedForPolicy job parameter, the policy's name.
SkipIfLate and LatencyTimeout differ in what they count from: LatencyTimeout counts from the job's creation and checks every run, so it can't be used on a delayed or recurring job whose wait is planned; SkipIfLate counts from the moment the job was meant to run, and checks once.
RunWindowAttribute and BlackoutAttribute
[RunWindow] lets a job start only inside a daily time window; [Blackout] keeps it from starting inside one. For calendars shared by many jobs, with days, holidays and date providers, stored and editable without a deployment, use schedule policies. A job picked up at the wrong time goes back to the Scheduled state for the next allowed moment, with the reason in its history, so heavy work waits for the night and customer-facing work waits for office hours without any code in the job.
using Masterly.NightShift;
public class WarehouseService
{
// Only at night, Berlin time. Overnight windows (end before start) are fine.
[RunWindow("22:00-06:00", "Europe/Berlin")]
public Task RebuildAggregates() => Task.CompletedTask;
// Only in office hours (UTC), Monday to Friday.
[RunWindow("09:00-17:00", WeekdaysOnly = true)]
public Task CallSupplier(int supplierId) => Task.CompletedTask;
// Never during the nightly database maintenance (UTC).
[Blackout("02:00-04:00")]
public Task ImportOrders() => Task.CompletedTask;
// Spare the trading day in New York; at the weekend it may run at any time.
[Blackout("09:30-16:00", "America/New_York", WeekdaysOnly = true)]
public Task RecalculatePortfolio(int portfolioId) => Task.CompletedTask;
}
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
window (constructor) | string | required | start-end, each as HH:mm, H:mm, HH:mm:ss or a bare hour ("22-6"). The start is inclusive, the end exclusive. Any other form, or a start equal to the end, throws ArgumentException when the attribute is read. |
timeZoneId (constructor) | string? | null (UTC) | Time zone the window is expressed in, looked up with TimeZoneInfo.FindSystemTimeZoneById. |
Window | string | The window as given, without surrounding spaces. | |
Start, End | TimeSpan | The parsed times of day. End is before Start for an overnight window. | |
TimeZoneId | string | "UTC" | The resolved time zone id. |
WeekdaysOnly | bool | false | Limits the rule to Monday to Friday, in the window's time zone. RunWindow: also keeps the job off Saturdays and Sundays. Blackout: the blackout applies on weekdays only, and the job may start at any time at the weekend. The day is the one of the moment checked, so the hours after midnight of an overnight window belong to the next day: [Blackout("22:00-06:00", WeekdaysOnly = true)] blocks Friday 22:00 to midnight and Monday 00:00 to 06:00, but not Saturday morning. |
Order | int | 15 | Filter order. |
How it works: an election filter. When the candidate state is Processing, the filter checks the current time in the window's time zone, taken from the storage's clock where the storage has one: the delayed job scheduler finds jobs due by that clock, so a server whose own clock is behind doesn't put the job back for a moment that is already due. If the job may not start, the candidate becomes Scheduled for the next allowed moment, with a reason such as Outside the run window 22:00-06:00 (Europe/Berlin); runs at 2026-11-06 21:00 UTC or Blackout 02:00-04:00 (UTC); runs at .... Daylight-saving changes are handled: a window that opens inside the skipped hour starts at the end of the gap, and a window that opens inside the hour repeated when clocks go back starts at the first of the two moments that is still ahead (the daylight-saving one, then the standard one), so the job neither starts an hour late nor skips the night. GetNextStart(utcNow, out bool allowedNow) is public, so you can ask the same question from code:
var window = new RunWindowAttribute("22:00-06:00", "Europe/Berlin");
DateTime next = window.GetNextStart(DateTime.UtcNow, out var allowedNow);
RunWindow and Blackout are different types, so you can put both on one method, for example a night window with a blackout for a backup inside it. Each can appear only once per job, and a method attribute replaces a class attribute of the same type. For two separate blackouts on one job, write a custom election filter.
ExpireAfterAttribute
Sets how long a job's data is kept after it finishes, per method, instead of the storage-wide default (JobStorage.JobExpirationTimeout, one day unless you changed it with WithJobExpirationTimeout).
using Masterly.NightShift;
public class AuditService
{
// Audit-relevant: keep for a month.
[ExpireAfter(days: 30)]
public Task RecordLogin(int userId) => Task.CompletedTask;
// Chatty: drop after ten minutes.
[ExpireAfter("00:10:00")]
public Task PingHealthEndpoint() => Task.CompletedTask;
// A week, written as d.hh:mm:ss.
[ExpireAfter("7.00:00:00")]
public Task ExportGdprReport(int userId) => Task.CompletedTask;
}
| Parameter / property | Type | Default | Meaning |
|---|---|---|---|
timeSpan (constructor) | string | required | How long to keep finished jobs, as hh:mm:ss or d.hh:mm:ss. Must be positive and at most ExpireAfterAttribute.MaxDays (36,500 days, about 100 years); otherwise ArgumentException. |
days (constructor) | int | required | The same, in days: from 1 to MaxDays; otherwise ArgumentOutOfRangeException. |
Duration | TimeSpan | The parsed duration. | |
IncludeFailed | bool | false | Also expires the job after Duration when it ends up Failed, that is when its retries are used up. |
Order | int | 50 | Filter order. |
The limit exists because storages add the duration to the current time, and a longer one would overflow the date when the state change is saved.
How it works: an apply filter. When the new state is final (Succeeded or Deleted), it sets ApplyStateContext.JobExpirationTimeout in OnStateUnapplied, which runs before any filter's OnStateApplied. So everything that expires with the job uses the per-method value, whatever its filter order: the job's console lines (expired by the global JobConsoleStateFilter) and the job itself, which the state machine expires last.
Failed jobs
By default a failed job never expires: it stays until someone retries or deletes it. Teams with flaky integrations can end up with thousands of them. Give them a retention period per method with IncludeFailed, or for every job with UseFailedJobRetention.
using Masterly.NightShift;
// Every job: once its retries are used up, a failed job is kept for 30 days.
GlobalConfiguration.Configuration.UseFailedJobRetention(TimeSpan.FromDays(30));
public class PartnerSync
{
// This method: succeeded and failed runs are both kept for a week.
[AutomaticRetry(Attempts = 5)]
[ExpireAfter(days: 7, IncludeFailed = true)]
public Task PushOrders() => Task.CompletedTask;
}
- Only a final failure expires. While
AutomaticRetryhas attempts left, a failure becomes a retry (Scheduled), which never expires. The expiration is set when the job stays Failed, in the same transaction as that state. - Leaving Failed cancels it. Requeue, reschedule or retry the job (dashboard or API) before it expires and it is kept again, like any job that isn't finished. Deleting it uses the normal expiration of deleted jobs.
- The most specific setting wins.
[ExpireAfter(..., IncludeFailed = true)]on the method or class wins overUseFailedJobRetention. An[ExpireAfter]withoutIncludeFailedleaves failed jobs toUseFailedJobRetention(or keeps them, without it). - Everything goes. The job, its history, parameters and console lines expire together. On SQL Server the Failed page and count read the job table, so an expired failed job disappears from both once the expiration manager deletes it; the
stats:failedhistory counters are not affected. - Continuations and batches keep waiting. A continuation of the failed job stays Awaiting when the parent expires, as it does while the parent stays failed (continuations start when the parent succeeds or is deleted). A batch keeps counting the job as failed. Retry or delete the job before it expires, or delete the continuation.
- Other storages. The job is expired with
IWriteOnlyTransaction.ExpireJob, which every storage implements. A storage that keeps its own index of failed jobs and doesn't clean it up when a job expires may keep a stale entry there; check its documentation.
Things to know
LatencyTimeoutandDeadlinecount from creation. For a job scheduled an hour ahead, an hour has already passed when it becomes due;[LatencyTimeout(600)]on it would delete it every time. Use[SkipIfLate]for delayed and recurring jobs. Retry back-offs, rescheduling byThrottleorRunWindow, and time spent in a paused queue all count too.JobTimeoutneeds a cancellation parameter. Without one the job is not stopped, only marked withTimeoutExceeded.JobTimeoutandRetryInProcessnest byOrder. Both default toOrder = 0; setOrderto choose which one is outermost. With the timeout outside, it covers all in-process attempts; inside, each attempt gets its own timeout, and aTimeoutExceptionis retried in process because it is not anOperationCanceledException.- Interceptors are not de-duplicated. A
[JobTimeout]on the class and another on the method both apply, nested, with aCancellationTokenparameter as with anIJobCancellationTokenone: the token the method gets is canceled by whichever timeout expires first, and theTimeoutExceptionnames that one (Timed out after 1 seconds (JobTimeout)). For a method without a cancellation parameter,TimeoutExceededrecords the shorter timeout that was exceeded. - Time zone ids must exist on the server. An unknown id throws
TimeZoneNotFoundExceptionwhen the attribute is read, which breaks creating and processing the job. Both IANA (Europe/Berlin) and Windows (W. Europe Standard Time) ids usually work. See Time zones and misfires. - A rescheduled job doesn't use a retry attempt.
RunWindowmoves the job to Scheduled during election, before it runs. - A rescheduled job goes back to its queue. When a job's queue came only from its Enqueued state (
new EnqueuedState("emails")),RunWindowremembers it in theHeldQueuejob parameter, like a job held by drain mode, and puts the job back in that queue when the window opens, also whenDrainModeFilterwas removed from the global filters. For a job requeued while it ran (still in the Processing state), the queue is read from the last Enqueued state in its history.