Skip to main content

Job filters

Filters are attributes and classes that run around a background job's life: when it is created, when a state change is chosen, when the change is written, and when the job runs. Retries, queues, concurrency limits, timeouts and run windows are all filters. You add one to a job by putting an attribute on its method or class, or to every job by registering it globally.

PageWhat it covers
RetriesAutomaticRetry (the default retry policy), AttemptsExceededAction, RetryLaterException, TerminateJobException, RetryInProcess, CircuitBreaker
ConcurrencyDisableConcurrentExecution, Throttle, RateLimit, ConcurrencyAction, PreventDuplicates, IdempotentCompletion
TimingJobTimeout, Deadline, LatencyTimeout, SkipIfLate, RunWindow, Blackout, ExpireAfter
BehaviorQueue, JobDisplayName, CaptureCulture, LogInvocation, Redacted
Custom filtersWriting client, server, election, apply and exception filters, interceptors and state handlers, and registering them

Quick example​

using Masterly.NightShift;

public class EmailService
{
[Queue("critical")]
[AutomaticRetry(Attempts = 3)]
public void SendPasswordReset(int userId)
{
// ...
}
}

// In a controller or service that gets IBackgroundJobClient from DI:
jobs.Enqueue<EmailService>(x => x.SendPasswordReset(42));

The job goes to the critical queue and is retried at most three times instead of the default ten.

Which one should I use?​

I want to...UsePage
Change how often and how long a failed job is retried[AutomaticRetry]Retries
Let the job say "run me again at this time"throw RetryLaterExceptionRetries
End a job without retries: nothing to do, or it can never workthrow TerminateJobExceptionRetries
Retry a short glitch (deadlock, dropped connection) at once, in the same worker[RetryInProcess]Retries
Stop running jobs against a dependency that keeps failing[CircuitBreaker]Retries
Run at most N copies at a time[Throttle] (or [DisableConcurrentExecution] for a blocking lock)Concurrency
Start at most N jobs per minute (an API quota)[RateLimit]Concurrency
Not queue the same job twice[PreventDuplicates]Concurrency
Not redo a job whose success was already recorded[IdempotentCompletion]Concurrency
Stop a run that takes too long[JobTimeout]Timing
See jobs that finish later than expected, without stopping them[Deadline]Timing
Drop a job that waited too long to start[LatencyTimeout]Timing
Skip a delayed or recurring run that starts too late[SkipIfLate]Timing
Run only at night, or never during a maintenance window[RunWindow], [Blackout]Timing
Keep a job's data longer or shorter after it finishes[ExpireAfter]Timing
Send a job to a specific queue[Queue]Behavior
Give a job a readable name in the dashboard[JobDisplayName]Behavior
Run a job with the culture of the request that created itCaptureCulture (on by default)Behavior
See the start, end and errors of a job in its console[LogInvocation]Behavior
Hide an argument (a password, a token) in the dashboard[Redacted]Behavior
Something none of these doyour own filterCustom filters

How it works​

A filter is an object that implements one or more filter interfaces. NightShift calls them at fixed points of the job lifecycle:

InterfaceNamespaceCalledTypical use
IClientFilterMasterly.NightShift.ClientOnCreating before a job is stored, OnCreated afterAdd job parameters, cancel creation (PreventDuplicates, Deadline, CaptureCulture)
IServerFilterMasterly.NightShift.ServerOnPerforming before the job method runs, OnPerformed afterLocks, slots, context restore (DisableConcurrentExecution, Throttle, RateLimit)
IElectStateFilterMasterly.NightShift.StatesOnStateElection while the next state is chosenReplace the candidate state (AutomaticRetry turns Failed into Scheduled, Queue sets the queue)
IApplyStateFilterMasterly.NightShift.StatesOnStateUnapplied / OnStateApplied inside the state-change transactionExtra storage writes (ExpireAfter, the retries set)
IClientExceptionFilterMasterly.NightShift.ClientWhen creating a job throwsLog or swallow creation errors
IServerExceptionFilterMasterly.NightShift.ServerWhen performing a job throwsLog or swallow job errors

Interceptors (IJobInterceptor, JobInterceptorAttribute, AsyncJobInterceptorAttribute) are a separate, newer mechanism. They wrap the call of the job method itself, inside the server filters. JobTimeout, RetryInProcess and LogInvocation are interceptors. See Custom filters.

Where a filter applies​

WhereHowScope
One methodAttribute on the job methodJobFilterScope.Method
Every method of a typeAttribute on the class or interfaceJobFilterScope.Type
Every jobGlobalConfiguration.Configuration.UseFilter(...), config.UseFilter(...) inside AddNightShift, or GlobalJobFilters.Filters.Add(...)JobFilterScope.Global

Filters are read from the type and method in the job expression. For Enqueue<IEmailService>(x => x.Send(...)) that is the interface and its method, so put the attributes on the interface, not on the class that implements it.

Most filter attributes can appear only once per job. When the same filter type is found more than once (globally, on the class and on the method), the one that sorts last wins: the one with the higher Order, and for the same Order the method attribute beats the class attribute, which beats the global one. That is how [AutomaticRetry(Attempts = 0)] on a method replaces the global retry filter: both have Order 20. Don't give an attribute a lower Order than the filter it should replace: [AutomaticRetry(Attempts = 0, Order = 5)] on a method loses to the global AutomaticRetry, and the job is retried ten times.

Order​

Within one kind of filter, filters run sorted by their Order property (lower first, -1 by default), then by scope: global, type, method. OnCreated and OnPerformed run in reverse, like nested using blocks, and exception filters run in reverse order too. Some built-in filters set their own Order: AutomaticRetry 20, PreventDuplicates 10, RunWindow 15, ExpireAfter 50, Queue Int32.MaxValue (it always decides the queue last). Details and examples are in Custom filters.

Filters registered by default​

GlobalJobFilters.Filters starts with these filters, so every job gets them without any configuration:

FilterWhat it does
JobChainStepFilterLets a step of a chain run only when its chain's record lists it (Chains)
CaptureCultureAttributeRuns the job with the culture of the code that created it (Behavior)
TerminateJobFilterEnds a job that threw TerminateJobException as skipped or canceled (Retries)
LateStartFilterSkips a run that starts later than its SkipAfter allows (Time zones and misfires)
SchedulePolicyFilterApplies the schedule policy a job was created with (Schedule policies)
AutomaticRetryAttributeRetries failed jobs up to 10 times (Retries)
StatisticsHistoryAttributeRecords succeeded and failed counts for the dashboard graphs
ContinuationsSupportAttributeStarts continuations when their parent finishes (Continuations)
FanInContinuationsSupportAttributeStarts fan-in continuations (Fan-in and batches)
JobConsoleStateFilterExpires a job's console lines together with the job
DrainModeFilterHolds jobs while the server is draining (Drain mode)
ActivityFeedFilterFeeds the dashboard activity feed once it is turned on with UseActivityFeed() (Audit log and activity)
BatchSupportAttributeTracks batch jobs (Fan-in and batches)
StaleJobFilterCounts the restarts of a job whose server stopped, and refuses a former owner's late state change (Reliability and cluster)

Removing one of these, for example GlobalJobFilters.Filters.Remove<AutomaticRetryAttribute>(), turns that feature off for every job. Removing DrainModeFilter turns off drain mode only: RunWindow, Throttle, RateLimit and CircuitBreaker still put the jobs they hold back in their queue on their own. GlobalJobFilters.Filters is a JobFilterCollection; its members are listed in Custom filters.

Filters described on other pages​

Some features are filters too, but they belong to another part of the documentation:

FilterTurned on byPage
TagAttribute[Tag("...")] on a job method or classTags and notes
JobInsightsAttribute, QueueInsightsFilterUseJobInsights()Insights
AlertsFilterUseAlerts(...) / UseWebhookAlerts(...)Alerts and webhooks
JobParameterInjectionFilterUseResultsInContinuations()Continuations
Telemetry filter (internal)UseTelemetry()OpenTelemetry
Failed-job retention filter (internal)UseFailedJobRetention(...)Timing
Job logging scope (internal)On by default with AddNightShiftServerLogging

Things to know​

  • Register global filters in every process that creates jobs, runs a server or hosts the dashboard. Client filters run where Enqueue is called, server filters run on the server, and state filters run wherever a state changes, which includes the dashboard when someone clicks Requeue or Delete.
  • One filter instance serves every job and every worker thread at the same time. Keep filters free of per-job fields; put per-job data in context.Items or in job parameters.

See also​