Skip to main content

Managing jobs

Once a job exists you can delete it, run it again, move it to another time, or put it in any state, from code or from the dashboard. This page also explains every state a job can be in, and how to read jobs, their states and counts through the monitoring API.

Quick example​

using Masterly.NightShift;

bool deleted = jobs.Delete(jobId); // to Deleted
bool requeued = jobs.Requeue(jobId); // to Enqueued, runs again
bool moved = jobs.Reschedule(jobId, TimeSpan.FromMinutes(30)); // to Scheduled

jobs is an IBackgroundJobClient. The static facade has the same methods: BackgroundJob.Delete(jobId), BackgroundJob.Requeue(jobId), BackgroundJob.Reschedule(jobId, TimeSpan.FromMinutes(30)).

How it works​

Nothing is ever edited in place: every action is a state change. Delete moves the job to the Deleted state, Requeue to Enqueued, Reschedule to Scheduled. They all call IBackgroundJobClient.ChangeState, which goes through the state machine:

  1. It takes a distributed lock on the job and reads its current state.
  2. If you gave an expected state (fromState) and the job is in another state, nothing changes and the method returns false.
  3. Election filters can replace the requested state. For example, AutomaticRetry turns a requested Failed into Scheduled while attempts are left.
  4. The new state is applied in one transaction: the old state's handler undoes its side effects (for example, removes the job from the schedule), the new one applies its own (adds it to a queue), and the state is appended to the job's history.

The methods return true only when the requested state was applied. They return false when the job doesn't exist (or expired), when the expected state didn't match, or when a filter chose a different state.

Changing the state of a running job doesn't stop the method that is running. Its cancellation token is canceled within a few seconds, so a job that observes its token stops; one that doesn't runs to the end and its result is discarded. See Cancellation.

Job states​

StateFinalHow a job gets thereWhat happens next
EnqueuedNoEnqueue, Requeue, a scheduled job falling due, a continuation whose parent finishedA worker on a server that listens to its queue fetches it and moves it to Processing
ScheduledNoSchedule, Reschedule, AutomaticRetry after a failureAt EnqueueAt, the DelayedJobScheduler moves it to Enqueued
AwaitingNoContinueJobWith, ContinueJobWithAll/Any, BatchJob.ContinueWithWhen its parent(s) finish, it moves to its next state or to Deleted
ProcessingNoA worker started itThe method returns (Succeeded), throws (Failed), or the job is deleted or moved while running
SucceededYesThe method returnedKept until it expires
FailedNoThe method threw and no retry is left (or retries are off)Stays until you requeue or delete it, or until its retention period ends (Failed jobs)
DeletedYesDelete, the dashboard, a continuation whose condition wasn't met, a filter that canceled the job, AutomaticRetry with AttemptsExceededAction.DeleteKept until it expires

When a job enters a final state (Succeeded or Deleted), its data gets an expiration time: JobStorage.JobExpirationTimeout, one day by default. Set it with WithJobExpirationTimeout(...) after the storage, or per method with [ExpireAfter] (see Timing filters). When a job leaves a final state, for example when you requeue a deleted job, the expiration is removed again.

Each state is a class in Masterly.NightShift.States:

ClassNotable membersCreated by you?
EnqueuedStateEnqueuedState(), EnqueuedState(string? queue), Queue, EnqueuedAt, DefaultQueue ("default")Yes
ScheduledStateScheduledState(TimeSpan enqueueIn), ScheduledState(DateTime enqueueAt) (UTC), EnqueueAt, ScheduledAtYes
AwaitingStateAwaitingState(string parentId, IState nextState, JobContinuationOptions options), ParentId, NextState, OptionsYes, see Continuations
AwaitingJobsStateParentIds, NextState, Options, ModeYes, see Fan-in and batches
ProcessingStateServerId, WorkerId, StartedAtNo, only workers create it
SucceededStateResult, Latency, PerformanceDuration (milliseconds), SucceededAtNo
FailedStateFailedState(Exception exception), FailedState(Exception exception, string? serverId), Exception, FailedAt, ServerIdRarely
DeletedStateDeletedState(), DeletedState(ExceptionInfo? exceptionInfo), ExceptionInfo, DeletedAtYes

Every state has Name, a settable Reason (shown in the job's history), IsFinal, and a static StateName ("Enqueued", "Scheduled", ...) to compare with or pass as an expected state. Create a new state object for each call; state objects are not thread-safe.

API​

Delete, Requeue, Reschedule, ChangeState​

Extension methods on IBackgroundJobClient; all but ChangeState also exist as static methods on BackgroundJob. All return bool. fromState is the state name the job must be in for the change to happen; null means any state.

MethodNew state
Delete(string jobId)Deleted
Delete(string jobId, string? fromState)Deleted, only from fromState
Requeue(string jobId)Enqueued, on the queue its [Queue] attribute names, else the job's own queue, else default
Requeue(string jobId, string? fromState)Enqueued, only from fromState
Reschedule(string jobId, TimeSpan delay)Scheduled, now plus delay
Reschedule(string jobId, TimeSpan delay, string? fromState)The same, only from fromState
Reschedule(string jobId, DateTimeOffset enqueueAt)Scheduled, at enqueueAt
Reschedule(string jobId, DateTimeOffset enqueueAt, string? fromState)The same, only from fromState
ChangeState(string jobId, IState state)Any state (extension method)
ChangeState(string jobId, IState state, string? expectedState)Any state, only from expectedState (interface method)

When the storage fails, the methods throw BackgroundJobClientException.

Querying jobs​

JobStorage.GetMonitoringApi() returns an IMonitoringApi (Masterly.NightShift.Storage), the read side the dashboard uses. Get the JobStorage from DI, or use JobStorage.Current. Lists are paged with from (zero-based offset) and count, and come back as JobList<TDto>, a list of KeyValuePair<string, TDto> keyed by job ID. DTOs are in Masterly.NightShift.Storage.Monitoring.

MethodReturns
JobDetails(string jobId)JobDetailsDto?: Job, CreatedAt, ExpireAt, Properties (job parameters), History (newest first: StateName, Reason, CreatedAt, Data), LoadException
GetStatistics()StatisticsDto: counts of Enqueued, Scheduled, Processing, Succeeded, Failed, Deleted, Awaiting, Retries, Recurring, Servers, Queues
Queues()Each queue's Name, Length, Fetched and FirstJobs
Servers()The running servers
EnqueuedJobs(string queue, int from, int perPage), FetchedJobs(string queue, int from, int perPage)Jobs on a queue
ProcessingJobs, ScheduledJobs, SucceededJobs, FailedJobs, DeletedJobs (int from, int count)Jobs in that state, with state-specific fields (for example FailedJobDto.ExceptionMessage, ScheduledJobDto.EnqueueAt, SucceededJobDto.Result as JSON)
ScheduledCount(), EnqueuedCount(string queue), FetchedCount(string queue), FailedCount(), ProcessingCount(), SucceededListCount(), DeletedListCount()Counts
SucceededByDatesCount(), FailedByDatesCount(), HourlySucceededJobs(), HourlyFailedJobs()Counts per day or hour, for graphs

StatisticsDto.Retries and StatisticsDto.Awaiting are long?: null when the storage doesn't count them.

Some members exist only on JobStorageMonitor, the base class that the monitoring API of SQL Server (and other storages) derives from. Cast to it, and check the feature first:

MemberFeature to check
AwaitingJobs(int from, int count), AwaitingCount()JobStorageFeatures.Monitoring.AwaitingJobs
DeletedByDatesCount(), HourlyDeletedJobs()JobStorageFeatures.Monitoring.DeletedStateGraphs
QueueCounts(): the same queues as Queues(), without reading their first jobsJobStorageFeatures.Monitoring.QueueCounts (without it, QueueCounts() works but is no cheaper than Queues())
using Masterly.NightShift.Storage;
using Masterly.NightShift.Storage.Monitoring;

if (storage.GetMonitoringApi() is JobStorageMonitor monitor
&& storage.HasFeature(JobStorageFeatures.Monitoring.AwaitingJobs))
{
long waiting = monitor.AwaitingCount();
JobList<AwaitingJobDto?> firstPage = monitor.AwaitingJobs(0, 20);
}

Without the feature, these members throw NotSupportedException.

A job's state and result​

JobStorageExtensions (namespace Masterly.NightShift) reads one job without its history:

MethodReturns
storage.GetJobState(string jobId)JobStateInfo?: the current state's Name, Reason and Data (by name, ignoring case), and the job's JobCreatedAt. null when the job doesn't exist, expired, or has no state yet.
storage.WaitForCompletionAsync(string jobId, TimeSpan timeout, CancellationToken cancellationToken = default)Task<JobStateInfo?>: waits until the job is completed (IsCompleted) or timeout has passed, and returns its state then. null when the job doesn't exist or expired. Timeout.InfiniteTimeSpan waits until the job completes; another negative timeout throws ArgumentOutOfRangeException, and a canceled token throws OperationCanceledException.
storage.TryGetResult<T>(string jobId, out T? result)true with the value the job returned, read from its Succeeded state with your serializer options. false when the job doesn't exist, hasn't succeeded, returned nothing (void, Task or null), or returned a value that couldn't be serialized. It throws JsonException when the stored value can't be read as T.
storage.RequeueFailedJobs(Func<FailedJobDto, bool>? filter = null, TimeSpan? timeBudget = null)Requeues every failed job, or those the filter selects, and returns a BulkJobResult (Processed, Remaining, Complete). An overload takes the IBackgroundJobClient to use.

JobStateInfo.IsCompleted is true for the Succeeded, Failed and Deleted states. A failed job counts as completed because a job only stays failed once its retries are used up: one that will be retried is Scheduled. A completed failed job can still be requeued by hand.

WaitForCompletionAsync reads the state every 50 milliseconds at first, then less often, up to once a second. It runs in the calling process and needs only access to the storage, so a web app can wait for a job a server elsewhere runs. Waiting doesn't change or keep the job.

GetJobState, WaitForCompletionAsync and TryGetResult read through storage.GetReadOnlyConnection(). For more, storage.GetConnection() returns a connection; dispose it when done. GetStateData(jobId) returns the current state, GetJobData(jobId) the job itself (Job, State, CreatedAt, ParametersSnapshot, and LoadException when its type or method can't be loaded), and GetJobParameter(jobId, name) one parameter, as stored JSON.

To read a job's state and act on it without a state change slipping in between, hold the job's state lock, the one every state change takes: connection.AcquireDistributedJobLock(jobId, timeout) (StorageConnectionExtensions, namespace Masterly.NightShift.Storage) returns an IDisposable, and throws DistributedLockTimeoutException when it can't get the lock in time. Hold it briefly: every state change of that job waits for it.

Delete pending jobs​

Extension methods on JobStorage that delete jobs which haven't started: every queue's enqueued jobs, the scheduled jobs and, on storages that list them (SQL Server does), the continuations waiting for their parents.

MethodDeletes
DeletePendingJobs<T>(string? methodName = null)The pending jobs of T's methods (or of T's subtypes), or of one method
DeletePendingFunctionJobs(string functionName)The pending jobs of a job function: a method marked with [JobFunction] or a delegate registered with AddNightShiftFunction. The name is compared without case
DeletePendingJobs(Func<PendingJob, bool> filter, string? reason = null)The pending jobs a condition selects
GetPendingJobs()Nothing: lists them as PendingJob (Id, State, Job, Queue, EnqueueAt)

Each returns the number of jobs deleted. A job is deleted only from the state it was listed in, so a job a worker took meanwhile runs, and continuations are deleted before the other jobs, so deleting a parent doesn't start them.

int canceled = storage.DeletePendingJobs<NewsletterService>(nameof(NewsletterService.Send));
int cleared = storage.DeletePendingJobs(x => x.Queue == "imports" && x.State == "Enqueued", "Import backlog cleared");

Listing reads the monitoring API page by page, so on a large backlog it takes a while; it isn't for request paths.

More examples​

Cancel a reminder that hasn't been sent yet​

using Masterly.NightShift.States;

bool canceled = jobs.Delete(reminderJobId, ScheduledState.StateName);
if (!canceled)
{
// Too late: the reminder was already enqueued, is running, or has finished.
}

Change to any state​

jobs.ChangeState(jobId, new EnqueuedState("critical") { Reason = "Escalated by support" });
jobs.ChangeState(jobId, new DeletedState { Reason = "Customer canceled the order" }, ProcessingState.StateName);

Retry every failed job​

BulkJobResult result = storage.RequeueFailedJobs();
Console.WriteLine($"requeued {result.Processed} jobs");

It reads the failed list a page at a time, like "Requeue all" on the dashboard's Failed page, and requeues each job only if it is still failed when its turn comes, so a job someone else handled in the meantime is left alone. A job that fails again while it runs isn't requeued twice. It goes on until the end of the list; pass timeBudget to stop earlier, for example inside a web request. The result's Complete is then false, Remaining says how many jobs are left at most, and another call goes on where this one stopped.

To requeue only some jobs, pass a filter. It gets each job's FailedJobDto:

using Masterly.NightShift.Storage.Monitoring;

BulkJobResult result = storage.RequeueFailedJobs(
failed => failed.ExceptionType == typeof(TimeoutException).FullName,
timeBudget: TimeSpan.FromSeconds(20));

The jobs are requeued through a new BackgroundJobClient for the storage. With dependency injection, pass the registered client so its filters apply: storage.RequeueFailedJobs(client, filter).

List failures with their messages​

using Masterly.NightShift.Storage.Monitoring;

IMonitoringApi monitoring = storage.GetMonitoringApi();
long total = monitoring.FailedCount();
for (int from = 0; from < total; from += 100)
{
foreach (KeyValuePair<string, FailedJobDto?> pair in monitoring.FailedJobs(from, 100))
{
FailedJobDto? failed = pair.Value;
Console.WriteLine($"{pair.Key} {failed?.Job?.Method.Name}: {failed?.ExceptionMessage}");
}
}

A job status endpoint​

app.MapGet("/jobs/{id}", (string id, JobStorage storage) =>
{
JobDetailsDto? details = storage.GetMonitoringApi().JobDetails(id);
if (details == null)
{
return Results.NotFound();
}

StateHistoryDto? current = details.History.FirstOrDefault();
return Results.Ok(new
{
id,
state = current?.StateName,
reason = current?.Reason,
createdAt = details.CreatedAt,
expireAt = details.ExpireAt,
});
});

Only expose job IDs and states this way to callers that are allowed to see them; job IDs are sequential numbers in SQL Server.

Only the current state​

JobStateInfo? state = storage.GetJobState(id);
string? stateName = state?.Name;

Without the extension, read it through a connection:

using IStorageConnection connection = storage.GetReadOnlyConnection();
StateData? state = connection.GetStateData(id);

Wait for a job and use its result​

string? jobId = jobs.Enqueue<ImportService>(x => x.ImportAsync(fileId, CancellationToken.None));

JobStateInfo? state = await storage.WaitForCompletionAsync(jobId!, TimeSpan.FromMinutes(2), cancellationToken);

if (state is { Name: "Succeeded" } && storage.TryGetResult(jobId!, out int rows))
{
Console.WriteLine($"imported {rows} rows");
}
else if (state is { IsCompleted: false })
{
Console.WriteLine($"still {state.Name} after two minutes");
}

A job's result​

if (storage.TryGetResult<int>(id, out int count))
{
Console.WriteLine($"the job imported {count} rows");
}

Counts for a health page​

StatisticsDto stats = storage.GetMonitoringApi().GetStatistics();
Console.WriteLine($"enqueued {stats.Enqueued}, processing {stats.Processing}, failed {stats.Failed}");

Keep finished jobs longer​

services.AddNightShift(config => config
.UseSqlServerStorage(connectionString)
.WithJobExpirationTimeout(TimeSpan.FromDays(7)));

Things to know​

  • Delete doesn't remove data. It moves the job to Deleted; the data goes away when it expires. To remove everything at once, see Schema and upgrades (purge).
  • Delete doesn't guarantee the job won't run. A job enqueued a moment ago may already be running, and a running job runs to the end unless it observes its cancellation token.
  • Requeue runs a job again from the start, including a succeeded one. Make jobs safe to run twice.
  • false is not an error. Check it when the outcome matters (the job had already moved on, expired, or a filter chose another state).
  • The monitoring API reads directly from storage. It is meant for dashboards and tools. Don't call it in a tight loop or on every web request; large count values are slow.
  • Failed jobs don't expire by default. Requeue or delete them, or give them a retention period with [ExpireAfter(..., IncludeFailed = true)] or UseFailedJobRetention (Failed jobs), or they pile up.

See also​