Skip to main content

Troubleshooting

This page lists the problems people run into most often with Masterly.NightShift, each with its usual causes and the fix. Start with the symptom you see; most sections point to a page in the dashboard where you can confirm the cause.

SymptomSection
Jobs are created but never runJobs stay in Enqueued, Jobs are held
Delayed jobs or retries start lateJobs stay in Scheduled, or start late
A job shows Processing foreverJobs stuck in Processing
A job ran more than onceA job runs twice, The app stops and jobs start over
Enqueue returned nullEnqueue returns null
Jobs fail after a deploy or renameA job fails because it can't be loaded, Serialization errors
Errors about services or DbContext in a jobA job fails with a dependency injection error
Exceptions at start-up"Unable to find the required services", "Current JobStorage instance has not been initialized yet", SQL Server: errors at start-up
Recurring jobs don't run, run twice, or run at the wrong hourA recurring job doesn't fire, A recurring job runs twice
The dashboard refuses access or buttons failThe dashboard returns 401, 403 or 404
SQL timeouts, deadlocksSQL Server: timeouts and deadlocks
Jobs stop when an IIS site is idleThe server stops when IIS goes idle
No traces, alerts, statistics or logsMonitoring shows nothing

Quick example​

Three places answer most questions:

  1. The job's details page in the dashboard (click a job ID). Besides the state history and exception details, it shows a diagnosis for jobs that aren't moving: no server listens to the queue, processing is paused, the scheduled time has passed with no server alive, the parent of a continuation failed, the server running it is gone, or why a failed job won't be retried.
  2. The Servers page. It lists every running server with its queues, worker count and last heartbeat. If your server isn't there, nothing processes jobs.
  3. The log. Turn on debug logging for the library's categories, which all start with Masterly.NightShift. In an ASP.NET Core or Worker Service app, in appsettings.json:
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Masterly.NightShift": "Debug"
}
}
}

How it works​

Most problems come down to one of the parts in Core concepts: a job is a record in the storage, servers fetch it from a queue, a worker runs it, and filters decide which state comes next. When a job "does nothing", either no server is fetching from its queue, a filter (drain mode, throttling, a run window) is holding it, or it can't be loaded. When a job "runs twice", it's the at-least-once guarantee doing its work. The sections below go through these cases.

Options / API​

The settings that come up on this page, with their defaults:

SettingTypeDefaultRelevant to
BackgroundJobServerOptions.Queuesstring[]{ "default" }Jobs that stay Enqueued
BackgroundJobServerOptions.WorkerCountintMath.Min(Environment.ProcessorCount * 5, 20)Jobs that wait for a free worker
BackgroundJobServerOptions.SchedulePollingIntervalTimeSpan15 secondsDelayed jobs and retries starting late; recurring job scheduling
BackgroundJobServerOptions.StopTimeoutTimeSpan500 msWhen running jobs' cancellation tokens are canceled on shutdown
BackgroundJobServerOptions.ShutdownTimeoutTimeSpan15 secondsJobs interrupted on shutdown
BackgroundJobServerOptions.ServerTimeoutTimeSpan5 minutesWhen a silent server is removed from the Servers page
SqlServerStorageOptions.SlidingInvisibilityTimeoutTimeSpan?5 minutesWhen a job held by a crashed server becomes available again
SqlServerStorageOptions.CommandTimeoutTimeSpan?null (the driver's default)SQL timeouts
SqlServerStorageOptions.CommandBatchMaxTimeoutTimeSpan?5 minutesTimeouts on batched writes
SqlServerStorageOptions.PrepareSchemaIfNecessarybooltrueSchema installation on start-up
SqlServerStorageOptions.EnableHeavyMigrationsboolfalseUpgrading old schemas
AutomaticRetryAttribute.Attemptsint10How often a failed job is retried before it reaches Failed
DrainMode.HoldIntervalTimeSpan30 secondsHow long held jobs wait before they are looked at again
DashboardOptions.AuthorizationIEnumerable<IDashboardAuthorizationFilter>LocalRequestsOnlyAuthorizationFilterDashboard 401
RecurringJobOptions.TimeZoneTimeZoneInfoUTCRecurring jobs firing at the wrong hour
KeepAliveOptions.IntervalTimeSpan5 minutesIIS idle shutdown

Jobs stay in Enqueued​

The job is in its queue, but no worker takes it.

CauseHow to tellFix
No server is runningThe Servers page is empty; the job's diagnosis says no server is aliveCall AddNightShiftServer() (or start a BackgroundJobServer) in at least one process. An app that only calls AddNightShift(...) creates jobs but never runs them
No server listens to the job's queueThe diagnosis says the queue isn't served; the Servers page shows other queuesAdd the queue to BackgroundJobServerOptions.Queues on some server, or move the job to a served queue. [Queue("billing")] on a method sends its jobs to billing
The server uses a different database or schemaYour server is missing from this dashboard's Servers page, or the job never shows on the other app's dashboardUse the same connection string and SqlServerStorageOptions.SchemaName in every app
All workers are busyProcessing shows WorkerCount jobs per server, often long onesRaise WorkerCount, add servers, or give long jobs their own queue and server
services.AddNightShiftServer(options =>
{
options.Queues = new[] { "billing", "default" };
});

Queue names are lowercase: they may only contain lowercase letters, digits, underscores and dashes. [Queue("Billing")] throws ArgumentException as soon as the attribute is read, so creating any job of that method fails. A template such as [Queue("tenant-{0}")] is checked when the job is enqueued: an argument that produces an invalid name, such as Contoso, throws InvalidOperationException naming the template and the result, and a placeholder for an argument the method doesn't have ({3} on a method with two parameters) throws FormatException.

Jobs are held: "processing is paused"​

Jobs move from Enqueued to Scheduled with the reason Held: processing is paused, Held: queue '{name}' is paused or Held: server '{id}' is paused, come back after 30 seconds, and are held again.

This is drain mode: someone paused all processing, one queue, or one server, from the dashboard or in code. Running jobs finish; new ones are held until processing resumes. The Servers page shows the switch and a Resume button, and the Queues page shows paused queues. In code:

using (var connection = storage.GetConnection())
{
var status = DrainMode.Read(connection);
Console.WriteLine($"Drain: {status.Active}, paused queues: {string.Join(", ", status.PausedQueues.Keys)}");

DrainMode.Disable(connection);
DrainMode.ResumeQueue(connection, "billing");
}

The switch lives in the storage (the hangfire:control hash), so restarting the app doesn't turn it off. Each process caches the answer for 2 seconds, so a change takes effect almost at once.

Jobs stay in Scheduled, or start late​

  • No server is running. The delayed job scheduler runs inside servers. Without one, nothing moves Scheduled jobs to Enqueued. The diagnosis says the scheduled time has passed and no server is alive.
  • Up to 15 seconds late is normal. Servers check for due jobs every SchedulePollingInterval. Lower it if you need delayed jobs and retries to start sooner, at the cost of more queries.
  • The job is waiting for a retry. A failed job with retries left is Scheduled, and it is listed on the Retries page. The delay grows with each attempt, up to a couple of hours for the tenth (see Retries).
  • A filter rescheduled it. Throttling, rate limits, circuit breakers, run windows and RetryLaterException all reschedule jobs instead of running them. The state history shows the reason each time.

Jobs stuck in Processing​

A job shows as Processing for a long time, and its diagnosis says the server running it is gone or has stopped sending heartbeats.

The server crashed or was killed while the job ran. The job isn't lost: with SQL Server, the queue entry becomes visible to other workers once SlidingInvisibilityTimeout (5 minutes by default) passes without the crashed server refreshing it, and another worker runs the job again from the start. The dead server is removed from the Servers page after ServerTimeout. If you can't wait, requeue the job from the dashboard.

If the server is alive and the job is slow, the job details page compares the run time with the job's usual duration. Use [JobTimeout] to stop runs that take too long.

A job runs twice​

Jobs run at least once, not exactly once. The usual reasons:

CauseWhat you see
The job threw and was retriedThe history shows Scheduled states with "Retry attempt n of m" reasons. Work done before the exception is repeated
The server stopped while the job ranThe job ran on one server, then started again after a restart or deploy. A job canceled by the shutdown goes back to its queue at once. One still running after ShutdownTimeout is abandoned and taken by another worker once SlidingInvisibilityTimeout passes. See The app stops and jobs start over
The server crashed, or lost the database for longer than SlidingInvisibilityTimeoutAnother worker fetched the job because the queue entry looked abandoned. Rarely, both runs overlap
The same work was enqueued twiceTwo different job IDs with the same method and arguments. Common with recurring jobs triggered by hand, or when a request that enqueues is retried
Two environments share a databaseA developer machine or a test environment pointed at the production database runs production jobs. The Servers page shows unexpected server names

Make jobs idempotent first: check whether the work is already done, use upserts, and pass idempotency keys to external services. Then, where it fits:

public class ImportService
{
// Only one ImportCatalog job runs at a time across all servers
[DisableConcurrentExecution(timeoutInSeconds: 60)]
public void ImportCatalog() { }

// A second Reindex(productId) isn't created while the first one is still pending
[PreventDuplicates]
public void Reindex(int productId) { }
}

See Concurrency, and Core concepts for why this can't be fully avoided.

Enqueue returns null and no job appears​

Enqueue, Schedule and ContinueJobWith return the new job's id, or null when a client filter canceled the creation. Nothing is stored and nothing is thrown.

CauseFix
[PreventDuplicates] found a pending job with the same method and arguments (OnDuplicate = DuplicateHandling.Cancel, the default)This is what the attribute is for. Check the id for null if the caller needs to know. See PreventDuplicatesAttribute
Your own client filter set context.Canceled = true in OnCreatingCheck the filter's condition

Treat the id as string? and handle null where it matters, for example before passing it to ContinueJobWith.

The app stops and jobs start over​

During a deploy or restart, running jobs are interrupted and run again from the start later.

When the host stops, the server stops taking new jobs, cancels the token of every running job after StopTimeout (500 ms), and waits up to ShutdownTimeout (15 seconds) for them to return:

  • A job that takes a CancellationToken and throws OperationCanceledException goes back to its queue at once, and runs again on the next server that takes it.
  • A job that ignores the token keeps running until the process exits. The server stops waiting after ShutdownTimeout and logs that it stopped non-gracefully. The job stays in Processing and another worker takes it once SlidingInvisibilityTimeout (5 minutes) passes.

Fixes:

  • Take a CancellationToken (pass CancellationToken.None when you create the job) and pass it to every awaited call, so the job stops quickly and is requeued at once. See Cancellation.
  • Give long jobs more time: raise ShutdownTimeout, and the host's own HostOptions.ShutdownTimeout, which ends the whole shutdown (see Graceful shutdown). Kubernetes and other orchestrators also have a grace period before they kill the process.
  • Make long jobs resumable, for example by saving progress, so a second run continues instead of starting over.

A job fails because it can't be loaded​

The server couldn't rebuild the method call from the stored record, so it couldn't move the job to the next state. The reason says what failed: the job's type or method was not found. when the class, its assembly or the method can't be found, or the job's arguments could not be deserialized. when an argument can't be read. It starts with Can not change the state to '<state>':, naming the state the server was moving the job to: Processing when a worker fetched the job, Enqueued when a scheduled or retried job was due. The exception details show the error.

The failure goes through the global filters like any other, so the default AutomaticRetry filter retries it 10 times: the job first shows under Retries, scheduled with a reason like Retry attempt 1 of 10: Could not load type ..., and reaches Failed only when the attempts are used up. Each attempt fails the same way until a server that can load the job picks it up. Attributes on the job's class or method (for example [AutomaticRetry(Attempts = 0)]) don't apply, because they can't be read without the method; only global filters do. Look at the inner exception:

Inner exceptionCauseFix
TypeLoadException, FileNotFoundException (assembly)The class, its namespace or its assembly was renamed or moved, or the server doesn't reference the assembly at allKeep the old type and method as a thin wrapper that calls the new code until old jobs are gone. Make sure every server references the assembly with the job classes
InvalidOperationException or MissingMethodException about the methodThe method was renamed, or its parameter list changedSame: keep the old signature as an overload that forwards to the new one
A JSON exception (JsonException, NotSupportedException about deserialization)An argument can't be read: its type changed, or it isn't readable with System.Text.JsonSee Serialization errors after an upgrade
FileLoadException about an assembly versionStored type names contain an assembly version that the server doesn't haveUse UseSimpleAssemblyNameTypeSerializer() for new jobs (it stores names without versions), and UseIgnoredAssemblyVersionTypeResolver() to read old ones

Other common setups that cause it:

  • Rolling deploys. A new client creates jobs for a method that old servers don't have yet. Deploy servers before clients, or create the new jobs with a new queue passed as an argument (see the next point) that only updated servers listen to.
  • Several apps share one database. Each server takes any job from the queues it listens to, including jobs from other apps whose classes it doesn't have. Separate queues isolate the apps only partly. Any server moves due scheduled and retried jobs to their queue, and that loads the job's class unless the job was created with the queue passed as an argument (BackgroundJob.Enqueue("app-a", ...)); a queue set with [Queue], or the default queue, needs the class. Any server also triggers recurring jobs, and the server that finishes a parent job starts its continuations, which always loads their class. A recurring job that a server can't load is retried and, after several failures in a row, disabled. To isolate apps, give each one its own schema with SqlServerStorageOptions.SchemaName (or its own database).

Once fixed, jobs still under Retries run again at their next attempt; requeue the failed ones from the Failed page.

Serialization errors after an upgrade​

Jobs created before an upgrade fail with "the job's arguments could not be deserialized" and a JSON exception, or run with missing data (for example a property that is now null).

Arguments are stored as JSON. When the reading side changes, old payloads may no longer fit:

  • Coming from Hangfire: Newtonsoft.Json was replaced by System.Text.Json. Properties with private setters need [JsonInclude], types without a public parameterless constructor need a single public constructor or [JsonConstructor], Newtonsoft attributes such as [JsonProperty] are ignored, and an argument declared as an interface or abstract class needs a "$type" naming its actual type, which Hangfire wrote only with its recommended settings (TypeNameHandling). See Migrating from Hangfire.
  • Custom serializer options: whatever options you pass to UseSerializerSettings, user data is read leniently (property names in any case, quoted numbers, public fields). What can still break old payloads is a custom converter that reads only its own format; make it accept the old format too.
  • You changed an argument type: renaming or removing properties, or changing their types, affects jobs that are already stored. Add new properties with defaults; keep old ones until old jobs are processed.

To see what is stored, open the job in the dashboard, or read the Arguments column of the HangFire.Job table. Once the type can read the old payloads, requeue the failed jobs. See Serialization.

The dashboard returns 401, 403 or 404​

ResponseCauseFix
401 for everyone but youThe default authorization filter, LocalRequestsOnlyAuthorizationFilter, only lets in requests from the same machine. Requests from another computer, from outside a Docker container, or with a forwarding header (Forwarded, X-Forwarded-For, X-Forwarded-Host or X-Real-IP, which proxies add) are refused. A proxy on the same machine that adds no forwarding header makes every request look localSet DashboardOptions.Authorization to your own filter, or use MapNightShiftDashboardWithAuthorizationPolicy (see below)
403 on every pageYou're signed in, but an authorization filter refused you. Unauthenticated users get 401, authenticated ones 403Check the filter or policy, for example the role it requires
403 when clicking buttons (Requeue, Delete, Trigger now)The antiforgery token was rejected. Common with several instances behind a load balancer that don't share Data Protection keys, so a token issued by one instance is invalid on another. Without AddAntiforgery(), the dashboard only accepts POSTs that carry X-Requested-With and come from its own origin (same scheme, host and port), so a script or a form on another site gets 403 with an explanation in the body. Behind a proxy that changes the Host header or the scheme, browsers that don't send Sec-Fetch-Site get 403 on every button, because their Origin no longer matchesRegister antiforgery with services.AddAntiforgery(), and share the Data Protection key ring between instances. DashboardOptions.IgnoreAntiforgeryToken = true turns the check off, which you should only do when nothing else protects the dashboard from cross-site requests
404The path is wrong: the default is /nightshift (Hangfire used /hangfire). In an IIS virtual directory, the path is below it, for example /shop/nightshiftUse the path you passed to MapNightShiftDashboard(...), or pass the one you want
404 for the dashboard's CSS and scripts behind a reverse proxyThe proxy serves the app under a prefix it strips, so the dashboard builds links without itSet DashboardOptions.PrefixPath to that prefix, or forward it as the path base
404 with UseNightShiftDashboardThe middleware was added after a middleware that ends the requestAdd it earlier in the pipeline, or use MapNightShiftDashboard

A dashboard for administrators only, with ASP.NET Core authorization:

builder.Services.AddAuthorization(options =>
options.AddPolicy("JobsAdmin", policy => policy.RequireRole("Admin")));

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapNightShiftDashboardWithAuthorizationPolicy("JobsAdmin");

Or with a dashboard filter:

using Masterly.NightShift.Dashboard;

public sealed class AdminDashboardFilter : IDashboardAuthorizationFilter
{
public bool Authorize(DashboardContext context)
{
var user = context.GetHttpContext().User;
return user.Identity?.IsAuthenticated == true && user.IsInRole("Admin");
}
}

app.MapNightShiftDashboard(new DashboardOptions
{
Authorization = new[] { new AdminDashboardFilter() },
});

Note: Behind a reverse proxy on the same machine (nginx, or IIS forwarding to Kestrel) every request can look local, so the default filter may let everyone in. Configure the forwarded headers middleware so ASP.NET Core sees the real client address, and use real authorization for anything reachable from outside. See Dashboard authorization.

"Unable to find the required services"​

InvalidOperationException: Unable to find the required services. Please add all the required services by calling 'IServiceCollection.AddNightShift' inside the call to 'ConfigureServices(...)' in the application startup code.

AddNightShiftServer, MapNightShiftDashboard, UseNightShiftDashboard, MapNightShiftHealth and MapNightShiftMetrics need the services that AddNightShift(...) registers. Call builder.Services.AddNightShift(config => ...) in every app that uses them, including a dashboard-only app.

"Current JobStorage instance has not been initialized yet"​

InvalidOperationException: Current JobStorage instance has not been initialized yet. You must set it before using Masterly.NightShift Client or Server API. ...

The static BackgroundJob, RecurringJob or new BackgroundJobServer() were used before any storage was configured.

  • Without dependency injection: call GlobalConfiguration.Configuration.UseSqlServerStorage(...) before creating jobs or servers.
  • With AddNightShift: the configuration callback runs lazily, the first time a NightShift service is resolved: when a server starts, when the dashboard is mapped, or when you resolve IBackgroundJobClient or IRecurringJobManager. Code that runs between builder.Build() and app.Run() and uses the static classes can run too early. Resolve the services instead, which also runs the configuration:
var recurringJobs = app.Services.GetRequiredService<IRecurringJobManager>();
recurringJobs.AddOrUpdate<ReportService>("daily-report", x => x.BuildDaily(), Cron.Daily(6));

A job fails with a dependency injection error​

With AddNightShift, each job run gets its own IServiceScope: the job class is created from it, and the scope is disposed when the method returns. See Dependency injection.

SymptomCauseFix
InvalidOperationException from the container saying it is unable to resolve a service for a type, while activating your job classA constructor parameter of the job class isn't registered in the process that runs the job. Web and worker processes often register different servicesRegister the service in every process that runs the job. The job class itself doesn't need to be registered
IHttpContextAccessor.HttpContext is null, or a "current user" or "current tenant" service failsJobs don't run inside a web request, even when a request created themPass what the job needs (user id, tenant id, culture) as arguments, and look it up in the job
ObjectDisposedException for a DbContext or another scoped serviceWork continues after the job method returned: a Task that isn't awaited, Task.Run without await, or an async void helper the job calls. The scope is already disposedReturn Task from async job methods and await every call
A scoped service holds stale data, or two workers corrupt each other's stateThe job class is registered as a singleton, so all runs share it and its dependenciesRegister job classes as scoped or transient, or don't register them
Creating the job throws a serialization error, or an argument arrives empty or partialArguments are serialized to JSON when the job is created. Services, a DbContext, an HttpContext or an open connection can't bePass ids and plain data. Take services through the job class's constructor. See Job arguments

A recurring job doesn't fire​

Open Recurring Jobs and look at the job's row: its next execution, its last execution and any error.

CauseHow to tellFix
No server is runningNext execution is in the past and nothing happensRecurring jobs are scheduled by servers. Start one
It is pausedThe row has a Paused label and no next executionResume it on the page, or call RecurringJob.Resume("daily-report"). AddOrUpdate at start-up doesn't resume a paused job, on purpose
Wrong time zoneIt fires, but at the wrong hourRecurring jobs use UTC unless you set RecurringJobOptions.TimeZone
It can't be scheduledThe row shows an error, such as a type that can't be loaded or a time zone that doesn't exist on the serverThe scheduler retries 5 times, then stops scheduling the job. Fix the cause, then call AddOrUpdate again (a job with an error is rescheduled) or trigger it
The cron expression means something elseThe next execution is not what you expectedSix fields mean the first is seconds: */15 * * * * * is every 15 seconds. 0 0 31 2 * never fires. The Recurring Jobs page and Cron.Describe show what an expression means
It fired while no server was runningNothing ran during the downtimeMisfireHandlingMode.Relaxed (the default) runs once when a server is back, Strict runs every missed occurrence, Ignorable skips them
The occurrence was created, but the job didn't runJobs has a job "Triggered by recurring job scheduler" that is Enqueued or FailedTreat it like any other job: see the sections above
recurringJobs.AddOrUpdate<ReportService>(
"daily-report",
x => x.BuildDaily(),
Cron.Daily(6),
new RecurringJobOptions
{
TimeZone = TimeZoneInfo.FindSystemTimeZoneById("Europe/Berlin"),
MisfireHandling = MisfireHandlingMode.Relaxed,
});

The time zone ID is stored with the recurring job and resolved on every server that schedules it, so the ID must exist on all of them. Windows IDs (W. Europe Standard Time) and IANA IDs (Europe/Berlin) both resolve on Windows and Linux when ICU is available. To map IDs yourself, give the server an ITimeZoneResolver (see Time zones and misfires).

A recurring job runs twice, or at an odd time​

Each run is an ordinary job, so the job details page of each run shows who created it and when. Look at the reason of its first state and at the Recurring Jobs row.

CauseHow to tellFix
Two recurring jobs do the same workRecurring Jobs lists both, often under an old and a new id. Renaming the id in AddOrUpdate creates a new recurring job and leaves the old oneRemove the old one with RecurringJob.RemoveIfExists("old-id"), or from the page
Someone triggered it by handAn extra run between two scheduled ones, created with the reason Triggered using recurring job manager instead of Triggered by recurring job scheduler. A trigger from the dashboard is listed on the Audit page as Trigger recurring jobsNothing to fix. Use [PreventDuplicates] if a manual run must not overlap a pending scheduled one
A run failed and was retriedThe run's history shows Retry attempt n of m; the work done before the exception is repeatedMake the job idempotent, or lower the retries with [AutomaticRetry(Attempts = ...)]
A run was interrupted by a deploySee The app stops and jobs start over
Missed runs were made up after downtimeSeveral runs at once when a server came back, with MisfireHandling = MisfireHandlingMode.StrictUse Relaxed (one run for all missed occurrences) or Ignorable
Daylight saving timeIn a time zone with daylight saving, an hourly schedule runs twice in the repeated autumn hour and a daily job runs at an odd time on the spring dayExpected; see Daylight saving time. Keep the job in UTC if it must run at fixed intervals
Another environment runs itA test or developer machine shares the production database; the Servers page shows its nameGive every environment its own database or schema

A recurring job is scheduled by one server at a time, under a distributed lock, so adding servers doesn't make it fire more often.

SQL Server: errors at start-up​

ErrorCauseFix
ArgumentException: 'NightShift' is not a connection string: it has no 'key=value' pairsA connection string name was passed to UseSqlServerStorage or new SqlServerStorage(...). Names aren't looked up in configurationRead the value with builder.Configuration.GetConnectionString("NightShift") and pass it
SqlException: ... The certificate chain was issued by an authority that is not trustedMicrosoft.Data.SqlClient encrypts connections by default and checks the server certificate, which a local or development SQL Server usually doesn't have from a trusted authorityAdd TrustServerCertificate=True to the connection string for such a server, or install a trusted certificate on it (see The SQL client)
Cannot open database "NightShift" requested by the loginThe database doesn't exist, or the login has no accessThe storage creates tables, not the database. Create it first and grant the login access
Warning Was unable to perform the Masterly.NightShift schema migration due to an exception, then Invalid object name 'HangFire.Job'The schema couldn't be installed, usually because the login can't create tablesRun the install once with a privileged login, or run the script by hand (see Schema and upgrades), then set PrepareSchemaIfNecessary = false for the app's login
ArgumentException: The server id '...' is N characters long, but the SQL Server storage keeps server ids of at most 200 characters, and the server never processes jobsServerName is longer than 150 charactersSet a shorter BackgroundJobServerOptions.ServerName
The schema stays at version 5 (or 7): select Version from [HangFire].[Schema] doesn't reach 9An old database needs a heavy migration, which is skipped by default. The install script reports it only as SQL PRINT output ("Migration process STOPPED at schema version 5"), which shows when you run the script by hand, not in the application logIn a quiet window, start once with EnableHeavyMigrations = true, or run the script by hand

SQL Server: timeouts and deadlocks​

  • Old schema. On start-up, the storage reads the schema version and turns on the options it supports: no global locks from version 6, and IGNORE_DUP_KEY instead of MERGE from version 8, which avoids most deadlocks on the Hash and Set tables. A database stuck at an old version because a heavy migration was skipped deadlocks more. Upgrade it (see the table above). The dashboard home page shows the schema version.
  • Command timeouts. CommandTimeout (the driver's default, 30 seconds, when null) applies to single commands, and CommandBatchMaxTimeout (5 minutes) to the batched writes of a transaction. Raise them if a busy or remote database times out:
using Masterly.NightShift.SqlServer;

config.UseSqlServerStorage(connectionString, new SqlServerStorageOptions
{
CommandTimeout = TimeSpan.FromSeconds(60),
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
});
  • Distributed lock timeouts. DistributedLockTimeoutException in a failed job means a lock, usually from [DisableConcurrentExecution], was held by another job for longer than its timeout. That's the filter working; the job is retried. Raise the timeout or reduce how often the job is created.
  • Expiration on very large tables. Expired jobs are deleted in batches by a background process. On a database with millions of rows, see DeleteExpiredBatchSize and JobExpirationCheckInterval in SQL Server options.
  • Too many queries on idle queues. Every server polls its queues. With many servers, see QueuePollInterval in SQL Server options.

The server stops when IIS goes idle​

Jobs only run while requests come in; recurring jobs fire late or not at all; the Servers page shows the server disappearing and coming back.

IIS shuts an app pool down after 20 minutes without requests (the default idle timeout), and the server inside it stops. Two settings together keep it running:

  1. Keep-alive. AddNightShiftKeepAlive makes the server request one of the app's own URLs every 5 minutes, so the pool never counts as idle. Behind IIS, give the public address, because the request has to go through IIS to count:

    services.AddNightShiftServer();
    services.AddNightShiftKeepAlive(options =>
    {
    options.Url = "https://jobs.example.com/nightshift/ping";
    });

    /nightshift/ping answers without touching the storage; it exists when the dashboard is mapped at /nightshift. Any URL that reaches the app works.

  2. Start after a recycle. The keep-alive runs inside the app, so it can't restart a pool that IIS recycled. In IIS, set the app pool's start mode to AlwaysRunning and enable preload for the site (Application Initialization), so the app starts without waiting for a request.

See Keep-alive. Alternatively, run the server in a Windows Service or Worker Service and keep only the client and dashboard in IIS.

Monitoring shows nothing​

Most monitoring features are turned on per process, and must be turned on in the process where the work happens: alerts, insights and telemetry where jobs run, the dashboard's features where the dashboard runs.

SymptomCauseFix
No create or process spans, no nightshift.job.* metricsUseTelemetry() isn't called, or the OpenTelemetry setup lacks AddSource(NightShiftTelemetry.ActivitySourceName) / AddMeter(NightShiftTelemetry.MeterName).Call UseTelemetry() in every process that creates or performs jobs, and subscribe to the source and meter. See OpenTelemetry
Each job starts a new trace instead of joining the request'sThe process that created the job didn't call UseTelemetry(), or there was no Activity when it created the job (the recurring job scheduler, a console app)Call UseTelemetry() in the web app too
No alert for a failed jobAlerts are raised where the job runs, and only when it reaches Failed: with the default 10 retries that is hours after the first failure. Or the sink was registered only in the web app, or its Events leave the kind outRegister the sink with UseWebhookAlerts/UseAlerts in every server process. Look for warnings from Masterly.NightShift.WebhookAlertSink (rejected, timed out or dropped requests). See Alerts and webhooks
The Insights page is empty, or the Queues page has no wait timesUseJobInsights() isn't called in the processes that run jobsCall it in every server process. See Insights
The home page's live activity and the Timeline are emptyThe activity feed is off by defaultCall UseActivityFeed() in every process that changes job states. See Audit log and activity feed
The Audit page shows every action as anonymousThe dashboard isn't behind authentication, so there is no user namePut the dashboard behind an authorization policy. See Dashboard authorization
Prometheus gets 401 from /metricsThe dashboard's /metrics route uses the dashboard's authorization, which allows local requests only by defaultMap MapNightShiftMetrics("/metrics/nightshift") outside the dashboard and limit it with endpoint conventions. See Health checks and probes
The Process health check is unhealthyNo server runs in this process yet, the server wasn't added with AddNightShiftServer, or it was given its own list of background processesSee Health checks and probes
No log messages from Masterly.NightShift in an app without AddNightShiftWithout AddNightShift, messages go to Serilog, NLog, log4net and a few others when one is loaded, and nowhere otherwiseCall a Use...LogProvider() method first thing at start-up. See Logging

See also​