A Kill Switch That Halts New Work Without Wiping the Trail

An automation kill switch is only useful if it stops new work while leaving the evidence of what already happened. A switch that deletes queues or logs will remove the information you need afterwards. This article proposes a conditional design that separates three actions and a rehearsal to check them. Adapt it to your own system.

Three separate actions

Stop: no new job is started or accepted for processing. Jobs may still be queued, though the preferable choice is to refuse new submissions with a clear message.
Drain: jobs already running are allowed to finish, or reach a safe point, and no more are picked up.
Cancel: a specific job is told to stop. Cancel is a request, and it may arrive too late.

Stop is the kill switch. Drain and cancel are separate, deliberate decisions made after it.

The primary artifact: a stop/drain/cancel boundary rehearsal

A table that you fill in before real use. For each job at the moment of the stop, record: job_id, state at stop time (queued, running, finished), the planned action (leave queued, drain, request cancel), and the observed final state. Also record the switch event itself: who, when, and the reason. Nothing in the table is deleted when the switch is activated.

Decision branches

Job queued, not started: stays queued and unstarted. It is not deleted, and it can be resumed or cancelled by a person later.
Job running and the step is safely interruptible: allow it to finish its current step, then hold it.
Job running and mid-way through an external write whose outcome could be uncertain: let that write finish or time out, and record the outcome. If the outcome is unknown, mark the job needs_review. Do not retry automatically.
Job already finished: untouched, with its evidence intact.

Worked hypothetical example

At the moment of the stop there are 7 jobs: 3 queued, 2 running, 2 finished.

The 3 queued jobs remain queued. None starts.
Of the 2 running jobs, one is between steps and is held after its current step. The other is mid-write to an external service and times out with an unknown result, so it is marked needs_review.
The 2 finished jobs are unchanged.

Final tally: 3 queued, 1 held, 1 needs_review, 2 finished, and 7 in total. A new submission after the stop is refused, and the refusal is logged but not counted among the 7. The log records the switch event, the 7 jobs and their states.

Resuming is a separate act. A person releases the queued jobs, and the held job continues. The needs_review job is resumed only after someone checks the external result.

Stop conditions for the rehearsal itself

If any job disappears from the table during a rehearsal, treat it as a failure of the design.
If the observed counts do not match the counts at the stop moment, stop and find the cause before relying on the switch.
Run the rehearsal on test jobs, never on live work you cannot afford to disturb.

Proposed acceptance tests (not performed)

None of these has been run.

Activate the switch with jobs in each state and confirm that no queued job starts.
Confirm that logs and results of finished jobs are still readable afterwards.
Submit a new job after the stop and confirm it is refused with a visible message.
Request a cancel on a running job and confirm that the record shows whether it took effect, rather than assuming so.
Simulate a timeout during an external write and confirm that the job is marked needs_review and not retried.
Release the switch and confirm that resumption is a deliberate action.

A kill switch narrows what can go wrong. It does not guarantee that nothing in flight will complete, so the design must record in-flight outcomes honestly.