ADR 0029: Tighten the notifier's stop and its dropped count¶
- Status: Accepted
- Date: 2026-11-12
Context¶
The Telegram notifier queues the operator's messages for one thread of its
own (ADR 0027). At a stop, main
calls end(5) in its finally, and the exit function stop() waits for
the queue until those 5 seconds run out. It then reports in red what is
left. When more than 100 messages are waiting, the newest are dropped, and
the chat later gets one message saying how many.
The last review of step 6o found four limits there. ADR 0027 listed them under "What this does not cover", and each one costs a message only with a log line that says so:
- The seconds count from the end of
main. A signal still running 5 seconds after that leaves its last line no time at all. - If
mainnever reaches itsfinally, the stop waits no time at all. That happens after a second Ctrl+C at the wrong moment. - The dropped count waits for an empty queue. Under steady traffic with a slow chat that can take hours. Once queued, it takes a slot that a line could need.
- The stop's report can be wrong. It can say a message was not sent while the thread still sends it, on its next try, during the rest of the shutdown.
When does a signal outlive main? The server installed in the image,
uvicorn 0.54.0, answers SIGTERM, and a first Ctrl+C, by waiting for every
open connection and every request still being handled. It only gives up
waiting on a second SIGINT (Server.handle_exit). So under
docker stop, which sends one SIGTERM, main ends only after the last
signal, and end comes after its last line. Limits 1 and 2 happen only
after a second Ctrl+C, on a terminal. A terminal kills nothing after a
grace period: the process then waits for the signal's thread, since
Python waits for every thread that is not a daemon before its exit
functions.
Decision¶
- A floor under the stop's wait: 2 seconds (
STOP_FLOOR_SECONDS). The stop waits until the later of two times: the deadlineendset, and now plus the floor. Without anendit waits the floor alone.- The floor is long enough for one try and its answer from a chat that is up.
- It is short enough that an operator who pressed Ctrl+C twice gets the terminal back soon.
- It costs nothing when the queue is empty: the wait ends as soon as nothing is pending.
- Under
docker stopthe deadline is 5 seconds after the last line, so the floor is never the longer of the two.
- One count, waiting in order, growing in place.
- When a line does not fit, the count joins the queue at once, behind the lines that filled it.
- Each later drop adds to that same count for as long as it waits.
- It does not take a slot: the queue holds 100 lines, and at most one count beside them.
- Its text is built when the thread takes it, and is final from then on. A drop after that starts a new count, so each drop is counted once, by one message.
- A count is dated by its first drop. If it goes a minute late, its lateness note covers every drop in it.
- A drop joins the waiting count even when lines were queued behind it since its first drop. The chat then reads the count before a line the log has before some of the drops it counts. A new count after each such line would keep the log's order exactly, but in a long flood with a slow chat every freed slot would alternate a line and a count. The count says how many were dropped; the log has the order.
- No try begins after the stop's report, and each message gets one
red line.
- The stop sets a flag in the same step in which it counts what is left.
- From then on the thread takes nothing more from the queue and ends.
Deliveryreads the flag before each try, and stops trying when it is set.Deliveryreturns the report of a message it did not send, rather than printing it. The thread prints it only if the stop has not counted that message. It decides this in the same step in which it marks the message as handled, under the condition the stop counts under. A try under way at the stop that then fails, or raises, adds no second red line to the stop's.- A stop still waiting also waits for that print. The thread prints
outside the condition, which
notifytakes on the trading path, so it marks the report as being printed first. The stop waits for the mark to clear, without counting the message, which is handled. A failed login'snot sentline therefore reaches the log before the process ends. Only a stop whose time has run out can return while such a line is still being printed.
Deliverymoves to a module of its own,delivery. It decides how often and how long to try one message; the notifier decides which message goes and when the tries must stop. The notifier's file had reached 300 lines, the limit for a code file.SendFailure, theChatport andnamed, the words a report names a message by, stay innotifier.- The flag is a
threading.Event, set while the condition is held.Deliveryreads it before each try without taking the lock thatnotifytakes on the trading path. - The flag reaches
Deliveryas a third argument, the event'sis_set. Building one flag in the composition root and handing it to both would have put the notifier's state in two owners.
- The flag is a
- A third Ctrl+C cuts the stop's wait, not its report. An operator who presses it during the floor wants out sooner, and still has to learn what was lost. The stop catches the interrupt around its wait, then counts and reports as usual.
- A line queued after the stop is reported, not queued. The thread
has ended, so a queued line would wait for nobody, without a word. It
gets the usual red report instead:
Telegram: the "Bot failed" message was not sent: the bot had stopped. Today nothing callsnotifythat late, because the exit function runs only once every webhook thread has ended. The report covers a later change that would.
flowchart TD
N["notify(text)"] --> S{"stopped?"}
S -- "yes" --> R1["red line:<br/>not sent: the bot had stopped"]
S -- "no" --> F{"fewer than 100<br/>lines waiting?"}
F -- "yes" --> L["queue the line"]
F -- "no" --> C{"a count<br/>waiting?"}
C -- "no" --> NC["queue a count of 1,<br/>dated now"]
C -- "yes" --> INC["add 1 to it"]
NC --> R2["red line:<br/>not sent: the queue of 100 was full"]
INC --> R2
L --> T["the thread takes the oldest;<br/>a count's text is final now"]
NC --> T
T --> D["Delivery: before each try,<br/>stopped? then stop trying"]
D --> REP{"a report returned,<br/>and the stop has<br/>not counted it?"}
REP -- "yes" --> R3["red line: not sent"]
REP -- "no" --> T
R3 --> T
sequenceDiagram
participant M as main
participant P as Python at exit
participant B as BackgroundNotifier
participant W as notifier thread
M->>B: end(5) in its finally
Note over P: waits for every webhook thread
P->>B: stop()
B->>B: wait until max(deadline, now + 2 s),<br/>or until nothing is pending
B->>B: count what is left, set stopped
B-->>P: red lines: lines not sent, dropped count
W->>B: next message?
B-->>W: stopped: the thread ends
What changes¶
| Before | After | |
|---|---|---|
A signal's line queued after end's seconds ran out |
Reported as not sent at once | Gets up to 2 seconds, then is reported if still pending |
No end before the exit function |
The stop waits no time | The stop waits up to 2 seconds |
| A full queue drops lines | One count queued once the queue is empty, in a free slot | One count queued at the first drop, behind the lines that filled the queue; it grows while it waits and takes no slot |
| What the chat sees in a flood | The lines, then the count | The lines that fit, the count, then the lines queued after it |
| The stop's report, then a try | The thread could still send a message the report named | No try begins after the report |
| A try under way at the stop fails after it | Two red lines for one message: the stop's and the delivery's | The stop's only |
| A third Ctrl+C during the stop's wait | (no wait to cut) | The wait ends, and the report still comes |
notify after the stop |
Queued; sent if the thread got to it | A red line: not sent: the bot had stopped |
The texts of every report are unchanged. The one new text is the last row's reason.
Tests¶
tests/unit/adapters/test_notifier_stop.pyholds the stop's tests:- the ones that moved from
test_background_notifier.py; - a line queued after the seconds ran out gets the floor;
- without an
endthe stop waits the floor; - the floor does not cut the seconds
endset aside; - after the stop the thread ends, and the message behind the one being sent is never sent;
Deliverylearns of the stop between its tries;- a try under way at the stop that then fails, or raises, adds no second red line;
- a delivery's report is printed when no stop counted the message;
- an interrupt during the stop's wait cuts the wait, not the report;
- the stop waits for a delivery's report to be printed;
- a line after the stop is reported;
- a count being sent and one waiting are reported as one count, the line between them as a line.
- the ones that moved from
tests/unit/adapters/test_background_notifier.pypins the count:- it waits where the first drop was;
- it takes no line's slot;
- it grows while it waits, dated by its first drop;
- a drop while it is sent starts a new count;
- the old test of a count that waited for an empty queue is replaced, since it pinned the order this ADR reverses.
tests/unit/adapters/test_notifier.py:- no try begins once the notifier has stopped;
- a stop during the wait between tries ends them;
- its other tests read the report
Deliveryreturns, where they read the one it printed before.
tests/characterization/test_notifier_at_exit.pyruns a real interpreter twice more, withend(0)and with noend. In both, the last line of a signal still running is sent before the process ends. Its fakeurlopennow takes 0.3 seconds, so the stop really has to wait. Both runs failed before this change.tests/unit/adapters/notifier_doubles.pyholds the gated deliveries both unit files share, and finds the notifier's thread by its name.
Each of these mutations of the code fails at least one test:
- no floor once the seconds ran out;
- no seconds without an
end; - the floor in place of the seconds
endset aside; Deliverytrying after the stop;- the thread going on after the stop;
Deliverynever told of the stop;- a delivery's report printed after the stop counted its message;
- a delivery's report never printed;
- a third Ctrl+C skipping the report;
- the stop not waiting for a report being printed;
- a report printed with no mark for the stop to wait on;
- a line after the stop queued for nobody;
- the count taking a line's slot;
- a new count for every drop;
- the count in flight still taking drops;
- the count dated by its last drop;
- the stop forgetting the message being sent;
- the stop reporting counts as lines;
- the composition root not passing the floor;
- a floor of 0.
What this does not cover¶
- A try already under way when the stop reports. The flag cannot recall a request already sent: its message can still arrive after the report has named it as not sent. The same holds for a try that began a moment before the flag was set.
- A thread that outlives
mainunderdocker stop. The floor rests on uvicorn waiting for every request at a SIGTERM, which holds for 0.54.0, and on anyio ending its worker threads with the event loop. If a later version let a thread run on, and Python reached the exit function more than 3 seconds afterend, the floor would add up to 2 seconds past the deadline, insidedocker stop's 10. A run that long could then be killed before its report. - A connection that hangs. A try can take the full 10-second timeout, longer than the floor. Its message is reported as not sent, and may still arrive.
- A signal that holds
docker stoppast its grace period. The server waits for it, soendcomes late. The process can then be killed inside the stop's 5 seconds, before any report. That was so before; ADR 0027 lists it. - Which messages a flood dropped. The count says how many, and the log has each one.
Alternatives considered¶
- Count the stop's seconds from the exit function, not from
end. Rejected. Underdocker stop, Python reaches the exit function only after the webhook threads, so those seconds would run past the grace period. ADR 0027 choseendfor that reason. - A floor of the full 5 seconds. Rejected. The floor applies only after a second Ctrl+C, so it should be short. One try and its answer fit in 2 seconds.
- Keep the count out of the queue, sent whenever a slot is free. Rejected. It would go ahead of lines logged before it, or wait like the old count. A queued count goes out where the first drop was, after the lines logged before it.
- A new count after each line queued behind the waiting one. Rejected, above: in a long flood the chat would get a count after every line.
- Let the count take a slot. Rejected. In a flood, each slot freed would go to a count rather than a line.
- Wait at the stop for the try under way. Rejected. A hung connection holds it for 10 seconds, the whole floor and more.
- A flag shared through the composition root. Rejected, as above. The third argument keeps the stop in one owner.
Deliverychecking the flag before it prints its report. Rejected: the stop could count the message between that check and the print, and the message would get two red lines. Only a decision made under the stop's own condition is exact.- Cap the floor at the deadline when
endran. Rejected: a signal's line queued after the deadline, the case the floor is for, would again get no time.