mirror of
https://github.com/zulip/zulip.git
synced 2025-11-01 20:44:04 +00:00
queue: Set a max consume timeout with SIGALRM.
SIGALRM is the simplest way to set a specific maximum duration that queue workers can take to handle a specific message. This only works in non-threaded environments, however, as signal handlers are per-process, not per-thread. The MAX_CONSUME_SECONDS is set quite high, at 10s -- the longest average worker consume time is embed_links, which hovers near 1s. Since just knowing the recent mean does not give much information[1], it is difficult to know how much variance is expected. As such, we set the threshold to be such that only events which are significant outliers will be timed out. This can be tuned downwards as more statistics are gathered on the runtime of the workers. The exception to this is DeferredWorker, which deals with quite-long requests, and thus has no enforceable SLO. [1] https://www.autodesk.com/research/publications/same-stats-different-graphs
This commit is contained in:
committed by
Tim Abbott
parent
baf882a133
commit
d47637fa40
@@ -46,6 +46,7 @@ class Command(BaseCommand):
|
||||
worker: QueueProcessingWorker = NoopWorker(count, options["slow"])
|
||||
if options["batch"]:
|
||||
worker = BatchNoopWorker(count, options["slow"])
|
||||
worker.ENABLE_TIMEOUTS = True
|
||||
worker.setup()
|
||||
assert worker.q is not None
|
||||
assert worker.q.channel is not None
|
||||
|
||||
Reference in New Issue
Block a user