paulmataruso/zulip
1
0
Fork 0
mirror of https://github.com/zulip/zulip.git synced 2025-11-03 05:23:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

puppet: Use lazy-apps and uwsgi control sockets for rolling reloads.

Browse Source 
Restarting the uwsgi processes by way of supervisor opens a window
during which nginx 502's all responses.  uwsgi has a configuration
called "chain reloading" which allows for rolling restart of the uwsgi
processes, such that only one process at once in unavailable; see
uwsgi documentation ([1]).

The tradeoff is that this requires that the uwsgi processes load the
libraries after forking, rather than before ("lazy apps"); in theory
this can lead to larger memory footprints, since they are not shared.
In practice, as Django defers much of the loading, this is not as much
of an issue.  In a very basic test of memory consumption (measured by
total memory - free - caches - buffers; 6 uwsgi workers), both
immediately after restarting Django, and after requesting `/` 60 times
with 6 concurrent requests:

                      |  Non-lazy  |  Lazy app  | Difference
    ------------------+------------+------------+-------------
    Fresh             |  2,827,216 |  2,870,480 |   +43,264
    After 60 requests |  3,332,284 |  3,409,608 |   +77,324
    ..................|............|............|.............
    Difference        |   +505,068 |   +539,128 |   +34,060

That is, "lazy app" loading increased the footprint pre-requests by
43MB, and after 60 requests grew the memory footprint by 539MB, as
opposed to non-lazy loading, which grew it by 505MB.  Using wsgi "lazy
app" loading does increase the memory footprint, but not by a large
percentage.

The other effect is that processes may be served by either old or new
code during the restart window.  This may cause transient failures
when new frontend code talks to old backend code.

Enable chain-reloading during graceful, puppetless restarts, but only
if enabled via a zulip.conf configuration flag.

Fixes #2559.

[1]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#chain-reloading-lazy-apps
This commit is contained in:
Alex Vandiver
2021-12-31 20:20:49 -08:00
committed by Tim Abbott
parent 4aaa250623
commit 6218ed91c2
4 changed files with 45 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
puppet/zulip/manifests/app_frontend_base.pp
View File

@@ -119,6 +119,12 @@ class zulip::app_frontend_base {
notify => Service[$zulip::common::supervisor_service],
}
$uwsgi_rolling_restart = zulipconf('application_server', 'rolling_restart', '')
if $uwsgi_rolling_restart == '' {
file { '/home/zulip/deployments/uwsgi-control':
ensure => absent,
}
}
$uwsgi_listen_backlog_limit = zulipconf('application_server', 'uwsgi_listen_backlog_limit', 128)
$uwsgi_buffer_size = zulipconf('application_server', 'uwsgi_buffer_size', 8192)
$uwsgi_processes = zulipconf('application_server', 'uwsgi_processes', $uwsgi_default_processes)

7
puppet/zulip/templates/uwsgi.ini.template.erb
View File

@@ -16,6 +16,13 @@ gid=zulip
stats=/home/zulip/deployments/uwsgi-stats
<% if @uwsgi_rolling_restart != '' -%>
master-fifo=/home/zulip/deployments/uwsgi-control
# lazy-apps are required for rolling restarts:
# https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy
lazy-apps=true
<% end -%>
ignore-sigpipe = true
ignore-write-errors = true
disable-write-exception = true