Attempting to "upgrade" from `main` to 4.x should abort; Django does
not prevent running old code against the new database (though it
likely errors at runtime), and `./manage.py migrate` from the old
version during the "upgrade" does not downgrade the database, since
the migrations are entirely missing in that directory, so don't get
reversed.
Compare the list of applied migrations to the list of on-disk
migrations, and abort if there are applied migrations which are not
found on disk.
Fixes: #19284.
The RabbitMQ docs state ([1]):
RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to
determine whether they are allowed to communicate with each
other. [...] The cookie is just a string of alphanumeric
characters up to 255 characters in size. It is usually stored in a
local file.
...and goes on to state (emphasis ours):
If the file does not exist, Erlang VM will try to create one with
a randomly generated value when the RabbitMQ server starts
up. Using such generated cookie files are **appropriate in
development environments only.**
The auto-generated cookie does not use cryptographic sources of
randomness, and generates 20 characters of `[A-Z]`. Because of a
semi-predictable seed, the entropy of this password is thus less than
the idealized 26^20 = 94 bits of entropy; in actuality, it is 36 bits
of entropy, or potentially as low as 20 if the performance of the
server is known.
These sizes are well within the scope of remote brute-force attacks.
On provision, install, and upgrade, replace the default insecure
20-character Erlang cookie with a cryptographically secure
255-character string (the max length allowed).
[1] https://www.rabbitmq.com/clustering.html#erlang-cookie
5c450afd2d, in ancient history, switched from `check_call` to
`check_output` and throwing away its result.
Use check_call, so that we show the steps to (re)starting the server.
As a consequence:
• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We've had a number of unhappy reports of upgrades failing due to
webpack requiring too much memory. While the previous commit will
likely fix this issue for everyone, it's worth improving the error
message for failures here.
We avoid doing the stop+retry ourselves, because that could cause an
outage in a production system if webpack fails for another reason.
Fixes#20105.
Since the upgrade to Webpack 5, we've been seeing occasional reports
that servers with roughly 4GiB of RAM were getting OOM kills while
running webpack.
Since we can't readily optimize the memory requirements for webpack
itself, we should raise the RAM requirements for doing the
lower-downtime upgrade strategy.
Fixes#20231.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Add a helper `run_psql_as_postgres` function in
`scripts/lib/zulip_tools.py`. This is preparatory refactoring for the
work to add custom database and user names.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.
Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.
Fixes#15819.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell. Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.
Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.
Fixes#18258.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files. However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.
Fix the supervisor configuration file paths.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian. We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.
Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.
Fixes#18179.
The class names need to be renamed even if we are not about to run
puppet ourselves; otherwise, deployments which rely on running puppet
themselves will still have the wrong class names.
Using `config_file.write()` only writes out what python stored of the
file; as such, it strips all comments and whitespace.
Use `crudini --set`, which only modifies the line whose contents are
changed.
We can compute the intended number of processes from the sharding
configuration. In doing so, also validate that all of the ports are
contiguous.
This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding. Having behaviour which
changes merely based on if an empty section exists is surprising.
This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.
After this commit, configuring sharding can be done by adding to
`zulip.conf`:
```
[tornado_sharding]
9800 = # default
9801 = other_realm
```
Followed by running `./scripts/refresh-sharding-and-restart`.
We would prefer to use the postgres packages from Postgres themselves,
if available. However, this requires ensures that, for existing
installs, we preserve the same version of postgres as their base
distribution installed.
Move the version-determination logic from being computed at puppet
interpolation time, to being computed at install time and pinned into
zulip.conf.
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
