This ensures that anything that looks like a full release will
identify itself as such. It requires that the version have a tag at
the current commit (though it doesn't need to be pushed yet), as well
as the commits exist in `main` or a release branch in the remote.
(cherry picked from commit 2d132bf342)
When editing an old message in a private stream with protected
history, the server would incorrectly send an API event including the
edited message to all of the stream’s current subscribers, including
those who should not have access to the old message. This API event is
ignored by official clients, so it could only be observed by a user
using a modified client or their browser’s developer tools.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`postgresql-14.4` is a notable upgrade in the PostgreSQL series, as it
fixes potential database corruption from `CREATE INDEX CONCURRENTLY`
statements which are run while rows are modified[1]. However, it also
requires an upgrade from `libllvm9` to `libllvm10`, which means it is
not installed by a mere `apt-get upgrade`.
Add the `--with-new-pkgs` flag to all of the potentially relevant
`apt-get upgrade` calls, so that this (and similar) packages are
upgraded successfully.
[1]: https://www.postgresql.org/docs/release/14.4/
(cherry picked from commit a35af3f38b)
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
(cherry picked from commit 09dc166b45)
Using http://localhost:9991 is incorrect - e.g. messages sent with file
urls constructed trigger do_claim_attachments to be called with empty
list in potential_path_ids.
realm.host should be used in all these places, like in the other tests
in the file.
(cherry picked from commit 5ff4754090)
30457ecd02 removed the `--mirror` from
initial clones, but did not add back `--bare`, which `--mirror`
implies. This leads to `/srv/zulip.git` having a working tree in it,
with a `/srv/zulip.git/.git` directory.
This is mostly harmless, and since the bug was recent, not worth
introducing additional complexity into the upgrade process to handle.
Calling `git clone --bare`, however, would clone the refs into
`refs/heads/`, not the `refs/remotes/origin/` we want. Instead, use
`git init --bare`, followed by `git remote add origin`. The remote
will be fetched by the usual `git fetch --all --prune` which is below.
(cherry picked from commit 5bdc4b3562)
While the `remote.origin.mirror` boolean being set is a very good
proxy for having been cloned with `--mirror`, is technically only used
when pushing into the remote[1]. What we care about is if fetches
from this remote will overwrite `refs/heads/`, or all of `refs/` --
the latter of which is most likely, from having run `git clone
--bare`.
Detect either of these fetch refspecs, and not the mirror flag. We
let the upgrade process error out if `remote.origin.fetch` is unset,
as that represents an unexpected state. We ignore failures to unset
the `remote.origin.mirror` flag, in case it is not set already.
[1]: https://git-scm.com/docs/git-config#Documentation/git-config.txt-remoteltnamegtmirror
(cherry picked from commit 1639792e9e)
The local `/srv/zulip.git` directory has been cloned with `--mirror`
since it was first created as a local cache in dc4b89fb08. This
made some sense at the time, since it was purely a cache of the
remote, and not a home to local branches of its own.
That changed in 3f83b843c2, when we began using `git worktree`,
which caused the `deployment-...` branches to begin being stored in
`/src/zulip.git`. This caused intermixing of local and remote
branches.
When 02582c6956 landed, the addition of `--prune` caused all but the
most recent deployment branch to be deleted upon every fetch --
leaving previous deployments with non-existent branches checked out:
```
zulip@example-prod-host:~/deployments/last$ git status
On branch deployment-2022-04-15-23-07-55
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: .browserslistrc
new file: .codecov.yml
new file: .codespellignore
new file: .editorconfig
[...snip list of every file in repo...]
```
Switch `/srv/zulip.git` to no longer be a `--mirror` cache of the
origin. We reconfigure the remote to drop `remote.origin.mirror`, and
delete all refs under `refs/pulls/` and `refs/heads/`, while
preserving any checked-out branches. `refs/pulls/`, if the remote is
the canonical upstream, contains _tens of thousands_ of refs, so
pruning those refs trims off 20% of the repository size.
Those savings require a `git gc --prune=now`, otherwise the dangling
objects are ejected from the packfiles, which would balloon the
repository up to more than three times its previous size. Repacking
the repository is reasonable, in general, after removing such a large
number of refs -- and the `--prune=now` is safe and will not lose
data, as the `--mirror` was good at ensuring that the repository could
not be used for any local state.
The refname in the upgrade process was previously resolved from the
union of local and remote refs, since they were in the same namespace.
We instead now only resolve arguments as tags, then origin branches;
this means that stale local branches will be skipped. Users who want
to deploy from local branches can use `--remote-url=.`.
Because the `scripts/lib/upgrade-zulip-from-git` file is "stage 1" and
run from the old version's code, this will take two invocations of
`upgrade-zulip-from-git` to take effect.
Fixes#21901.
(cherry picked from commit 30457ecd02)
This adds a --skip-restart which makes `deployments/next` in a state
where it can be restarted into, but holds off on conducting that
restart.
This requires many of the same guarantees as `--skip-tornado`, in
terms of there being no Puppet or database schema changes between the
versions. Enforce those with `--skip-restart`, and also broaden both
flags to prevent other, less common changes which nonetheless
potentially might affect the other deploy.
(cherry picked from commit 6337f17923)
Because Tornado and Django use memcached as a shared cache for
checking session information, they must agree on the prefix used to
store those values.
Subsequent commits will work to ensure that it is always _safe_ to
share that cache.
(cherry picked from commit ef7c2ea0ea)
These are expensive, and moving them to one explicit call early has
considerable time savings in the critical period:
```
$ hyperfine './manage.py fill_memcached_caches' './manage.py fill_memcached_caches --skip-checks'
Benchmark #1: ./manage.py fill_memcached_caches
Time (mean ± σ): 5.264 s ± 0.146 s [User: 4.885 s, System: 0.344 s]
Range (min … max): 5.119 s … 5.569 s 10 runs
Benchmark #2: ./manage.py fill_memcached_caches --skip-checks
Time (mean ± σ): 3.090 s ± 0.089 s [User: 2.853 s, System: 0.214 s]
Range (min … max): 2.950 s … 3.204 s 10 runs
Summary
'./manage.py fill_memcached_caches --skip-checks' ran
1.70 ± 0.07 times faster than './manage.py fill_memcached_caches'
```
(cherry picked from commit fa77be6e6c)
Treating the restart as a start is important in reducing the critical
period during upgrades -- we call restart even when we suspect the
services are stopped, because puppet has a small possibility of
placing them in indeterminate state. However, restart orders the
workers first, then tornado/django, which prolongs the outage.
Recognize when no services are currently started, and switch to acting
like a start, not a restart, which places tornado/django first.
(cherry picked from commit 3928606886)
This hides ugly output if the services were already stopped:
```
2022-03-25 23:26:04,165 upgrade-zulip-stage-2: Stopping Zulip...
process-fts-updates: ERROR (not running)
zulip-django: ERROR (not running)
zulip_deliver_scheduled_emails: ERROR (not running)
zulip_deliver_scheduled_messages: ERROR (not running)
Zulip stopped successfully!
```
Being able to skip having to shell out to `supervisorctl`, if all
services are already stopped is also a significant performance
improvement.
(cherry picked from commit 3717c329b8)
This optimization makes sense if the server is already running, but if
it is already stopped, it is just prolonging the downtime.
(cherry picked from commit b15d8e0118)
Backups are written every 16k of WAL archive, and by default do not
have an upper limit on how out of date they are, as `archive_timeout`
defaults to 0.
Also emphasize that these are streaming backups, not just one
point-in-time backup daily.
Fixes#21976.
(cherry picked from commit 18230fcd99)
Commit a9ca5f603b (#15863) incorrectly
translated this; stream_name_error is not a jQuery object.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f2d0ae3255)
This reverts commit 9c8d2b7be3 (#21115).
The PostgreSQL fix was released 2022-05-12.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f8957863a2)
This is a reprise of c97162e485, but for the case where certbot
certs are no longer in use by way of enabling `http_only` and letting
another server handle TLS termination.
Fixes: #22034.
This allows system-level configuration to be done by `apt-get install`
of nginx modules, which place their load statements in this directory.
The initial import in ed0cb0a5f8 of the stock nginx config omitted
this include -- one potential explanation was in an effort to reduce
the memory footprint of the server.
The default nginx install enables:
50-mod-http-auth-pam.conf
50-mod-http-dav-ext.conf
50-mod-http-echo.conf
50-mod-http-geoip2.conf
50-mod-http-geoip.conf
50-mod-http-image-filter.conf
50-mod-http-subs-filter.conf
50-mod-http-upstream-fair.conf
50-mod-http-xslt-filter.conf
50-mod-mail.conf
50-mod-stream.conf
While Zulip doesn't actively use any of these, they likely don't do
any harm to simply be loaded -- they are loaded into every nginx by
default.
Having the `modules-enabled` include allows easier extension of the
server, as neither of the existing wildcard
includes (`/etc/nginx/conf.d/*.conf` and
`/etc/nginx/zulip-include/app.d/*.conf`) are in the top context, and
thus able to load modules.
(cherry picked from commit 62f234328d)
Initializing the Zulip client opens a long-lived TCP connection due to
connection pooling in urllib3. In Github Actions, the network kills
such requests after ~270s, making the later `send_message` call fail.
Use a singular call to `zulip.Client()` early on to verify the
credentials, and do not cache the resulting client object. Instead,
re-create it during the final step when it is needed, so we do not run
afoul of bad TCP connection state.
This would ideally be fixed via connection keepalive or retry at the
level of the Zulip module.
(cherry picked from commit ff647dff03)
According to the documentation: “Pika does not have any notion of
threading in the code. If you want to use Pika with threading, make
sure you have a Pika connection per thread, created in that thread. It
is not safe to share one Pika connection across threads, with one
exception: you may call the connection method add_callback_threadsafe
from another thread to schedule a callback within an active pika
connection.”
https://pika.readthedocs.io/en/stable/faq.html
This also means that synchronous Django code running in Tornado will
use its own synchronous SimpleQueueClient rather than sharing the
asynchronous TornadoQueueClient, which is unfortunate but necessary as
they’re about to be on different threads.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c263bfdb41)
