Commit Graph

1438 Commits

Author SHA1 Message Date
Anders Kaseorg
4ebc734632 apt-repos: Temporarily work around Ubuntu 24.04 non-installable gnupg.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-31 13:51:31 -07:00
Anders Kaseorg
5d6e616808 Revert "apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04)."
This reverts commit 44e38e8ea7.
2024-03-30 11:29:07 -07:00
Anders Kaseorg
da984d0ab3 install-node: Upgrade Node.js from 20.11.1 to 20.12.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-29 09:37:38 -07:00
Anders Kaseorg
708c3e7182 node_cache: Invoke pnpm via corepack.
This avoids prompting the user that corepack is about to download
pnpm.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-29 09:37:38 -07:00
Anders Kaseorg
44e38e8ea7 apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04).
This package is replaced by libmagic1t64 1:5.45-3 for the Ubuntu
64-bit time_t transition, but hasn’t been deleted from the archive
yet.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 16:47:30 -07:00
Tim Abbott
8f0f41e236 pythonrc: Add various frequently handy imports. 2024-03-27 12:49:24 -07:00
Anders Kaseorg
9015cd7da3 apt-repos: Use Ubuntu 22.04 Teleport repo on Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2024-03-25 17:57:24 -04:00
Tim Abbott
2980d14cb4 scripts: Use Apache Arrow from Groonga repository.
This lets us avoid CI failures related to that repository's signing
key having expired. See https://github.com/apache/arrow/issues/40744.
2024-03-24 20:46:24 -07:00
Anders Kaseorg
a1a341f0ae ruff: Fix UP032 Use f-string instead of format call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Alex Vandiver
c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver
0f70ab7679 config: Use logic from zulip_tools, to support bools. 2024-03-15 15:34:12 -07:00
Alex Vandiver
b36f389e9e zulip_tools: Make default default_value be None, not "". 2024-03-15 15:34:12 -07:00
Alex Vandiver
f9f9392b93 zulip_tools: Unify get_config and get_config_bool using overloads. 2024-03-15 15:34:12 -07:00
Alex Vandiver
0ff1fb2e21 zulip_tools: Support non-lowercase true values.
This matches the ruby implementation.
2024-03-15 15:34:12 -07:00
Alex Vandiver
3905416a0e zulip_tools: Fix path to zulipconf.rb.
This was moved in 16aa7c0923.
2024-03-15 15:34:12 -07:00
Alex Vandiver
e0cf82a526 email-mirror-postfix: Use get_config_bool, instead of checking "true". 2024-03-15 15:34:12 -07:00
Alex Vandiver
5123477d55 upgrade-zulip-from-git: Provide a flag to use a local branch.
While this could be done previously by calling
`upgrade-zulip-from-git --remote-url /srv/zulip.git`, the explicit
argument makes this more straightforward, and avoids churning the
`refs/remotes/origin/` namespace.
2024-03-08 12:15:25 -08:00
Alex Vandiver
b0283c42f9 create-database: Treat "zulip" db without "zerver_messages" as empty.
A user who somehow got an empty `zulip` database, but without a
`zerver_messages` table in it, would get stuck in the installer at:

```
++ su postgres -c 'cd / && psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
ERROR:  relation "zulip.zerver_message" does not exist
LINE 1: SELECT COUNT(*) FROM zulip.zerver_message;
                             ^
+ records=
```

Treat a failure to select from `zerver_messages` as having 0 messages,
and continue with the `DROP DATABASE IF EXISTS` / `CREATE DATABASE`
that `create-db.sql` usually does.

Fixes: #29110.
2024-03-08 10:10:19 -08:00
Anders Kaseorg
066ea3ebf9 install: Support Ubuntu 24.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg
55954c9819 build-pgroonga: Upgrade PGroonga from 3.1.0 to 3.1.8.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Alex Vandiver
3480d204de reload-clients: Log how many reload events were actually sent. 2024-03-01 09:31:20 -08:00
Alex Vandiver
2d59e07e58 reload-clients: Only sleep between batches.
This avoids an unnecessary sleep at the end of all of the Tornado
shards.
2024-03-01 09:31:20 -08:00
Anders Kaseorg
82a9fd927b ruff: Fix E226 Missing whitespace around arithmetic operator.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg
553f268b04 ruff: Fix RUF027 Possible f-string without an f prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg
865febb307 ruff: Fix RUF021 Parenthesize when chaining and and or together.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg
570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver
175e61486e log-search: Auto-limit to the correct logfile when searching datetimes. 2024-02-27 17:04:03 -08:00
Alex Vandiver
9eaaacba52 log-search: Return the filter terms, rather than changing them.
This leaves `args.filter_terms` as the raw values the user specified,
before they may have been transformed to the shape that we use for
substring matching.
2024-02-27 17:04:03 -08:00
Alex Vandiver
39a7c5d106 log-search: Add datetime prefix limit. 2024-02-27 17:04:03 -08:00
Alex Vandiver
281b74a264 log-search: Factor out nginx date conversion. 2024-02-27 17:04:03 -08:00
Alex Vandiver
bdc210702c log-search: Document HTTP method limits. 2024-02-27 17:04:03 -08:00
Alex Vandiver
2122780d41 log-search: Fix attempts to limit to events requests in Django logs. 2024-02-26 08:19:57 -08:00
Alex Vandiver
b7e42ca17a log-search: Support user-agent / client limit. 2024-02-26 08:19:57 -08:00
Alex Vandiver
9ca0cd9bbd log-search: Remove old /report/ endpoints. 2024-02-26 08:19:57 -08:00
Anders Kaseorg
94cf99bcf2 install-node: Upgrade Node.js from 20.9.0 to 20.11.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-22 12:46:40 -08:00
Alex Vandiver
ec6f64f7b0 tornado: Perform rolling client restarts after servers are restarted.
Decouple the sending of client restart events from the restarting of
the servers.  Restarts use the new Tornado restart-clients endpoint to
inject "restart" events into queues of clients which were loaded from
the previous Tornado process.  The rate is controlled by the
`application_server.client_restart_rate`, in clients per minute, or a
flag to `restart-clients` which overrides it.  Note that a web client
will also spread its restart over 5 minutes, so artificially-slow
client restarts are generally not very necessary.

Restarts of clients are deferred to until after post-deploy hooks are
run, such that the pre- and post- deploy hooks are around the actual
server restarts, even if pushing restart events to clients takes
significant time.
2024-02-15 15:42:50 -08:00
Alex Vandiver
27d53ecbe1 restart-server: Remove --skip-tornado flag.
This flag was generally used not because we wanted to avoid
restarting Tornado, but because we wanted to avoid increasing load
the server when all of the clients were told to reload.

Since we have laid the groundwork for separately telling Tornado to
tell clients to restart, we remove the --skip-tornado flag; the next
commit will add the ability to skip client restarts.
2024-02-15 15:42:50 -08:00
Alex Vandiver
e6fa1f5436 email_mirror_message: Move endpoint under /api/internal/.
This also has the side effect that it saves the endpint from being
improperly redirected by the HostDomainMiddleware's canonicalization
code.
2024-02-15 15:42:50 -08:00
Alex Vandiver
0115fa9c60 start-server/restart-server: Drop privileges if necessary.
Rather than tell the user to re-run the command as `zulip` instead of
`root`, do the privilege-dropping ourselves.
2024-02-07 12:33:00 -08:00
Alex Vandiver
24d3832926 teleport: Upgrade to Teleport 14. 2024-01-31 16:41:04 -08:00
Anders Kaseorg
93198a19ed requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-29 10:41:54 -08:00
Alex Vandiver
5786a38cec log-search: Match against path without query parameters.
The nginx logs have query parameters; display them, but do not use
them for URL matching.
2024-01-26 09:02:00 -08:00
Alex Vandiver
1ba2f39854 install: Support PostgreSQL 16. 2023-12-23 14:57:12 -08:00
Anders Kaseorg
a13e42f18a docs: Add missing spaces around code spans.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 00:05:44 -08:00
Anders Kaseorg
223b626256 python: Use urlsplit instead of urlparse.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 13:03:07 -08:00
Anders Kaseorg
8a7916f21a python: Consistently use from…import for datetime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 12:01:18 -08:00
Alex Vandiver
7c8eacfa12 queue: Increase paging thresholds for deferred_work queue.
This queue is used to things which definitionally may take longer than
a request, so paging after 60s is rather aggressive.  This is
especially true because this queue has a very long tail of very slow
tasks -- p99 of task time in this queue is 8.5s, while p99.9 is 197s.

Raise the paging threshold to 15 minutes.  While there are
semi-user-facing tasks which use this queue (primarily marking
messages as read), those being delayed for minutes is already a real
possibility if they are stuck behind a large realm export -- and this
is not a situation which should necessarily page, since it is not
solvable by the administrator.
2023-11-09 13:51:00 -08:00
Anders Kaseorg
dcdbb39c31 install-node: Upgrade Node.js from 18.18.0 to 20.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-08 16:11:15 -08:00
Alex Vandiver
fb80ecd9c3 upgrade-zulip: Defer cache-filling to just outside the critical period.
Filling caches needs to happen close to when the server is restarted,
as the gap opens us up to race conditions with user modifications.  If
there are migrations, however, it must happen within the critical
period after the migrations are applied.

Move the call to fill the caches to within the `shutdown_server`
function, so that we push it as close to the server shutdown as
possible.
2023-11-08 11:00:00 -08:00
Anders Kaseorg
a50eb2e809 mypy: Enable new error explicit-override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-12 12:28:41 -07:00