Compare commits

...

128 Commits
10.1 ... 4.7

Author SHA1 Message Date
Alex Vandiver
4e724c1ec6 Release Zulip Server 4.7 2021-10-04 17:31:57 +00:00
Alex Vandiver
e2d303c1bb CVE-2021-41115: Use re2 for user-supplied linkifier patterns.
Zulip attempts to validate that the regular expressions that admins
enter for linkifiers are well-formatted, and only contain a specific
subset of regex grammar.  The process of checking these
properties (via a regex!) can cause denial-of-service via
backtracking.

Furthermore, this validation itself does not prevent the creation of
linkifiers which themselves cause denial-of-service when they are
executed.  As the validator accepts literally anything inside of a
`(?P<word>...)` block, any quadratic backtracking expression can be
hidden therein.

Switch user-provided linkifier patterns to be matched in the Markdown
processor by the `re2` library, which is guaranteed constant-time.
This somewhat limits the possible features of the regular
expression (notably, look-head and -behind, and back-references);
however, these features had never been advertised as working in the
context of linkifiers.

A migration removes any existing linkifiers which would not function
under re2, after printing them for posterity during the upgrade; they
are unlikely to be common, and are impossible to fix automatically.

The denial-of-service in the linkifier validator was discovered by
@erik-krogh and @yoff, as GHSL-2021-118.
2021-10-04 17:24:37 +00:00
Alex Vandiver
d3091a6096 requirements: Add google-re2, a drop-in replacement for re using re2.
re2[1] compiles (strictly) regular expressions to deterministic finite
automata, which guarantees linear-time behavior; `google-re2` is a
drop-in replacement for the `re` module which uses re2 under the hood.

[1]: https://github.com/google/re2/
2021-10-02 01:01:14 +00:00
Alex Vandiver
313bcfd02a github: Ignore CodeQL analysis in private repos.
CodeQL only runs in public repos; private forks will otherwise error
their CI runs.

(cherry picked from commit acbe7ae7a8)
2021-10-01 18:00:52 -07:00
Gaurav Pandey
09bfd485e9 ci: Remove unnecessary steps from production upgrade script.
This removes some steps which are no longer necessary to be run
in the production upgrade script. The steps were used due to
errors related to supervisor failing to restart which was resolved
in the commit 08c39a7388.

(cherry picked from commit dc2066c7e8)
2021-10-01 18:00:52 -07:00
Anders Kaseorg
576ae9cc9f ci: Use apt-get -y in production-upgrade test.
We currently configure ‘APT::Get::Assume-Yes’ in our custom Docker
image, but this is the only place we rely on it (outside of the
Dockerfile itself), and it’s better not to.

Also ‘apt-get remove && apt-get purge’ is the same as just ‘apt-get
purge’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit db476bdc51)
2021-10-01 18:00:52 -07:00
Alex Vandiver
300447ddd9 ci: Use an init process to reap defunct processes.
When Github Actions run in Docker, the default pid 1 entrypoint is
`tail -f /dev/null`.  PID 1 is responsible for propagating signals to
its children, and calling `waitpid()` on defunct processes; `tail`
does not do these things.  This results in zombie processes piling up
inside the container, which is not an issue in most contexts.

However, it affects `start-stop-daemon`, which hangs when stopping
daemon processes, as they are never reaped.  This appears in CI as
`/etc/init.d/supervisor restart` never being able to succeed.

Run the docker container with `--init`, which spawns a
`/sbin/docker-init` PID 1 to handle the job of an init process.

(cherry picked from commit 2daad58afa)
2021-10-01 18:00:52 -07:00
Gaurav Pandey
f8149b0d5a ci: Add prod upgrade step to prod suite.
This adds a check in the current production suite of
CI that upgrades a previous release of zulip server
with a newer one.

Fixes #18346.

(cherry picked from commit e648ad3477)
2021-10-01 18:00:52 -07:00
Priyank Patel
b579dad7d9 github-actions: Upgrade styfle/cancel-workflow-action.
(cherry picked from commit 05510a8c04)
2021-10-01 18:00:52 -07:00
Priyank Patel
fdfabb800d github-actions: Ensure cancel previous run job never fails.
(cherry picked from commit 607110ca33)
2021-10-01 18:00:52 -07:00
Tim Abbott
2c4156678c docs: Inline some upgrade instructions.
It feels like the "Same as" content was unnecessarily requiring the
user to bounce around in these cases.

(I've left the "Same as" text for the Ubuntu ones, where it's two
steps in a row to follow).
2021-10-01 11:10:13 -07:00
Gaurav Pandey
0a87276a27 docs: Document upgrade steps from buster to bullseye.
Fixes #17863.
2021-10-01 11:10:12 -07:00
Tim Abbott
19aed43817 version: Update version after 4.6 release. 2021-09-23 16:14:53 -07:00
Tim Abbott
d370aefe3a Release Zulip Server 4.6. 2021-09-23 16:09:51 -07:00
Anders Kaseorg
0f5657b0ed setup_venv: Skip virtualenv’s automatic download of setuptools.
It recently started failing on Debian 10 (buster).  We immediately
follow this by replacing these packages with our own versions from
pip.txt, anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 902883d818)
2021-09-23 15:06:39 -07:00
Tim Abbott
24277a144e outgoing webhooks: Fix inconsistencies with Slack's API.
Apparently, our slack compatible outgoing webhook format didn't
exactly match Slack, especially in the types used for values.  Fix
this by using a much more consistent format, where we preserve their
pattern of prefixing IDs with letters.

This fixes a bug where Zulip's team_id could be the empty string,
which tripped up using GitLab's slash commands with Zulip.

Fixes #19588.
2021-09-23 14:49:36 -07:00
Tim Abbott
df8b8b9836 i18n: Update translation data from Transifex. 2021-09-23 12:17:05 -07:00
Anders Kaseorg
64fab06adb ci: Remove legacy-os test.
As of yesterday, the GitHub Actions ubuntu-16.04 environment has been
removed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e96abc3c5a)
2021-09-22 16:00:30 -07:00
Gaurav Pandey
9391840d34 docs: Add documentation for bullseye support.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.

Since now bullseye is released as a stable
version, it's support can be documented.

(cherry picked from commit 502697d239)
2021-09-14 22:02:48 +00:00
Eeshan Garg
658e641d12 docs: Indicate latest Zulip version in installation and upgrade docs.
With copy-editing from tabbott, and also a migration to use
LATEST_RELEASE_VERSION, which will be correct even on the /latest/
paths.

Fixes #19695.

(cherry picked from commit 3b1cb0b25a)
2021-09-10 17:07:53 -07:00
Alex Vandiver
467723145b tools: Switch to download.zulip.com from www.zulip.org.
(cherry picked from commit 7d7d727865)
2021-09-10 17:07:34 -07:00
Anders Kaseorg
4ce37176db docs: Migrate from recommonmark to MyST-Parser.
Recommonmark is no longer maintained, and MyST-Parser is much more
complete.

https://myst-parser.readthedocs.io/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 16:12:52 -07:00
Anders Kaseorg
82bf185b1b lint: Add Markdown files to Prettier linter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c3448370a4)
2021-09-10 16:02:22 -07:00
Anders Kaseorg
d81ce3ba76 docs: Format Markdown with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a4dbc1edd4)
2021-09-10 16:02:22 -07:00
Anders Kaseorg
aa6e70382d docs: Apply sentence single-spacing from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 35c1c8d41b)
2021-09-10 16:02:22 -07:00
Anders Kaseorg
0147c6adce docs: Apply bullet style changes from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 915884bff7)
2021-09-10 16:02:22 -07:00
Anders Kaseorg
5ae8fe292d docs: Rewrap to avoid line breaks in inline code spans.
This works around https://github.com/prettier/prettier/issues/11372.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6145fdf678)
2021-09-10 16:02:22 -07:00
Anders Kaseorg
2e8d8ca044 docs: Fix pip compile typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a6e01b35fc)
2021-09-10 16:02:22 -07:00
Shelly
ec0835b947 models: Add setters for is_realm_owner and is_moderator.
This fixes a regression where one could end up deactivating all owners
of a realm when trying to synchronize LDAP with the `is_realm_admin`
flag configured in `AUTH_LDAP_USER_FLAGS_BY_GROUP`.

With tweaks by tabbott to add is_moderator as well.

Fixes #18677.
2021-09-07 17:16:20 -07:00
Anders Kaseorg
e5e7e58c99 docs: Display main branch name as inline code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f4d2d199e2)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
6a6c6d469b Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 646c04eff2)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
34512727e4 integrations: Document default branch name updates.
53e59c8c09

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e5a818b869)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
da3396b4d7 docs: Update links for other repository branch renames.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 1ce12191aa)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
3f1b444a9a prettier: Exclude backend-processed Markdown files.
Our backend processor is not yet sufficiently CommonMark compliant to
accept Prettier formatted Markdown files.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7df2be0965)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
d5a5d0a3e7 prettier: Disable embedded language formatting for Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 08fb51483b)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
bac90f6a9d editorconfig: Restore indent_size = 2 for Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 676fc93e1f)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
9fbfdb0aca docs: Avoid [GitHub] as an internal Markdown link reference name.
To avoid confusing the linter later when Prettier lowercases these.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit fdb7ec8c9e)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
7fe1e55483 reading-list: Inline links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0e4a30daad)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
cb0d29d845 docs: Escape asterisks for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7b3d4ff1de)
2021-09-07 13:56:41 -07:00
Anders Kaseorg
1c83ebfc71 docs: Adjust list item indentation for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0a3e022376)
2021-09-07 13:56:40 -07:00
Anders Kaseorg
8d040d36ed docs: Fix list item indentation mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 4bfffc9f74)
2021-09-07 13:56:40 -07:00
Anders Kaseorg
f4b955f2ee docs: Fix “sinternet” typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 962f14995e)
2021-09-07 13:56:40 -07:00
Anders Kaseorg
aa3f9004ba docs: Add missing blockquote.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit fe3db63381)
2021-09-07 13:56:40 -07:00
Anders Kaseorg
90bf44bde0 docs: Add syntax highlighting languages to code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit b29b6f6526)
2021-09-07 13:56:40 -07:00
Anders Kaseorg
dbb7bc824c docs: Remove trailing newlines from code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 8f2827a65d)
2021-09-07 13:30:53 -07:00
Anders Kaseorg
3d4071fea7 docs: Fix misaligned Markdown source indentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit cb61ea69c6)
2021-09-07 13:30:53 -07:00
Anders Kaseorg
eb7464c68d docs: Fix code span syntax in embedded reST block.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 25c6d5c99c)
2021-09-07 13:30:22 -07:00
Anders Kaseorg
1c2deb0cd3 docs: Move authentication-methods#ldap anchor to appropriate heading.
Commit 30eaed0378 (#15001) incorrectly
inserted a different section between the anchor and the heading.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c3646ec67f)
2021-09-07 13:18:07 -07:00
Anders Kaseorg
26f4ab9a9d upgrade-zulip-from-git: Run git fetch with --prune.
This prevents upgrading to an obsolete version of a branch that has
been deleted or renamed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 02582c6956)
2021-09-01 15:56:45 -07:00
Alex Vandiver
5feba78939 upgrade-postgresql: Do not remove other supervisor configs.
We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.

Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.

It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added.  This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.

Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration.  We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.

Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete.  This brings the documentation into alignment with
what CI is testing.
2021-08-24 19:02:24 -07:00
Mateusz Mandera
04600acbbb management: Rename clear_auth_rate_limit_history command.
(cherry picked from commit 7ef1a024db)
2021-08-23 11:54:09 -07:00
Mateusz Mandera
6ffbb6081b rate_limit: Add management command to reset auth rate limit.
The auth attempt rate limit is quite low (on purpose), so this can be a
common scenario where a user asks their admin to reset the limit instead
of waiting. We should provide a tool for administrators to handle such
requests without fiddling around with code in manage.py shell.

(cherry picked from commit fdbde59b07)
2021-08-23 11:54:02 -07:00
Iam-VM
1f2767f940 migrations: Fix possible 0257_fix_has_link_attribute.py failure.
While it should be an invariant that message.rendered_content is never
None for a row saved to the database, it is possible for that
invariant to be violated, likely including due to bugs in previous
versions of data import/export tools.

While it'd be ideal for such messages to be rendered to fix the
invariant, it doesn't make sense for this has_link migration to crash
because of such a corrupted row, so we apply the similar policy we
already have for rendered_content="".
2021-08-04 12:52:22 -07:00
Tim Abbott
9173ed0fb9 message_edit: Fix live update bug in left sidebar.
We've had for years a subtle bug, where after editing a topic in the
left sidebar that had previously had unread messages (but doesn't
anymore), the old topic might still appear in the sidebar.

The bug was hard to notice except for new organizations or in the
development environment, because the pre-edit topic appeared with a
sort key of -Infinity (that being the max ID in an empty list of
message IDs). But this is an important onboarding bug in reducing
faith in Zulip's topic editing just working, so I'm glad to have it
fixed.

Fixes #11901.
2021-07-29 15:01:39 -07:00
Mateusz Mandera
303bde6c55 email-mirror-postfix: Choose scheme based on http_only config.
Fixes #16659.
If the server is behind a reverse proxy with http_only=True, the
requests made by email-mirror-postfix need to use http, as https
doesn't work.
2021-07-29 15:00:39 -07:00
Tim Abbott
bc118496a2 i18n: Update translation data from Transifex. 2021-07-27 16:35:41 -07:00
Tim Abbott
f118da6b86 version: Update version after 4.5 release. 2021-07-25 16:03:39 -07:00
Tim Abbott
1ba708ca96 Release Zulip Server 4.5. 2021-07-25 15:40:46 -07:00
Alex Vandiver
e156db2bc7 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-25 15:36:11 -07:00
Alex Vandiver
d0235add03 version: Update version after 4.4 release. 2021-07-22 17:10:37 -07:00
Alex Vandiver
a6b06df895 Release Zulip Server 4.4. 2021-07-22 22:32:34 +00:00
Anders Kaseorg
2df2f7eec6 fenced_code: Optimize FENCE_RE to fix cubic worst-case complexity.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 21:31:36 +00:00
Anders Kaseorg
ad858d2c79 fenced_code: Write FENCE_RE with a raw string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 21:31:36 +00:00
Alex Vandiver
5290f17adb puppet: Run the supervisor-restart step only after it is started.
In an initial install, the following is a potential rule ordering:
```
Notice: /Stage[main]/Zulip::Supervisor/File[/etc/supervisor/conf.d/zulip]/ensure: created
Notice: /Stage[main]/Zulip::Supervisor/File[/etc/supervisor/supervisord.conf]/content: content changed '{md5}99dc7e8a1178ede9ae9794aaecbca436' to '{md5}7ef9771d2c476c246a3ebd95fab784cb'
Notice: /Stage[main]/Zulip::Supervisor/Exec[supervisor-restart]: Triggered 'refresh' from 1 event
[...]
Notice: /Stage[main]/Zulip::App_frontend_base/File[/etc/supervisor/conf.d/zulip/zulip.conf]/ensure: defined content as '{md5}d98ac8a974d44efb1d1bb2ef8b9c3dee'
[...]
Notice: /Stage[main]/Zulip::App_frontend_once/File[/etc/supervisor/conf.d/zulip/zulip-once.conf]/ensure: defined content as '{md5}53f56ae4b95413bfd7a117e3113082dc'
[...]
Notice: /Stage[main]/Zulip::Process_fts_updates/File[/etc/supervisor/conf.d/zulip/zulip_db.conf]/ensure: defined content as '{md5}96092d7f27d76f48178a53b51f80b0f0'
Notice: /Stage[main]/Zulip::Supervisor/Service[supervisor]/ensure: ensure changed 'stopped' to 'running'
```

The last line is misleading -- supervisor was already started by the
`supervisor-restart` process on the third line.  As can be shown with
`zulip-puppet-apply --debug`, the last line just installs supervisor
to run on startup, using `systemctl`:
```
Debug: Executing: 'supervisorctl status'
Debug: Executing: '/usr/bin/systemctl unmask supervisor'
Debug: Executing: '/usr/bin/systemctl start supervisor'
```

This means the list of processes started by supervisor depends
entirely on which configuration files were successfully written out by
puppet before the initial `supervisor-restart` ran.  Since
`zulip_db.conf` is written later than the rest, the initial install
often fails to start the `process-fts-updates` process.  In this
state, an explicit `supervisorctl restart` or `supervisorctl reread &&
supervisorctl update` is required for the service to be found and
started.

Reorder the `supervisor-restart` exec to only run after the service is
started.  Because all supervisor configuration files have a `notify`
of the service, this forces the ordering of:

```
(package) -> (config files) -> (service) -> (optional restart)
```

On first startup, this will start and them immediately restart
supervisor, which is unfortunate but unavoidable -- and not terribly
relevant, since the database will not have been created yet, and thus
most processes will be in a restart loop for failing to connect to it.
2021-07-22 14:23:41 -07:00
Alex Vandiver
9824a9d7cf puppet: Work around sysvinit supervisor init bug.
The sysvinit script for supervisor has a long-standing bug where
`/etc/init.d/supervisor restart` stops but does not then start the
supervisor process.

Work around this by making restart then try to start, and return if it
is currently running.
2021-07-22 14:23:41 -07:00
Alex Vandiver
88a2a80d81 ci: Use an init process to reap defunct processes.
When Github Actions run in Docker, the default pid 1 entrypoint is
`tail -f /dev/null`.  PID 1 is responsible for propagating signals to
its children, and calling `waitpid()` on defunct processes; `tail`
does not do these things.  This results in zombie processes piling up
inside the container, which is not an issue in most contexts.

However, it affects `start-stop-daemon`, which hangs when stopping
daemon processes, as they are never reaped.  This appears in CI as
`/etc/init.d/supervisor restart` never being able to succeed.

Run the docker container with `--init`, which spawns a
`/sbin/docker-init` PID 1 to handle the job of an init process.
2021-07-22 14:23:37 -07:00
Erik Tews
5b16ee0c08 auth: show _OR_ during login only when other methods are available.
There might be good reasons to have other external authentication
methods such as SAML configured, but none of them is available.

This happens, for example, when you have enabled SAML so that Zulip is
able to generate the metadata in XML format, but you haven't
configured an IdP yet. This commit makes sure that the phrase _OR_ is
only shown on the login/account page when there are actually other
authentication methods available. When they are just configured, but
not available yet, the page looks like as if no external
authentication methods are be configured.

We achieve this by deleting any_social_backend_enabled, which was very
similar to page_params.external_authentication_methods, which
correctly has one entry per configured SAML IdP.
2021-07-20 14:31:54 -07:00
Tim Abbott
17dced26ff i18n: Update translation data from Transifex. 2021-07-15 09:44:04 -07:00
Alex Vandiver
fc9c5b1f43 puppet: Ensure psycopg2 is installed before running process_fts_updates.
Not having the package installed will cause startup failures in
`process_fts_updates`; ensure that we've installed the package before
we potentially start the service.
2021-07-15 00:25:39 +00:00
Alex Vandiver
564873a207 smokescreen: Default to only listening on 127.0.0.1.
This prevents Smokescreen from acting as an open proxy.

Fixes #19214.
2021-07-14 15:41:33 -07:00
Mateusz Mandera
c692263255 management: Add change_password command.
Zulip identifies users by realm+delivery_email which means that the
Django changepassword command doesn't work well -
since it looks only at the .email field.
Thus we fork its code to our own change_password command.
2021-07-09 12:34:56 -07:00
Mateusz Mandera
bfe428f608 saml: Add setting to skip the "continue to registration" page.
It's a smoother Just-In-Time provisioning process to allow
creating the account and getting signed in on the first login by the
user.
2021-07-08 15:21:40 -07:00
Mateusz Mandera
d200e3547f embed_links: Interrupt consume() function on worker timeout.
This fixes a bug introduced in 95b46549e1
which made the worker simply log a warning about the timeout and then
continue consume()ing the event that should have also been interrupted.

The idea here is to introduce an exception which can be used to
interrupt the consume() process without triggering the regular handling
of exceptions that happens in _handle_consume_exception.
2021-07-07 09:25:13 -07:00
Tim Abbott
b6afa4a82b test_queue_worker: Fix order-dependent assertions. 2021-07-06 14:37:28 -07:00
Mateusz Mandera
4db187856d embed_links: Only log warning if worker times out.
Throwing an exception is excessive in case of this worker, as it's
expected for it to time out sometimes if the urls take too long to
process.

With a test added by tabbott.
2021-07-06 14:18:08 -07:00
Mateusz Mandera
36638c95b9 queue_processors: Make timer_expired receive list of events as argument.
This will give queue workers more flexibility when defining their own
override of the method.
2021-07-06 14:18:04 -07:00
Mateusz Mandera
85f14eb4f7 queue_processors: Make timer_expired() a method.
This allows specific queue workers to override the defaut behavior and
implement their own response to the timer expiring. We will want to use
this for embed_links queue at least.
2021-07-06 14:18:01 -07:00
Steve Howell
0fab79c027 widgets: Add range checks on backend for indexes. 2021-07-01 15:15:11 -07:00
Steve Howell
7d46bed507 widgets: Validate todo data on the backend. 2021-07-01 15:15:11 -07:00
Alex Vandiver
a89ba9c7d6 puppet: Catch when a comma is left out of puppet_classes.
With two space-separated classes in `puppet_classes`, the second one
is silently ignored.  With three of more, puppet generates the
following very opaque error message:

```
Error: Could not parse for environment production: This
Name has no effect. A value was produced and then forgotten (one or
more preceding expressions may have the wrong form)
```

Catch when this has happened, and give an error message to the user.

Fixes #18992.
2021-06-28 17:59:46 -07:00
Tim Abbott
8f735f4683 install: Use a period at end of root error message. 2021-06-23 09:10:12 -07:00
Gaurav Pandey
e7cfd30d53 upgrade: Modify upgrade scripts to handle failure.
The current `upgrade-zulip` and `upgrade-zulip-from-git`
bash scripts exit with a zero status even if the
upgrade commands exit with a non-zero status.
Hence add `set -e` command which exits the script with
the same status as the non-zero command.

For pipe commands however, the net status of a command
is the status of the last command, hence if the other parts
fail, the net status is only determined by the last command.
This is the case with our main /lib/upgrade-zulip* command
in the scripts whose status is determined by the `tee` command
instead. Hence add a small condition to get the status of the
actual upgrade command and exit the script if it fails with
a non-zero command.

We also check whether the script is being run as root, matching the
install script logic.
2021-06-23 09:10:11 -07:00
Mateusz Mandera
10c8c0e071 upload: Use URL manipulation for get_public_upload_url logic.
This is much faster than calling generate_presigned_url each time.

```
In [3]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.0010945796966552734
```
2021-06-22 09:36:29 -07:00
Mateusz Mandera
9f8b5e225d upload: Cache the boto client to improve performance.
Fixes #18915

This was very slow, causing performance issues. After investigating,
generate_presigned_url is the cheap part of this, but the
session.client() call is expensive - so that's what we should cache.

Before the change:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
6.408717393875122
```

After:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.48990607261657715
```

This is not good enough to avoid doing something ugly like replacing
generate_presigned_url with some manual URL manipulation, but it's a
helpful structure that we may find useful with further refactoring.
2021-06-22 09:36:28 -07:00
Steve Howell
62194eb20f poll widget: Add server validation. 2021-06-14 17:57:24 -07:00
Steve Howell
2492f4b60e submessages: Add verify_submessage_sender.
Before this change a rogue actor could try to
widgetize another person's message. (The
rogue actor would already have access to read
the message.)
2021-06-14 17:57:23 -07:00
Signior-X
1b2967ddb5 login: Remove browser show password in IE, edge.
The Microsoft browsers such as IE and Edge has their own
show password that is a bit bugy and also conflicts with
the show password in Zulip that was added in #17305.
This fixes the issue by making the display none for the
ms-reveal that comes in the input.

More details can be found at
https://chat.zulip.org/#narrow/stream/101-design/topic/Show.20password/near/1173890
2021-06-14 16:36:15 -07:00
Tim Abbott
42774b101f webhooks: Update link to BuildBot documentation. 2021-06-10 17:16:09 -07:00
Anders Kaseorg
716cba04de zulip_tools: Flush ‘set -x’-style messages in run.
Otherwise they often get buffered until after the command actually
runs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit d8cb418586)
2021-06-09 16:16:42 -07:00
Tim Abbott
332add3bb6 import: Fix propagation of subdomain error messages.
The previous logic would provide a very confusing error message if the
subdomain was already in use.
2021-06-09 13:22:23 -07:00
Anders Kaseorg
b596cd7607 webpack: Fix CSS source map generation on 1-CPU systems.
We were passing a SourceMapGenerator as `map`, but it seems that
css-minimizer-webpack-plugin expects a string, and only implicitly
stringifies it when running with parallelism.

Fixes #18727.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit aedc5af351)
2021-06-08 16:26:58 -07:00
Alex Vandiver
21cedabbdf subdomains: Extend "static" to include resources hosted on S3.
This causes avatars and emoji which are hosted by Zulip in S3 (or
compatible) servers to no longer go through camo.  Routing these
requests through camo does not add any privacy benefit (as the request
logs there go to the Zulip admins regardless), and may break emoji
imported from Slack before 1bf385e35f,
which have `application/octet-stream` as their stored Content-Type.
2021-06-08 15:28:32 -07:00
Alex Vandiver
f910d5b8a9 docs: Remove link to 16.04, which can be confusing.
The instructions do not just apply to 16.04; the block below describes
the settings, which are correct for all relevant Ubuntu versions.
2021-06-02 17:18:41 -07:00
Alex Vandiver
daf185705d send_test_email: Capture and show SMTP log on errors. 2021-06-02 13:18:26 -07:00
Tim Abbott
1fa7081a4c version: Update version after 4.3 release. 2021-06-02 12:54:04 -07:00
Tim Abbott
0d17a5e76d Release Zulip Server 4.3. 2021-06-02 11:40:33 -07:00
Tim Abbott
9815581957 i18n: Update translation data from Transifex. 2021-06-02 09:48:12 -07:00
Tim Abbott
33d7aa9d47 i18n: Adjust Transifex sync-translations download mode.
It appears that some server-side change to Transifex resulted in the
"onlytranslated" mode deleting some (all?) strings from django.po files that
were not translated.

Testing determined that the "translator" mode appears to now be the
only mode that works with both our django.po and translations.json
files (We want to avoid both copying the English strings and deleting
strings), so we're switching to that.

Background is available here:
https://chat.zulip.org/#narrow/stream/3-backend/topic/4.2Ex.20branch.20translations.20sync/near/1187324
2021-06-02 09:44:40 -07:00
Alex Vandiver
6c3a6ef6c1 docs: Add a missing close paren. 2021-06-01 16:33:10 -07:00
Alex Vandiver
a63150ca35 docs: Update path to nginx.conf, as it is now a template.
Also provide the right expansion for the one embedded variable
currently in the template.
2021-06-01 16:33:06 -07:00
Anders Kaseorg
7ab8455596 giphy: Load Giphy SDK lazily.
The Giphy SDK sends tracking pings when it loads; we don’t want those
to be sent for visitors who aren’t using Giphy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-28 15:45:07 -07:00
Tim Abbott
43be62c7ef upload: Use get_public_upload_url for export tarballs too.
This deduplicates the code so that we now just have one function for
constructing S3 URLs.
2021-05-27 23:30:00 -07:00
ryanreh99
7b15ce71c2 s3 uploads: Refactor to access objects via get_public_upload_url.
Our current logic only allows S3 block storage providers whose
upload URL matches with the format used by AWS. This also allows
other styles such as the "virtual host" format used by Oracle cloud.

Fixes #17762.
2021-05-27 23:29:59 -07:00
Sumanth V Rao
96c5a9e303 models: Fix bug in unique_together condition on RealmPlayground.
We don't need to worry about breaking already configured playgrounds
since this tweak makes the condition less strict.
2021-05-26 18:17:24 -07:00
Anders Kaseorg
0b337e0819 actions: Fix incorrect audit logging in bulk_remove_subscriptions.
modified_user=sub_info.user and modified_stream=sub_info.stream, added
by commit 6d1f9de7d3 (#16553), were
always coming from the last entry in the loop above, not from the
enclosing list comprehension.

Found by the Pylint rule undefined-loop-variable.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 18:17:08 -07:00
Anders Kaseorg
d4b3c20e48 markdown: Fix Dropbox image previews.
?dl=1 causes Dropbox to send Content-Type: application/binary, which
can’t be interpreted by Camo.  Use ?raw=1 instead.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 12:17:48 -07:00
Vishnu KS
31be0f04b9 i18n: Tag strings in status message settings for translation.
Fixes #18609
2021-05-26 11:04:25 -07:00
Vishnu KS
6af0e28e5d user status: Remove data attributes from user status options.
I don't see any good reason why we have to store the status
values in data attributes when they are already stored as
the content of the buttons.
2021-05-26 11:04:24 -07:00
Adam Birds
9cb538b08f integrations: Add label_create_activity to unsupported pivotal events.
Fixes #18580.
2021-05-25 20:57:17 -07:00
AdamVB
bf49f962c0 integrations: Enhance Grafana integration with alert state.
Having the alert state in the message body is useful when alert topics 
are not defined by alert description but encoded in the url.

E.g. in large environments having a topic for each alert [alerting] and [ok] would 
make it harder to properly track if an alert has been resolved.

When each alert is in a single topic, so far, the alert state has been missing.

This change will add the current alert state and a fitting icon in front
of the alert name.(Similar to the prometheus alertmanager integration)

The test cases have been amended to cover all possible alert states, even
though realistically grafana only fires the ok and alerting states via
webhook.
2021-05-24 14:25:47 -07:00
Alex Vandiver
2a69b4f3b7 update-prod-static: Ensure that it is run as the zulip user. 2021-05-21 16:53:02 -07:00
sahil839
540904aa9d giphy: Add a '?' icon besides the "GIPHY integration" label.
We add a '?' icon besides the "GIPHY integration" label of
giphy settings dropdown.

The icon links to readthedocs page for setting up giphy API
key when api key is not set, and it points to help center
article of GIFs when the api key is added.
2021-05-19 13:21:41 -07:00
sahil839
26bdf79642 css: Change width of upgrade-tip and to max-content.
We change the width of upgrade-tip to be max-content
such that it matches with the other elements in
settings overlay like dropdown, which are not of full
width.
2021-05-19 13:21:23 -07:00
sahil839
2c1ffaceca giphy: Fix live update of giphy icon when API key is empty.
We fix the code to show giphy icon live update only if the
updated setting is not disabled and API key has been added.
Though the dropdown is disabled,the setting can still be
changed using API, so this change is necessary.

Previously, we were not checking whether API key is there or
not and icon was shown on live update even if API key was
not there and then it went off on reload.
2021-05-19 13:21:19 -07:00
sahil839
dffff73654 giphy: Disable giphy settings dropdown if API key is not present. 2021-05-19 13:21:15 -07:00
Tim Abbott
2f9d4f5a96 settings: Fix setting JITSI_SERVER_URL to None.
This fixes a bug introduced in
55a23754c3, that resulted in Zulip
crashing on startup if JITSI_SERVER_URL=None.

Fixes #18512.
2021-05-18 19:17:13 -07:00
Tim Abbott
ce96018af4 version: Update version after 4.2 release. 2021-05-13 22:08:45 -07:00
Tim Abbott
a025fab082 Release Zulip Server 4.2. 2021-05-13 22:03:34 -07:00
Anders Kaseorg
812ad52007 install: Run git config commands from a known readable cwd.
Fixes this error when running the installer from a directory that
isn’t world-readable:

+ su zulip -c 'git config --global user.email anders@zulip.com'
fatal: cannot come back to cwd: Permission denied

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:01:01 -07:00
Anders Kaseorg
9066fcac9a postgresql-init-db: Fix installation from world-unreadable directory.
This reverts part of commit 476524c0c1
(#18215), to fix this error when running the installer from a
directory that isn’t world-readable:

+ '[' -e /var/run/supervisor.sock ']'
+++ dirname /root/zulip-server-4.1/scripts/setup/postgresql-init-db
++ dirname /root/zulip-server-4.1/scripts/setup
+ su zulip -c /root/zulip-server-4.1/scripts/stop-server
bash: /root/zulip-server-4.1/scripts/stop-server: Permission denied

Zulip installation failed (exit code 126)!

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:56 -07:00
Anders Kaseorg
a70ebdb005 purge-old-deployments: Check /srv/zulip.git existence before pruning it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 20:56:47 -07:00
Tim Abbott
956d4b2568 version: Link blog post from 4.0 release. 2021-05-13 18:59:53 -07:00
Tim Abbott
ea2256da29 version: Update version after 4.1 release. 2021-05-13 18:58:51 -07:00
Tim Abbott
d1bd8f3637 Release Zulip Server 4.1. 2021-05-13 18:35:06 -07:00
Tim Abbott
22d486bbf7 scripts: Fix check for services running when upgrading.
When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.
2021-05-13 18:10:08 -07:00
Aman Agrawal
977ff62fe8 message_edit_form: Fix vertical alignment of bottom elements. 2021-05-13 17:19:22 -07:00
Anders Kaseorg
5bfc162df9 changelog: Fix version number typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 17:19:12 -07:00
Tim Abbott
2aa643502a version: Update version after 4.0 release. 2021-05-13 15:53:02 -07:00
328 changed files with 23220 additions and 12613 deletions

View File

@@ -17,7 +17,7 @@ max_line_length = 100
[*.{py,pyi}]
max_line_length = 110
[*.{svg,rb,pp,yaml,yml}]
[*.{md,svg,rb,pp,yaml,yml}]
indent_size = 2
[package.json]

View File

@@ -1,14 +1,11 @@
<!-- What's this PR for? (Just a link to an issue is fine.) -->
**Testing plan:** <!-- How have you tested? -->
**GIFs or screenshots:** <!-- If a UI change. See:
https://zulip.readthedocs.io/en/latest/tutorials/screenshot-and-gif-software.html
-->
<!-- Also be sure to make clear, coherent commits:
https://zulip.readthedocs.io/en/latest/contributing/version-control.html
-->

View File

@@ -22,6 +22,7 @@ jobs:
# so this is required.
- name: Get workflow IDs.
id: workflow_ids
continue-on-error: true # Don't fail this job on failure
env:
# This is in <owner>/<repo> format e.g. zulip/zulip
REPOSITORY: ${{ github.repository }}
@@ -35,7 +36,8 @@ jobs:
ids=$(node -e "$script")
echo "::set-output name=ids::$ids"
- uses: styfle/cancel-workflow-action@0.4.1
- uses: styfle/cancel-workflow-action@0.9.0
continue-on-error: true # Don't fail this job on failure
with:
workflow_id: ${{ steps.workflow_ids.outputs.ids }}
access_token: ${{ github.token }}

View File

@@ -4,6 +4,7 @@ on: [push, pull_request]
jobs:
CodeQL:
if: ${{!github.event.repository.private}}
runs-on: ubuntu-latest
steps:

View File

@@ -1,24 +0,0 @@
name: Legacy OS
on: [push, pull_request]
jobs:
xenial:
name: Ubuntu 16.04 Xenial (Python 3.5, legacy)
runs-on: ubuntu-16.04
steps:
- uses: actions/checkout@v2
- name: Check tools/provision error message on xenial
run: |
{ { ! tools/provision 2>&1 >&3; } | tee provision.err; } 3>&1 >&2
grep -Fqx 'Error: ubuntu 16.04 is no longer a supported platform for Zulip.' provision.err
- name: Check scripts/lib/upgrade-zulip-stage-2 error message on xenial
run: |
{ { ! sudo scripts/lib/upgrade-zulip-stage-2 2>&1 >&3; } | tee upgrade.err; } 3>&1 >&2
grep -Fq 'upgrade-zulip-stage-2: Unsupported platform: ubuntu 16.04' upgrade.err
- name: Report status
if: failure()
env:
ZULIP_BOT_KEY: ${{ secrets.ZULIP_BOT_KEY }}
run: tools/ci/send-failure-message

View File

@@ -30,6 +30,8 @@ defaults:
jobs:
production_build:
# This job builds a release tarball from the current commit, which
# will be used for all of the following install/upgrade tests.
name: Bionic production build
runs-on: ubuntu-latest
@@ -106,6 +108,9 @@ jobs:
run: tools/ci/send-failure-message
production_install:
# This job installs the server release tarball built above on a
# range of platforms, and does some basic health checks on the
# resulting installer Zulip server.
strategy:
fail-fast: false
matrix:
@@ -133,7 +138,9 @@ jobs:
os: bullseye
name: ${{ matrix.name }}
container: ${{ matrix.docker_image }}
container:
image: ${{ matrix.docker_image }}
options: --init
runs-on: ubuntu-latest
needs: production_build
@@ -206,3 +213,63 @@ jobs:
env:
ZULIP_BOT_KEY: ${{ secrets.ZULIP_BOT_KEY }}
run: /tmp/send-failure-message
production_upgrade:
# The production upgrade job starts with a container with a
# previous Zulip release installed, and attempts to upgrade it to
# the release tarball built for the current commit being tested.
#
# This is intended to catch bugs that result in the upgrade
# process failing.
strategy:
fail-fast: false
matrix:
include:
# Base images are built using `tools/ci/Dockerfile.prod.template`.
# The comments at the top explain how to build and upload these images.
- docker_image: zulip/ci:buster-3.4
name: 3.4 Version Upgrade
is_focal: true
os: buster
name: ${{ matrix.name }}
container:
image: ${{ matrix.docker_image }}
options: --init
runs-on: ubuntu-latest
needs: production_build
steps:
- name: Download built production tarball
uses: actions/download-artifact@v2
with:
name: production-tarball
path: /tmp
- name: Add required permissions and setup
run: |
# This is the GitHub Actions specific cache directory the
# the current github user must be able to access for the
# cache action to work. It is owned by root currently.
sudo chmod -R 0777 /__w/_temp/
# Since actions/download-artifact@v2 loses all the permissions
# of the tarball uploaded by the upload artifact fix those.
chmod +x /tmp/production-upgrade
chmod +x /tmp/production-verify
chmod +x /tmp/send-failure-message
- name: Upgrade production
run: sudo /tmp/production-upgrade
# TODO: We should be running production-verify here, but it
# doesn't pass yet.
#
# - name: Verify install
# run: sudo /tmp/production-verify
- name: Report status
if: failure()
env:
ZULIP_BOT_KEY: ${{ secrets.ZULIP_BOT_KEY }}
run: /tmp/send-failure-message

View File

@@ -1,6 +1,8 @@
/corporate/tests/stripe_fixtures
/locale
/static/third
/templates/**/*.md
/tools/setup/emoji/emoji_map.json
/zerver/tests/fixtures
/zerver/webhooks/*/doc.md
/zerver/webhooks/*/fixtures

View File

@@ -18,15 +18,15 @@ all of us and the technical communities in which we participate.
The following behaviors are expected and requested of all community members:
* Participate. In doing so, you contribute to the health and longevity of
- Participate. In doing so, you contribute to the health and longevity of
the community.
* Exercise consideration and respect in your speech and actions.
* Attempt collaboration before conflict. Assume good faith.
* Refrain from demeaning, discriminatory, or harassing behavior and speech.
* Take action or alert community leaders if you notice a dangerous
- Exercise consideration and respect in your speech and actions.
- Attempt collaboration before conflict. Assume good faith.
- Refrain from demeaning, discriminatory, or harassing behavior and speech.
- Take action or alert community leaders if you notice a dangerous
situation, someone in distress, or violations of this code, even if they
seem inconsequential.
* Community event venues may be shared with members of the public; be
- Community event venues may be shared with members of the public; be
respectful to all patrons of these locations.
## Unacceptable behavior
@@ -34,24 +34,24 @@ The following behaviors are expected and requested of all community members:
The following behaviors are considered harassment and are unacceptable
within the Zulip community:
* Jokes or derogatory language that singles out members of any race,
- Jokes or derogatory language that singles out members of any race,
ethnicity, culture, national origin, color, immigration status, social and
economic class, educational level, language proficiency, sex, sexual
orientation, gender identity and expression, age, size, family status,
political belief, religion, and mental and physical ability.
* Violence, threats of violence, or violent language directed against
- Violence, threats of violence, or violent language directed against
another person.
* Disseminating or threatening to disseminate another person's personal
- Disseminating or threatening to disseminate another person's personal
information.
* Personal insults of any sort.
* Posting or displaying sexually explicit or violent material.
* Inappropriate photography or recording.
* Deliberate intimidation, stalking, or following (online or in person).
* Unwelcome sexual attention. This includes sexualized comments or jokes,
- Personal insults of any sort.
- Posting or displaying sexually explicit or violent material.
- Inappropriate photography or recording.
- Deliberate intimidation, stalking, or following (online or in person).
- Unwelcome sexual attention. This includes sexualized comments or jokes,
inappropriate touching or groping, and unwelcomed sexual advances.
* Sustained disruption of community events, including talks and
- Sustained disruption of community events, including talks and
presentations.
* Advocating for, or encouraging, any of the behaviors above.
- Advocating for, or encouraging, any of the behaviors above.
## Reporting and enforcement

View File

@@ -26,29 +26,30 @@ To make a code or documentation contribution, read our
[step-by-step guide](#your-first-codebase-contribution) to getting
started with the Zulip codebase. A small sample of the type of work that
needs doing:
* Bug squashing and feature development on our Python/Django
- Bug squashing and feature development on our Python/Django
[backend](https://github.com/zulip/zulip), web
[frontend](https://github.com/zulip/zulip), React Native
[mobile app](https://github.com/zulip/zulip-mobile), or Electron
[desktop app](https://github.com/zulip/zulip-desktop).
* Building out our
- Building out our
[Python API and bots](https://github.com/zulip/python-zulip-api) framework.
* [Writing an integration](https://zulip.com/api/integrations-overview).
* Improving our [user](https://zulip.com/help/) or
- [Writing an integration](https://zulip.com/api/integrations-overview).
- Improving our [user](https://zulip.com/help/) or
[developer](https://zulip.readthedocs.io/en/latest/) documentation.
* [Reviewing code](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html)
- [Reviewing code](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html)
and manually testing pull requests.
**Non-code contributions**: Some of the most valuable ways to contribute
don't require touching the codebase at all. We list a few of them below:
* [Reporting issues](#reporting-issues), including both feature requests and
- [Reporting issues](#reporting-issues), including both feature requests and
bug reports.
* [Giving feedback](#user-feedback) if you are evaluating or using Zulip.
* [Sponsor Zulip](https://github.com/sponsors/zulip) through the GitHub sponsors program.
* [Translating](https://zulip.readthedocs.io/en/latest/translating/translating.html)
- [Giving feedback](#user-feedback) if you are evaluating or using Zulip.
- [Sponsor Zulip](https://github.com/sponsors/zulip) through the GitHub sponsors program.
- [Translating](https://zulip.readthedocs.io/en/latest/translating/translating.html)
Zulip.
* [Outreach](#zulip-outreach): Star us on GitHub, upvote us
- [Outreach](#zulip-outreach): Star us on GitHub, upvote us
on product comparison sites, or write for [the Zulip blog](https://blog.zulip.org/).
## Your first (codebase) contribution
@@ -57,7 +58,8 @@ This section has a step by step guide to starting as a Zulip codebase
contributor. It's long, but don't worry about doing all the steps perfectly;
no one gets it right the first time, and there are a lot of people available
to help.
* First, make an account on the
- First, make an account on the
[Zulip community server](https://zulip.readthedocs.io/en/latest/contributing/chat-zulip-org.html),
paying special attention to the community norms. If you'd like, introduce
yourself in
@@ -65,12 +67,12 @@ to help.
your name as the topic. Bonus: tell us about your first impressions of
Zulip, and anything that felt confusing/broken as you started using the
product.
* Read [What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor).
* [Install the development environment](https://zulip.readthedocs.io/en/latest/development/overview.html),
- Read [What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor).
- [Install the development environment](https://zulip.readthedocs.io/en/latest/development/overview.html),
getting help in
[#development help](https://chat.zulip.org/#narrow/stream/49-development-help)
if you run into any troubles.
* Read the
- Read the
[Zulip guide to Git](https://zulip.readthedocs.io/en/latest/git/index.html)
and do the Git tutorial (coming soon) if you are unfamiliar with
Git, getting help in
@@ -84,7 +86,7 @@ Now, you're ready to pick your first issue! There are hundreds of open issues
in the main codebase alone. This section will help you find an issue to work
on.
* If you're interested in
- If you're interested in
[mobile](https://github.com/zulip/zulip-mobile/issues?q=is%3Aopen+is%3Aissue),
[desktop](https://github.com/zulip/zulip-desktop/issues?q=is%3Aopen+is%3Aissue),
or
@@ -93,18 +95,18 @@ on.
[#mobile](https://chat.zulip.org/#narrow/stream/48-mobile),
[#desktop](https://chat.zulip.org/#narrow/stream/16-desktop), or
[#integration](https://chat.zulip.org/#narrow/stream/127-integrations).
* For the main server and web repository, we recommend browsing
- For the main server and web repository, we recommend browsing
recently opened issues to look for issues you are confident you can
fix correctly in a way that clearly communicates why your changes
are the correct fix. Our GitHub workflow bot, zulipbot, limits
users who have 0 commits merged to claiming a single issue labeled
with "good first issue" or "help wanted".
* We also partition all of our issues in the main repo into areas like
- We also partition all of our issues in the main repo into areas like
admin, compose, emoji, hotkeys, i18n, onboarding, search, etc. Look
through our [list of labels](https://github.com/zulip/zulip/labels), and
click on some of the `area:` labels to see all the issues related to your
areas of interest.
* If the lists of issues are overwhelming, post in
- If the lists of issues are overwhelming, post in
[#new members](https://chat.zulip.org/#narrow/stream/95-new-members) with a
bit about your background and interests, and we'll help you out. The most
important thing to say is whether you're looking for a backend (Python),
@@ -119,10 +121,11 @@ have a new feature you'd like to add, we recommend you start by posting in
feature idea and the problem that you're hoping to solve.
Other notes:
* For a first pull request, it's better to aim for a smaller contribution
- For a first pull request, it's better to aim for a smaller contribution
than a bigger one. Many first contributions have fewer than 10 lines of
changes (not counting changes to tests).
* The full list of issues explicitly looking for a contributor can be
- The full list of issues explicitly looking for a contributor can be
found with the
[good first issue](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)
and
@@ -133,7 +136,7 @@ Other notes:
are fair game if Tim has written a clear technical design proposal
in the issue, or it is a bug that you can reproduce and you are
confident you can fix the issue correctly.
* For most new contributors, there's a lot to learn while making your first
- For most new contributors, there's a lot to learn while making your first
pull request. It's OK if it takes you a while; that's normal! You'll be
able to work a lot faster as you build experience.
@@ -144,20 +147,20 @@ the issue thread. [Zulipbot](https://github.com/zulip/zulipbot) is a GitHub
workflow bot; it will assign you to the issue and label the issue as "in
progress". Some additional notes:
* You can only claim issues with the
- You can only claim issues with the
[good first issue](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)
or
[help wanted](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)
labels. Zulipbot will give you an error if you try to claim an issue
without one of those labels.
* You're encouraged to ask questions on how to best implement or debug your
- You're encouraged to ask questions on how to best implement or debug your
changes -- the Zulip maintainers are excited to answer questions to help
you stay unblocked and working efficiently. You can ask questions on
chat.zulip.org, or on the GitHub issue or pull request.
* We encourage early pull requests for work in progress. Prefix the title of
- We encourage early pull requests for work in progress. Prefix the title of
work in progress pull requests with `[WIP]`, and remove the prefix when
you think it might be mergeable and want it to be reviewed.
* After updating a PR, add a comment to the GitHub thread mentioning that it
- After updating a PR, add a comment to the GitHub thread mentioning that it
is ready for another review. GitHub only notifies maintainers of the
changes when you post a comment, so if you don't, your PR will likely be
neglected by accident!
@@ -175,23 +178,23 @@ labels.
Zulip has a lot of experience working with new contributors. In our
experience, these are the best predictors of success:
* Posting good questions. This generally means explaining your current
- Posting good questions. This generally means explaining your current
understanding, saying what you've done or tried so far, and including
tracebacks or other error messages if appropriate.
* Learning and practicing
- Learning and practicing
[Git commit discipline](https://zulip.readthedocs.io/en/latest/contributing/version-control.html#commit-discipline).
* Submitting carefully tested code. This generally means checking your work
- Submitting carefully tested code. This generally means checking your work
through a combination of automated tests and manually clicking around the
UI trying to find bugs in your work. See
[things to look for](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html#things-to-look-for)
for additional ideas.
* Posting
- Posting
[screenshots or GIFs](https://zulip.readthedocs.io/en/latest/tutorials/screenshot-and-gif-software.html)
for frontend changes.
* Being responsive to feedback on pull requests. This means incorporating or
- Being responsive to feedback on pull requests. This means incorporating or
responding to all suggested changes, and leaving a note if you won't be
able to address things within a few days.
* Being helpful and friendly on chat.zulip.org.
- Being helpful and friendly on chat.zulip.org.
These are also the main criteria we use to select candidates for all
of our outreach programs.
@@ -227,17 +230,17 @@ hear about your experience with the product. If you're not sure what to
write, here are some questions we're always very curious to know the answer
to:
* Evaluation: What is the process by which your organization chose or will
- Evaluation: What is the process by which your organization chose or will
choose a group chat product?
* Pros and cons: What are the pros and cons of Zulip for your organization,
- Pros and cons: What are the pros and cons of Zulip for your organization,
and the pros and cons of other products you are evaluating?
* Features: What are the features that are most important for your
- Features: What are the features that are most important for your
organization? In the best-case scenario, what would your chat solution do
for you?
* Onboarding: If you remember it, what was your impression during your first
- Onboarding: If you remember it, what was your impression during your first
few minutes of using Zulip? What did you notice, and how did you feel? Was
there anything that stood out to you as confusing, or broken, or great?
* Organization: What does your organization do? How big is the organization?
- Organization: What does your organization do? How big is the organization?
A link to your organization's website?
## Outreach programs
@@ -252,15 +255,16 @@ summer interns from Harvard, MIT, and Stanford.
While each third-party program has its own rules and requirements, the
Zulip community's approaches all of these programs with these ideas in
mind:
* We try to make the application process as valuable for the applicant as
- We try to make the application process as valuable for the applicant as
possible. Expect high-quality code reviews, a supportive community, and
publicly viewable patches you can link to from your resume, regardless of
whether you are selected.
* To apply, you'll have to submit at least one pull request to a Zulip
- To apply, you'll have to submit at least one pull request to a Zulip
repository. Most students accepted to one of our programs have
several merged pull requests (including at least one larger PR) by
the time of the application deadline.
* The main criteria we use is quality of your best contributions, and
- The main criteria we use is quality of your best contributions, and
the bullets listed at
[What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor).
Because we focus on evaluating your best work, it doesn't hurt your
@@ -307,23 +311,24 @@ for ZSoC, we'll contact you when the GSoC results are announced.
perception of projects like Zulip. We've collected a few sites below
where we know Zulip has been discussed. Doing everything in the following
list typically takes about 15 minutes.
* Star us on GitHub. There are four main repositories:
- Star us on GitHub. There are four main repositories:
[server/web](https://github.com/zulip/zulip),
[mobile](https://github.com/zulip/zulip-mobile),
[desktop](https://github.com/zulip/zulip-desktop), and
[Python API](https://github.com/zulip/python-zulip-api).
* [Follow us](https://twitter.com/zulip) on Twitter.
- [Follow us](https://twitter.com/zulip) on Twitter.
For both of the following, you'll need to make an account on the site if you
don't already have one.
* [Like Zulip](https://alternativeto.net/software/zulip-chat-server/) on
- [Like Zulip](https://alternativeto.net/software/zulip-chat-server/) on
AlternativeTo. We recommend upvoting a couple of other products you like
as well, both to give back to their community, and since single-upvote
accounts are generally given less weight. You can also
[upvote Zulip](https://alternativeto.net/software/slack/) on their page
for Slack.
* [Add Zulip to your stack](https://stackshare.io/zulip) on StackShare, star
- [Add Zulip to your stack](https://stackshare.io/zulip) on StackShare, star
it, and upvote the reasons why people like Zulip that you find most
compelling. Again, we recommend adding a few other products that you like
as well.

View File

@@ -8,8 +8,8 @@ allows users to easily process hundreds or thousands of messages a day. With
over 700 contributors merging over 500 commits a month, Zulip is also the
largest and fastest growing open source group chat project.
[![GitHub Actions build status](https://github.com/zulip/zulip/actions/workflows/zulip-ci.yml/badge.svg?branch=master)](https://github.com/zulip/zulip/actions/workflows/zulip-ci.yml?query=branch%3Amaster)
[![coverage status](https://img.shields.io/codecov/c/github/zulip/zulip/master.svg)](https://codecov.io/gh/zulip/zulip/branch/master)
[![GitHub Actions build status](https://github.com/zulip/zulip/actions/workflows/zulip-ci.yml/badge.svg)](https://github.com/zulip/zulip/actions/workflows/zulip-ci.yml?query=branch%3Amain)
[![coverage status](https://img.shields.io/codecov/c/github/zulip/zulip/main.svg)](https://codecov.io/gh/zulip/zulip)
[![Mypy coverage](https://img.shields.io/badge/mypy-100%25-green.svg)][mypy-coverage]
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg)](https://github.com/prettier/prettier)
@@ -30,13 +30,13 @@ and tell us what's up!
You might be interested in:
* **Contributing code**. Check out our
- **Contributing code**. Check out our
[guide for new contributors](https://zulip.readthedocs.io/en/latest/overview/contributing.html)
to get started. Zulip prides itself on maintaining a clean and
well-tested codebase, and a stock of hundreds of
[beginner-friendly issues][beginner-friendly].
* **Contributing non-code**.
- **Contributing non-code**.
[Report an issue](https://zulip.readthedocs.io/en/latest/overview/contributing.html#reporting-issues),
[translate](https://zulip.readthedocs.io/en/latest/translating/translating.html) Zulip
into your language,
@@ -45,12 +45,12 @@ You might be interested in:
[give us feedback](https://zulip.readthedocs.io/en/latest/overview/contributing.html#user-feedback). We
would love to hear from you, even if you're just trying the product out.
* **Supporting Zulip**. Advocate for your organization to use Zulip, become a [sponsor](https://github.com/sponsors/zulip), write a
- **Supporting Zulip**. Advocate for your organization to use Zulip, become a [sponsor](https://github.com/sponsors/zulip), write a
review in the mobile app stores, or
[upvote Zulip](https://zulip.readthedocs.io/en/latest/overview/contributing.html#zulip-outreach) on
product comparison sites.
* **Checking Zulip out**. The best way to see Zulip in action is to drop by
- **Checking Zulip out**. The best way to see Zulip in action is to drop by
the
[Zulip community server](https://zulip.readthedocs.io/en/latest/contributing/chat-zulip-org.html). We
also recommend reading Zulip for
@@ -58,23 +58,23 @@ You might be interested in:
[companies](https://zulip.com/for/companies/), or Zulip for
[working groups and part time communities](https://zulip.com/for/working-groups-and-communities/).
* **Running a Zulip server**. Use a preconfigured [DigitalOcean droplet](https://marketplace.digitalocean.com/apps/zulip),
- **Running a Zulip server**. Use a preconfigured [DigitalOcean droplet](https://marketplace.digitalocean.com/apps/zulip),
[install Zulip](https://zulip.readthedocs.io/en/stable/production/install.html)
directly, or use Zulip's
experimental [Docker image](https://zulip.readthedocs.io/en/latest/production/deployment.html#zulip-in-docker).
Commercial support is available; see <https://zulip.com/plans> for details.
* **Using Zulip without setting up a server**. <https://zulip.com>
- **Using Zulip without setting up a server**. <https://zulip.com>
offers free and commercial hosting, including providing our paid
plan for free to fellow open source projects.
* **Participating in [outreach
- **Participating in [outreach
programs](https://zulip.readthedocs.io/en/latest/overview/contributing.html#outreach-programs)**
like Google Summer of Code.
You may also be interested in reading our [blog](https://blog.zulip.org/) or
following us on [Twitter](https://twitter.com/zulip).
Zulip is distributed under the
[Apache 2.0](https://github.com/zulip/zulip/blob/master/LICENSE) license.
[Apache 2.0](https://github.com/zulip/zulip/blob/main/LICENSE) license.
[beginner-friendly]: https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22

View File

@@ -12,7 +12,7 @@
# serve to show the default.
import os
import sys
from typing import Any, Dict, List, Optional
from typing import Any, Dict, Optional
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
@@ -20,7 +20,7 @@ from typing import Any, Dict, List, Optional
# sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from version import ZULIP_VERSION
from version import LATEST_RELEASE_VERSION, ZULIP_VERSION
# -- General configuration ------------------------------------------------
@@ -30,7 +30,14 @@ from version import ZULIP_VERSION
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions: List[str] = []
extensions = [
"myst_parser",
]
myst_enable_extensions = [
"colon_fence",
"substitution",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
@@ -99,6 +106,10 @@ pygments_style = "sphinx"
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
myst_substitutions = {
"LATEST_RELEASE_VERSION": LATEST_RELEASE_VERSION,
}
# -- Options for HTML output ----------------------------------------------
@@ -293,8 +304,6 @@ texinfo_documents = [
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
from recommonmark.transform import AutoStructify
# The suffix(es) of source filenames. You can specify multiple suffix
# as a dictionary mapping file extensions to file types
# https://www.sphinx-doc.org/en/master/usage/markdown.html
@@ -303,39 +312,11 @@ source_suffix = {
".md": "markdown",
}
# Temporary workaround to remove multiple build warnings caused by upstream bug.
# See https://github.com/zulip/zulip/issues/13263 for details.
from commonmark.node import Node
from recommonmark.parser import CommonMarkParser
class CustomCommonMarkParser(CommonMarkParser):
def visit_document(self, node: Node) -> None:
pass
suppress_warnings = [
"myst.header",
]
def setup(app: Any) -> None:
app.add_source_parser(CustomCommonMarkParser)
app.add_config_value(
"recommonmark_config",
{
"enable_eval_rst": True,
# Turn off recommonmark features we aren't using.
"enable_auto_doc_ref": False,
"auto_toc_tree_section": None,
"enable_auto_toc_tree": False,
"enable_math": False,
"enable_inline_math": False,
"url_resolver": lambda x: x,
},
True,
)
# Enable `eval_rst`, and any other features enabled in recommonmark_config.
# Docs: https://recommonmark.readthedocs.io/en/latest/auto_structify.html
# (But NB those docs are for master, not latest release.)
app.add_transform(AutoStructify)
# overrides for wide tables in RTD theme
app.add_css_file("theme_overrides.css") # path relative to _static

View File

@@ -6,14 +6,14 @@ In order to accommodate all users, Zulip strives to implement accessibility
best practices in its user interface. There are many aspects to accessibility;
here are some of the more important ones to keep in mind.
* All images should have alternative text attributes for the benefit of users
- All images should have alternative text attributes for the benefit of users
who cannot see them (this includes users who are utilizing a voice interface
to free up their eyes to look at something else instead).
* The entire application should be usable via a keyboard (many users are unable
- The entire application should be usable via a keyboard (many users are unable
to use a mouse, and many accessibility aids emulate a keyboard).
* Text should have good enough contrast against the background to enable
- Text should have good enough contrast against the background to enable
even users with moderate visual impairment to be able to read it.
* [ARIA](https://www.w3.org/WAI/intro/aria) (Accessible Rich Internet
- [ARIA](https://www.w3.org/WAI/intro/aria) (Accessible Rich Internet
Application) attributes should be used appropriately to enable screen
readers and other alternative interfaces to navigate the application
effectively.
@@ -32,19 +32,19 @@ as it is not possible for some content.)
There are tools available to automatically audit a web page for compliance
with many of the WCAG guidelines. Here are some of the more useful ones:
* [Accessibility Developer Tools][chrome-webstore]
- [Accessibility Developer Tools][chrome-webstore]
This open source Chrome extension from Google adds an accessibility audit to
the "Audits" tab of the Chrome Developer Tools. The audit is performed
against the page's DOM via JavaScript, allowing it to identify some issues
that a static HTML inspector would miss.
* [aXe](https://www.deque.com/products/axe/) An open source Chrome and Firefox
- [aXe](https://www.deque.com/products/axe/) An open source Chrome and Firefox
extension which runs a somewhat different set of checks than Google's Chrome
extension.
* [Wave](https://wave.webaim.org/) This web application takes a URL and loads
- [Wave](https://wave.webaim.org/) This web application takes a URL and loads
it in a frame, reporting on all the issues it finds with links to more
information. Has the advantage of not requiring installation, but requires
a URL which can be directly accessed by an external site.
* [Web Developer](https://chrispederick.com/work/web-developer/) This browser
- [Web Developer](https://chrispederick.com/work/web-developer/) This browser
extension has many useful features, including a convenient link for opening
the current URL in Wave to get an accessibility report.
@@ -60,7 +60,7 @@ Problems with Zulip's accessibility should be reported as
label. This label can be added by entering the following text in a separate
comment on the issue:
@zulipbot add "area: accessibility"
> @zulipbot add "area: accessibility"
If you want to help make Zulip more accessible, here is a list of the
[currently open accessibility issues][accessibility-issues].
@@ -70,15 +70,14 @@ If you want to help make Zulip more accessible, here is a list of the
For more information about making Zulip accessible to as many users as
possible, the following resources may be useful.
* [Font Awesome accessibility guide](https://fontawesome.com/how-to-use/on-the-web/other-topics/accessibility),
- [Font Awesome accessibility guide](https://fontawesome.com/how-to-use/on-the-web/other-topics/accessibility),
which is especially helpful since Zulip uses Font Awesome for its icons.
* [Web Content Accessibility Guidelines (WCAG) 2.0](https://www.w3.org/TR/WCAG/)
* [WAI-ARIA](https://www.w3.org/WAI/intro/aria) - Web Accessibility Initiative
- [Web Content Accessibility Guidelines (WCAG) 2.0](https://www.w3.org/TR/WCAG/)
- [WAI-ARIA](https://www.w3.org/WAI/intro/aria) - Web Accessibility Initiative
Accessible Rich Internet Application Suite
* [WebAIM](https://webaim.org/) - Web Accessibility in Mind
* The [MDN page on accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility)
* The [Open edX Accessibility Guidelines][openedx-guidelines] for developers
- [WebAIM](https://webaim.org/) - Web Accessibility in Mind
- The [MDN page on accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility)
- The [Open edX Accessibility Guidelines][openedx-guidelines] for developers
[chrome-webstore]: https://chrome.google.com/webstore/detail/accessibility-developer-t/fpkknkljclfencbdbgkenhalefipecmb
[openedx-guidelines]: https://edx.readthedocs.io/projects/edx-developer-guide/en/latest/conventions/accessibility.html

View File

@@ -2,21 +2,22 @@
Please include these elements in your bug report to make it easier for us to help you.
* A brief title
- A brief title
* An explanation of what you were expecting vs. the actual result
- An explanation of what you were expecting vs. the actual result
* Steps to take in order to reproduce the buggy behavior
- Steps to take in order to reproduce the buggy behavior
* Whether you are using Zulip in production or in the development
- Whether you are using Zulip in production or in the development
environment, and whether these are old versions
* Whether you are using the web app, a desktop app or a mobile device
- Whether you are using the web app, a desktop app or a mobile device
to access Zulip
* Any additional information that would help: screenshots, GIFs, a
- Any additional information that would help: screenshots, GIFs, a
pastebin of the error log
Further reading:
* [How to write a bug report that will make your engineers love you](https://testlio.com/blog/the-ideal-bug-report/)
* [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
- [How to write a bug report that will make your engineers love you](https://testlio.com/blog/the-ideal-bug-report/)
- [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html)

View File

@@ -12,28 +12,28 @@ minutes to a few hours, depending on the time of day.
## Community norms
* Send test messages to
- Send test messages to
[#test here](https://chat.zulip.org/#narrow/stream/7-test-here) or
as a PM to yourself to avoid disturbing others.
* When asking for help, provide the details needed for others to help
- When asking for help, provide the details needed for others to help
you. E.g. include the full traceback in a code block (not a
screenshot), a link to the code or a WIP PR you're having trouble
debugging, etc.
* Ask questions on streams rather than PMing core contributors.
- Ask questions on streams rather than PMing core contributors.
You'll get answers faster since other people can help, and it makes
it possible for other developers to learn from reading the discussion.
* Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's
- Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's
usually easy to see which message you're replying to, so you don't
need to mention your conversation partner in every reply.
Mentioning other users is great for timely questions or making sure
someone who is not online sees your message.
* Converse informally; there's no need to use titles like "Sir" or "Madam".
* Use
- Converse informally; there's no need to use titles like "Sir" or "Madam".
- Use
[gender-neutral language](https://en.wikipedia.org/wiki/Gender-neutral_language).
For example, avoid using a pronoun like her or his in sentences like
"Every developer should clean [their] keyboard at least once a week."
* Follow the community [code of conduct](../code-of-conduct.md).
* Participate! Zulip is a friendly and welcoming community, and we
- Follow the community [code of conduct](../code-of-conduct.md).
- Participate! Zulip is a friendly and welcoming community, and we
love meeting new people, hearing about what brought them to Zulip,
and getting their feedback. If you're not sure where to start,
introduce yourself and your interests in
@@ -54,7 +54,7 @@ streams that are only of occasional interest.
## This is a bleeding edge development server
The chat.zulip.org server is frequently deployed off of `master` from
The chat.zulip.org server is frequently deployed off of `main` from
the Zulip Git repository, so please point out anything you notice that
seems wrong! We catch many bugs that escape code review this way.
@@ -67,19 +67,19 @@ secret/embarrassing, etc.
There are a few streams worth highlighting that are relevant for
everyone, even non-developers:
* [#announce](https://chat.zulip.org/#narrow/stream/1-announce) is for
- [#announce](https://chat.zulip.org/#narrow/stream/1-announce) is for
announcements and discussions thereof; we try to keep traffic there
to a minimum.
* [#feedback](https://chat.zulip.org/#narrow/stream/137-feedback) is for
- [#feedback](https://chat.zulip.org/#narrow/stream/137-feedback) is for
posting feedback on Zulip.
* [#design](https://chat.zulip.org/#narrow/stream/101-design) is where we
- [#design](https://chat.zulip.org/#narrow/stream/101-design) is where we
discuss UI and feature design and collect feedback on potential design
changes. We love feedback, so don't hesitate to speak up!
* [#user community](https://chat.zulip.org/#narrow/stream/138-user-community) is
- [#user community](https://chat.zulip.org/#narrow/stream/138-user-community) is
for Zulip users to discuss their experiences using and adopting Zulip.
* [#production help](https://chat.zulip.org/#narrow/stream/31-production-help)
- [#production help](https://chat.zulip.org/#narrow/stream/31-production-help)
is for production environment related discussions.
* [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) is
- [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) is
for sending test messages without inconveniencing other users :).
We recommend muting this stream when not using it.
@@ -88,25 +88,25 @@ community (e.g. one for each app, etc.); check out the
[Streams page](https://chat.zulip.org/#streams/all) to see the
descriptions for all of them. Relevant to almost everyone are these:
* [#checkins](https://chat.zulip.org/#narrow/stream/65-checkins) is for
- [#checkins](https://chat.zulip.org/#narrow/stream/65-checkins) is for
progress updates on what you're working on and its status; usually
folks post with their name as the topic. Everyone is welcome to
participate!
* [#development help](https://chat.zulip.org/#narrow/stream/49-development-help)
- [#development help](https://chat.zulip.org/#narrow/stream/49-development-help)
is for asking for help with any Zulip server/webapp development work
(use the app streams for help working on one of the apps).
* [#code review](https://chat.zulip.org/#narrow/stream/91-code-review)
- [#code review](https://chat.zulip.org/#narrow/stream/91-code-review)
is for getting feedback on your work. We encourage all developers
to comment on work posted here, even if you're new to the Zulip
project; reviewing other PRs is a great way to develop experience,
and even just manually testing a proposed new feature and posting
feedback is super helpful.
* [#documentation](https://chat.zulip.org/#narrow/stream/19-documentation)
- [#documentation](https://chat.zulip.org/#narrow/stream/19-documentation)
is where we discuss improving Zulip's user, sysadmin, and developer
documentation.
* [#translation](https://chat.zulip.org/#narrow/stream/58-translation) is
- [#translation](https://chat.zulip.org/#narrow/stream/58-translation) is
for discussing Zulip's translations.
* [#learning](https://chat.zulip.org/#narrow/stream/92-learning) is for
- [#learning](https://chat.zulip.org/#narrow/stream/92-learning) is for
posting great learning resources one comes across.
There are also official private streams, including large ones for

View File

@@ -10,8 +10,9 @@ When you send a PR, try to think of a good person to review it --
outside of the handful of people who do a ton of reviews -- and
`@`-mention them with something like "`@person`, would you review
this?". Good choices include
* someone based in your timezone or a nearby timezone
* people working on similar things, or in a loosely related area
- someone based in your timezone or a nearby timezone
- people working on similar things, or in a loosely related area
Alternatively, posting a message in
[#code-review](https://chat.zulip.org/#narrow/stream/91-code-review) on [the Zulip
@@ -28,14 +29,14 @@ to dive right into reviewing the PR's core functionality.
Once you've received a review and resolved any feedback, it's critical
to update the GitHub thread to reflect that. Best practices are to:
* Make sure that CI passes and the PR is rebased onto recent master.
* Post comments on each feedback thread explaining at least how you
- Make sure that CI passes and the PR is rebased onto recent `main`.
- Post comments on each feedback thread explaining at least how you
resolved the feedback, as well as any other useful information
(problems encountered, reasoning for why you picked one of several
options, a test you added to make sure the bug won't recur, etc.).
* Mark any resolved threads as "resolved" in the GitHub UI, if
- Mark any resolved threads as "resolved" in the GitHub UI, if
appropriate.
* Post a summary comment in the main feed for the PR, explaining that
- Post a summary comment in the main feed for the PR, explaining that
this is ready for another review, and summarizing any changes from
the previous version, details on how you tested the changes, new
screenshots/etc. More detail is better than less, as long as you
@@ -60,10 +61,10 @@ Anyone can do a code review -- you don't have to have a ton of
experience, and you don't have to have the power to ultimately merge
the PR. If you
* read the code, see if you understand what the change is
- read the code, see if you understand what the change is
doing and why, and ask questions if you don't; or
* fetch the code (for Zulip server code,
- fetch the code (for Zulip server code,
[tools/fetch-rebase-pull-request][git tool] is super handy), play around
with it in your dev environment, and say what you think about how
the feature works
@@ -106,7 +107,7 @@ sooner is better.
## Things to look for
* *The CI build.* The tests need to pass. One can investigate
- _The CI build._ The tests need to pass. One can investigate
any failures and figure out what to fix by clicking on a red X next
to the commit hash or the Detail links on a pull request. (Example:
in [#17584](https://github.com/zulip/zulip/pull/17584),
@@ -120,25 +121,25 @@ sooner is better.
See our docs on [continuous integration](../testing/continuous-integration.md)
to learn more.
* *Technical design.* There are a lot of considerations here:
- _Technical design._ There are a lot of considerations here:
security, migration paths/backwards compatibility, cost of new
dependencies, interactions with features, speed of performance, API
changes. Security is especially important and worth thinking about
carefully with any changes to security-sensitive code like views.
* *User interface and visual design.* If frontend changes are
- _User interface and visual design._ If frontend changes are
involved, the reviewer will check out the code, play with the new
UI, and verify it for both quality and consistency with the rest of
the Zulip UI. We highly encourage posting screenshots to save
reviewers time in getting a feel for what the feature looks like --
you'll get a quicker response that way.
* *Error handling.* The code should always check for invalid user
- _Error handling._ The code should always check for invalid user
input. User-facing error messages should be clear and when possible
be actionable (it should be obvious to the user what they need to do
in order to correct the problem).
* *Testing.* The tests should validate that the feature works
- _Testing._ The tests should validate that the feature works
correctly, and specifically test for common error conditions, bad
user input, and potential bugs that are likely for the type of
change being made. Tests that exclude whole classes of potential
@@ -147,16 +148,16 @@ sooner is better.
Markdown processors](../subsystems/markdown.md), or the `GetEventsTest` test for
buggy race condition handling).
* *Translation.* Make sure that the strings are marked for
- _Translation._ Make sure that the strings are marked for
[translation].
* *Clear function, argument, variable, and test names.* Every new
- _Clear function, argument, variable, and test names._ Every new
piece of Zulip code will be read many times by other developers, and
future developers will grep for relevant terms when researching a
problem, so it's important that variable names communicate clearly
the purpose of each piece of the codebase.
* *Duplicated code.* Code duplication is a huge source of bugs in
- _Duplicated code._ Code duplication is a huge source of bugs in
large projects and makes the codebase difficult to understand, so we
avoid significant code duplication wherever possible. Sometimes
avoiding code duplication involves some refactoring of existing
@@ -164,32 +165,32 @@ sooner is better.
commits (not squashed into other changes or left as a thing to do
later). That series of commits can be in the same pull request as
the feature that they support, and we recommend ordering the history
of commits so that the refactoring comes *before* the feature. That
of commits so that the refactoring comes _before_ the feature. That
way, it's easy to merge the refactoring (and minimize risk of merge
conflicts) if there are still user experience issues under
discussion for the feature itself.
* *Completeness.* For refactorings, verify that the changes are
- _Completeness._ For refactorings, verify that the changes are
complete. Usually one can check that efficiently using `git grep`,
and it's worth it, as we very frequently find issues by doing so.
* *Documentation updates.* If this changes how something works, does it
- _Documentation updates._ If this changes how something works, does it
update the documentation in a corresponding way? If it's a new
feature, is it documented, and documented in the right place?
* *Good comments.* It's often worth thinking about whether explanation
- _Good comments._ It's often worth thinking about whether explanation
in a commit message or pull request discussion should be included in
a comment, `/docs`, or other documentation. But it's better yet if
verbose explanation isn't needed. We prefer writing code that is
readable without explanation over a heavily commented codebase using
lots of clever tricks.
* *Coding style.* See the Zulip [code-style] documentation for
- _Coding style._ See the Zulip [code-style] documentation for
details. Our goal is to have as much of this as possible verified
via the linters and tests, but there's always going to be unusual
forms of Python/JavaScript style that our tools don't check for.
* *Clear commit messages.* See the [Zulip version
- _Clear commit messages._ See the [Zulip version
control][commit-messages] documentation for details on what we look
for.
@@ -197,15 +198,15 @@ sooner is better.
Some points specific to the Zulip server codebase:
* *Testing -- Backend.* We are trying to maintain ~100% test coverage
- _Testing -- Backend._ We are trying to maintain ~100% test coverage
on the backend, so backend changes should have negative tests for
the various error conditions.
* *Testing -- Frontend.* If the feature involves frontend changes,
- _Testing -- Frontend._ If the feature involves frontend changes,
there should be frontend tests. See the [test
writing][test-writing] documentation for more details.
* *mypy annotations.* New functions should be annotated using [mypy]
- _mypy annotations._ New functions should be annotated using [mypy]
and existing annotations should be updated. Use of `Any`, `ignore`,
and unparameterized containers should be limited to cases where a
more precise type cannot be specified.
@@ -215,7 +216,7 @@ Some points specific to the Zulip server codebase:
To make it easier to review pull requests, if you're working in the
Zulip server codebase, use our [git tool]
`tools/fetch-rebase-pull-request` to check out a pull request locally
and rebase it against master.
and rebase it onto `main`.
If a pull request just needs a little fixing to make it mergeable,
feel free to do that in a new commit, then push your branch to GitHub
@@ -226,14 +227,14 @@ the maintainer time and get the PR merged quicker.
We also strongly recommend reviewers to go through the following resources.
* [The Gentle Art of Patch Review](https://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/)
- [The Gentle Art of Patch Review](https://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/)
article by Sarah Sharp
* [Zulip & Good Code Review](https://www.harihareswara.net/sumana/2016/05/17/0)
- [Zulip & Good Code Review](https://www.harihareswara.net/sumana/2016/05/17/0)
article by Sumana Harihareswara
* [Code Review - A consolidation of advice and stuff from the
sinternet](https://gist.github.com/porterjamesj/002fb27dd70df003646df46f15e898de)
- [Code Review - A consolidation of advice and stuff from the
internet](https://gist.github.com/porterjamesj/002fb27dd70df003646df46f15e898de)
article by James J. Porter
* [Zulip code of conduct](../code-of-conduct.md)
- [Zulip code of conduct](../code-of-conduct.md)
[code-style]: ../contributing/code-style.md
[commit-messages]: ../contributing/version-control.html#commit-messages

View File

@@ -34,11 +34,15 @@ When in doubt, ask in [chat.zulip.org](https://chat.zulip.org).
You can run them all at once with
```bash
./tools/lint
```
You can set this up as a local Git commit hook with
```bash
tools/setup-git-repo
```
The Vagrant setup process runs this for you.
@@ -66,17 +70,21 @@ to read secrets from `/etc/zulip/secrets.conf`.
Look out for Django code like this:
```python
bars = Bar.objects.filter(...)
for bar in bars:
foo = bar.foo
# Make use of foo
```
...because it equates to:
```python
bars = Bar.objects.filter(...)
for bar in bars:
foo = Foo.objects.get(id=bar.foo.id)
# Make use of foo
```
...which makes a database query for every Bar. While this may be fast
locally in development, it may be quite slow in production! Instead,
@@ -84,10 +92,12 @@ tell Django's [QuerySet
API](https://docs.djangoproject.com/en/dev/ref/models/querysets/) to
_prefetch_ the data in the initial query:
```python
bars = Bar.objects.filter(...).select_related()
for bar in bars:
foo = bar.foo # This doesn't take another query, now!
# Make use of foo
```
If you can't rewrite it as a single query, that's a sign that something
is wrong with the database schema. So don't defer this optimization when
@@ -118,7 +128,7 @@ different database queries:
For example, the following will, surprisingly, fail:
```
```python
# Bad example -- will raise!
obj: UserProfile = get_user_profile_by_id(17)
some_objs = UserProfile.objects.get(id=17)
@@ -127,15 +137,15 @@ assert obj in set([some_objs])
You should work with the IDs instead:
```
```python
obj: UserProfile = get_user_profile_by_id(17)
some_objs = UserProfile.objects.get(id=17)
assert obj.id in set([o.id for i in some_objs])
```
### user\_profile.save()
### user_profile.save()
You should always pass the update\_fields keyword argument to .save()
You should always pass the update_fields keyword argument to .save()
when modifying an existing Django model object. By default, .save() will
overwrite every value in the column, which results in lots of race
conditions where unrelated changes made by one thread can be
@@ -145,7 +155,7 @@ object before the first thread wrote out its change.
### Using raw saves to update important model objects
In most cases, we already have a function in zerver/lib/actions.py with
a name like do\_activate\_user that will correctly handle lookups,
a name like do_activate_user that will correctly handle lookups,
caching, and notifying running browsers via the event system about your
change. So please check whether such a function exists before writing
new code to modify a model object, since your new code has a good chance
@@ -158,18 +168,19 @@ cause time-related bugs that are hard to catch with a test suite, or bugs
that only show up during daylight savings time.
Good ways to make timezone-aware datetimes are below. We import timezone
libraries as `from datetime import datetime, timezone` and `from
django.utils.timezone import now as timezone_now`.
libraries as `from datetime import datetime, timezone` and
`from django.utils.timezone import now as timezone_now`.
Use:
* `timezone_now()` to get a datetime when Django is available, such as
- `timezone_now()` to get a datetime when Django is available, such as
in `zerver/`.
* `datetime.now(tz=timezone.utc)` when Django is not available, such as
- `datetime.now(tz=timezone.utc)` when Django is not available, such as
for bots and scripts.
* `datetime.fromtimestamp(timestamp, tz=timezone.utc)` if creating a
- `datetime.fromtimestamp(timestamp, tz=timezone.utc)` if creating a
datetime from a timestamp. This is also available as
`zerver.lib.timestamp.timestamp_to_datetime`.
* `datetime.strptime(date_string, format).replace(tzinfo=timezone.utc)` if
- `datetime.strptime(date_string, format).replace(tzinfo=timezone.utc)` if
creating a datetime from a formatted string that is in UTC.
Idioms that result in timezone-naive datetimes, and should be avoided, are
@@ -179,10 +190,11 @@ parameter, `datetime.utcnow()` and `datetime.utcfromtimestamp()`, and
the end.
Additional notes:
* Especially in scripts and puppet configuration where Django is not
- Especially in scripts and puppet configuration where Django is not
available, using `time.time()` to get timestamps can be cleaner than
dealing with datetimes.
* All datetimes on the backend should be in UTC, unless there is a good
- All datetimes on the backend should be in UTC, unless there is a good
reason to do otherwise.
### `x.attr('zid')` vs. `rows.id(x)`
@@ -231,10 +243,8 @@ generally use modern
[ECMAScript](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Language_Resources)
primitives such as [`for … of`
loops](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of),
[`Array.prototype.{entries, every, filter, find, indexOf, map,
some}`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array),
[`Object.{assign, entries, keys,
values}`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object),
[`Array.prototype.{entries, every, filter, find, indexOf, map, some}`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array),
[`Object.{assign, entries, keys, values}`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object),
[spread
syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax),
and so on. Our Babel configuration automatically transpiles and
@@ -256,8 +266,8 @@ code a lot uglier, in which case it's fine to go up to 120 or so.
Our JavaScript and TypeScript code is formatted with
[Prettier](https://prettier.io/). You can ask Prettier to reformat
all code via our [linter tool](../testing/linters.md) with `tools/lint
--only=prettier --fix`. You can also [integrate it with your
all code via our [linter tool](../testing/linters.md) with
`tools/lint --only=prettier --fix`. You can also [integrate it with your
editor](https://prettier.io/docs/en/editors.html).
Combine adjacent on-ready functions, if they are logically related.
@@ -266,18 +276,24 @@ The best way to build complicated DOM elements is a Mustache template
like `static/templates/message_reactions.hbs`. For simpler things
you can use jQuery DOM building APIs like so:
```js
var new_tr = $('<tr />').attr('id', object.id);
```
Passing a HTML string to jQuery is fine for simple hardcoded things
that don't need internationalization:
```js
foo.append('<p id="selected">/</p>');
```
but avoid programmatically building complicated strings.
We used to favor attaching behaviors in templates like so:
```js
<p onclick="select_zerver({{id}})">
```
but there are some reasons to prefer attaching events using jQuery code:
@@ -312,8 +328,9 @@ type changes in the future.
[Black](https://github.com/psf/black) and
[isort](https://pycqa.github.io/isort/). The [linter
tool](../testing/linters.md) enforces this by running Black and
isort in check mode, or in write mode with `tools/lint
--only=black,isort --fix`. You may find it helpful to [integrate
isort in check mode, or in write mode with
`tools/lint --only=black,isort --fix`. You may find it helpful to
[integrate
Black](https://black.readthedocs.io/en/stable/editor_integration.html)
and
[isort](https://pycqa.github.io/isort/#installing-isorts-for-your-preferred-text-editor)
@@ -328,8 +345,10 @@ type changes in the future.
reason to do otherwise.
- Unpacking sequences doesn't require list brackets:
```python
[x, y] = xs # unnecessary
x, y = xs # better
```
- For string formatting, use `x % (y,)` rather than `x % y`, to avoid
ambiguity if `y` happens to be a tuple.

View File

@@ -100,17 +100,17 @@ make contributions, and make a good proposal.
Your application should include the following:
* Details on any experience you have related to the technologies that
- Details on any experience you have related to the technologies that
Zulip has, or related to our product approach.
* Links to materials to help us evaluate your level of experience and
- Links to materials to help us evaluate your level of experience and
how you work, such as personal projects of yours, including any
existing open source or open culture contributions you've made and
any bug reports you've submitted to open source projects.
* Some notes on what you are hoping to get out of your project.
* A description of the project you'd like to do, and why you're
- Some notes on what you are hoping to get out of your project.
- A description of the project you'd like to do, and why you're
excited about it.
* Some notes on why you're excited about working on Zulip.
* A link to the initial contribution(s) you did.
- Some notes on why you're excited about working on Zulip.
- A link to the initial contribution(s) you did.
We expect applicants to either have experience with the technologies
relevant to their project or have strong general programming
@@ -391,7 +391,7 @@ CSS](https://github.com/zulip/zulip/).
being built into Zulip. And then for built-in bots, one should be able to click a few
buttons of configuration on the web to set them up and include them in
your organization. We've developed a number of example bots
in the (`zulip_bots`](https://github.com/zulip/python-zulip-api/tree/master/zulip_bots)
in the [`zulip_bots`](https://github.com/zulip/python-zulip-api/tree/main/zulip_bots)
PyPI package.
**Skills recommended**: Python and JavaScript/CSS, plus devops
skills (Linux deployment, Docker, Puppet etc.) are all useful here.

View File

@@ -225,7 +225,6 @@ materials](https://developers.google.com/open-source/gsoc/resources/manual).
your contributions to the open source world this summer will be something you
can be proud of for the rest of your life.
## What makes a successful summer
Success for the student means a few things, in order of importance:
@@ -247,7 +246,6 @@ Success for the student means a few things, in order of importance:
student has implemented. That section of code should be more readable,
better-tested, and have clearer documentation.
## Extra notes for mentors
- You're personally accountable for your student having a successful summer. If

View File

@@ -91,12 +91,13 @@ First, check out
of commits with good commit messages.
The first line of the commit message is the **summary**. The summary:
* is written in the imperative (e.g., "Fix ...", "Add ...")
* is kept short (max 76 characters, ideally less), while concisely
- is written in the imperative (e.g., "Fix ...", "Add ...")
- is kept short (max 76 characters, ideally less), while concisely
explaining what the commit does
* is clear about what part of the code is affected -- often by prefixing
- is clear about what part of the code is affected -- often by prefixing
with the name of the subsystem and a colon, like "zjsunit: ..." or "docs: ..."
* is a complete sentence.
- is a complete sentence.
### Good summaries:
@@ -105,7 +106,7 @@ prefix "provision:", using lowercase "**p**". Next, "Improve performance of
install npm." starts with a capital "**I**", uses imperative tense,
and ends with a period.
> *provision: Improve performance of installing npm.*
> _provision: Improve performance of installing npm._
Here are some more positive examples:
@@ -121,16 +122,15 @@ Here are some more positive examples:
> gather_subscriptions: Fix exception handling bad input.
Compare "_gather_subscriptions: Fix exception handling bad input._" with:
Compare "*gather_subscriptions: Fix exception handling bad input.*" with:
* "*gather_subscriptions was broken*", which doesn't explain how
- "_gather_subscriptions was broken_", which doesn't explain how
it was broken (and isn't in the imperative)
* "*Fix exception when given bad input*", in which it's impossible to
- "_Fix exception when given bad input_", in which it's impossible to
tell from the summary what part of the codebase was changed
* "*gather_subscriptions: Fixing exception when given bad input.*",
- "_gather_subscriptions: Fixing exception when given bad input._",
not in the imperative
* "*gather_subscriptions: Fixed exception when given bad input.*",
- "_gather_subscriptions: Fixed exception when given bad input._",
not in the imperative
The summary is followed by a blank line, and then the body of the

View File

@@ -13,47 +13,47 @@ model GitHub supports).
## Usage
* **Claim an issue** — Comment `@zulipbot claim` on the issue you want
- **Claim an issue** — Comment `@zulipbot claim` on the issue you want
to claim; **@zulipbot** will assign you to the issue and label the issue as
**in progress**.
* If you're a new contributor, **@zulipbot** will give you read-only
- If you're a new contributor, **@zulipbot** will give you read-only
collaborator access to the repository and leave a welcome message on the
issue you claimed.
* You can also claim an issue that you've opened by including
- You can also claim an issue that you've opened by including
`@zulipbot claim` in the body of your issue.
* If you accidentally claim an issue you didn't want to claim, comment
- If you accidentally claim an issue you didn't want to claim, comment
`@zulipbot abandon` to abandon an issue.
* **Label your issues** — Add appropriate labels to issues that you opened by
- **Label your issues** — Add appropriate labels to issues that you opened by
including `@zulipbot add` in an issue comment or the body of your issue
followed by the desired labels enclosed within double quotes (`""`).
* For example, to add the **bug** and **help wanted** labels to your
- For example, to add the **bug** and **help wanted** labels to your
issue, comment or include `@zulipbot add "bug" "help wanted"` in the
issue body.
* You'll receive an error message if you try to add any labels to your issue
- You'll receive an error message if you try to add any labels to your issue
that don't exist in your repository.
* If you accidentally added the wrong labels, you can remove them by commenting
- If you accidentally added the wrong labels, you can remove them by commenting
`@zulipbot remove` followed by the desired labels enclosed with double quotes
(`""`).
* **Find unclaimed issues** — Use the [GitHub search
- **Find unclaimed issues** — Use the [GitHub search
feature](https://help.github.com/en/articles/using-search-to-filter-issues-and-pull-requests)
to find unclaimed issues by adding one of the following filters to your search:
* `-label: "in progress"` (excludes issues labeled with the **in progress** label)
- `-label: "in progress"` (excludes issues labeled with the **in progress** label)
* `no:assignee` (shows issues without assignees)
- `no:assignee` (shows issues without assignees)
Issues labeled with the **in progress** label and/or assigned to other users have
already been claimed.
* **Collaborate in area label teams** — Receive notifications on
- **Collaborate in area label teams** — Receive notifications on
issues and pull requests within your fields of expertise on the
[Zulip server repository](https://github.com/zulip/zulip) by joining
the Zulip server
@@ -73,7 +73,7 @@ team. Feel free to join as many area label teams as as you'd like!
label as well as any pull requests that reference issues labeled with your
team's area label.
* **Track inactive claimed issues** — If a claimed issue has not been updated
- **Track inactive claimed issues** — If a claimed issue has not been updated
for a week, **@zulipbot** will post a comment on the inactive issue to ask the
assignee(s) if they are still working on the issue.
@@ -90,5 +90,5 @@ assignee(s) if they are still working on the issue.
If you wish to help develop and contribute to **@zulipbot**, check out the
[zulip/zulipbot](https://github.com/zulip/zulipbot) repository on GitHub and read
the project's [contributing
guidelines](https://github.com/zulip/zulipbot/blob/master/.github/CONTRIBUTING.md#contributing) for
guidelines](https://github.com/zulip/zulipbot/blob/main/.github/CONTRIBUTING.md#contributing) for
more information.

View File

@@ -31,102 +31,103 @@ control over their email address, and then allowing them to set a
password for their account. There are two development environment
details worth understanding:
* All of our authentication flows in the development environment have
- All of our authentication flows in the development environment have
special links to the `/emails` page (advertised in `/devtools`),
which shows all emails that the Zulip server has "sent" (emails are
not actually sent by the development environment), to make it
convenient to click through the UI of signup, password reset, etc.
* There's a management command, `manage.py print_initial_password
username@example.com`, that prints out **default** passwords for the
development environment users. Note that if you change a user's
password in the development environment, those passwords will no longer
work. It also prints out the user's **current** API key.
- There's a management command,
`manage.py print_initial_password username@example.com`, that prints
out **default** passwords for the development environment users.
Note that if you change a user's password in the development
environment, those passwords will no longer work. It also prints
out the user's **current** API key.
### Google
* Visit [the Google developer
- Visit [the Google developer
console](https://console.developers.google.com) and navigate to "APIs
& services" > "Credentials". Create a "Project", which will correspond
to your dev environment.
* Navigate to "APIs & services" > "Library", and find the "Identity
- Navigate to "APIs & services" > "Library", and find the "Identity
Toolkit API". Choose "Enable".
* Return to "Credentials", and select "Create credentials". Choose
- Return to "Credentials", and select "Create credentials". Choose
"OAuth client ID", and follow prompts to create a consent screen, etc.
For "Authorized redirect URIs", fill in
`http://zulipdev.com:9991/complete/google/` .
* You should get a client ID and a client secret. Copy them. In
- You should get a client ID and a client secret. Copy them. In
`dev-secrets.conf`, set `social_auth_google_key` to the client ID
and `social_auth_google_secret` to the client secret.
### GitHub
* Register an OAuth2 application with GitHub at one of
- Register an OAuth2 application with GitHub at one of
<https://github.com/settings/developers> or
<https://github.com/organizations/ORGNAME/settings/developers>.
Specify `http://zulipdev.com:9991/complete/github/` as the callback URL.
* You should get a page with settings for your new application,
- You should get a page with settings for your new application,
showing a client ID and a client secret. In `dev-secrets.conf`, set
`social_auth_github_key` to the client ID and `social_auth_github_secret`
to the client secret.
### GitLab
* Register an OAuth application with GitLab at
- Register an OAuth application with GitLab at
<https://gitlab.com/oauth/applications>.
Specify `http://zulipdev.com:9991/complete/gitlab` as the callback URL.
* You should get a page containing the Application ID and Secret for
- You should get a page containing the Application ID and Secret for
your new application. In `dev-secrets.conf`, enter the Application
ID as `social_auth_gitlab_key` and the Secret as
`social_auth_gitlab_secret`.
### Apple
* Visit <https://developer.apple.com/account/resources/>,
- Visit <https://developer.apple.com/account/resources/>,
Enable App ID and Create a Services ID with the instructions in
<https://help.apple.com/developer-account/?lang=en#/dev1c0e25352> .
When prompted for a "Return URL", enter
`http://zulipdev.com:9991/complete/apple/` .
* [Create a Sign in with Apple private key](https://help.apple.com/developer-account/?lang=en#/dev77c875b7e)
- [Create a Sign in with Apple private key](https://help.apple.com/developer-account/?lang=en#/dev77c875b7e)
* In `dev-secrets.conf`, set
* `social_auth_apple_services_id` to your
- In `dev-secrets.conf`, set
- `social_auth_apple_services_id` to your
"Services ID" (eg. com.application.your).
* `social_auth_apple_app_id` to "App ID" or "Bundle ID".
- `social_auth_apple_app_id` to "App ID" or "Bundle ID".
This is only required if you are testing Apple auth on iOS.
* `social_auth_apple_key` to your "Key ID".
* `social_auth_apple_team` to your "Team ID".
* Put the private key file you got from apple at the path
- `social_auth_apple_key` to your "Key ID".
- `social_auth_apple_team` to your "Team ID".
- Put the private key file you got from apple at the path
`zproject/dev_apple.key`.
### SAML
* Sign up for a [developer Okta account](https://developer.okta.com/).
* Set up SAML authentication by following
- Sign up for a [developer Okta account](https://developer.okta.com/).
- Set up SAML authentication by following
[Okta's documentation](https://developer.okta.com/docs/guides/saml-application-setup/overview/).
Specify:
* `http://localhost:9991/complete/saml/` for the "Single sign on URL"`.
* `http://localhost:9991` for the "Audience URI (SP Entity ID)".
* Skip "Default RelayState".
* Skip "Name ID format".
* Set 'Email` for "Application username format".
* Provide "Attribute statements" of `email` to `user.email`,
- `http://localhost:9991/complete/saml/` for the "Single sign on URL"`.
- `http://localhost:9991` for the "Audience URI (SP Entity ID)".
- Skip "Default RelayState".
- Skip "Name ID format".
- Set 'Email` for "Application username format".
- Provide "Attribute statements" of `email` to `user.email`,
`first_name` to `user.firstName`, and `last_name` to `user.lastName`.
* Assign at least one account in the "Assignments" tab. You'll use it for
- Assign at least one account in the "Assignments" tab. You'll use it for
signing up / logging in to Zulip.
* Visit the big "Setup instructions" button on the "Sign on" tab.
* Edit `zproject/dev-secrets.conf` to add the two values provided:
* Set `saml_url = http...` from "Identity Provider Single Sign-On
- Visit the big "Setup instructions" button on the "Sign on" tab.
- Edit `zproject/dev-secrets.conf` to add the two values provided:
- Set `saml_url = http...` from "Identity Provider Single Sign-On
URL".
* Set `saml_entity_id = http://...` from "Identity Provider Issuer".
* Download the certificate and put it at the path `zproject/dev_saml.cert`.
* Now you should have working SAML authentication!
* You can sign up to the target realm with the account that you've "assigned"
- Set `saml_entity_id = http://...` from "Identity Provider Issuer".
- Download the certificate and put it at the path `zproject/dev_saml.cert`.
- Now you should have working SAML authentication!
- You can sign up to the target realm with the account that you've "assigned"
in the previous steps (if the account's email address is allowed in the realm,
so you may have to change the realm settings to allow the appropriate email domain)
and then you'll be able to log in freely. Alternatively, you can create an account
@@ -161,6 +162,7 @@ actual flows for LDAP configuration.
`zproject/dev_settings.py` to one of the following options. For more
information on these modes, refer to
[our production docs](../production/authentication-methods.html#ldap-including-active-directory):
- `a`: If users' email addresses are in LDAP and used as username.
- `b`: If LDAP only has usernames but email addresses are of the form
username@example.com
@@ -187,11 +189,11 @@ We also have configured `AUTH_LDAP_USER_ATTR_MAP` in
`zproject/dev_settings.py` to sync several of those fields. For
example:
* Modes `a` and `b` will set the user's avatar on account creation and
- Modes `a` and `b` will set the user's avatar on account creation and
update it when `manage.py sync_ldap_user_data` is run.
* Mode `b` is configured to automatically have the `birthday` and
- Mode `b` is configured to automatically have the `birthday` and
`Phone number` custom profile fields populated/synced.
* Mode `a` is configured to deactivate/reactivate users whose accounts
- Mode `a` is configured to deactivate/reactivate users whose accounts
are disabled in LDAP when `manage.py sync_ldap_user_data` is run.
(Note that you'll likely need to edit
`zerver/lib/dev_ldap_directory.py` to ensure there are some accounts

View File

@@ -28,8 +28,8 @@ performs well.
Zulip also supports a wide range of ways to install the Zulip
development environment:
* On Linux platforms, you can **[install directly][install-direct]**.
* On Windows, you can **[install directly][install-via-wsl]** via WSL 2.
- On Linux platforms, you can **[install directly][install-direct]**.
- On Windows, you can **[install directly][install-via-wsl]** via WSL 2.
## Slow internet connections
@@ -65,8 +65,8 @@ need to.
Once you've installed the Zulip development environment, you'll want
to read these documents to learn how to use it:
* [Using the development environment][using-dev-env]
* [Testing][testing] (and [Configuring CI][ci])
- [Using the development environment][using-dev-env]
- [Testing][testing] (and [Configuring CI][ci])
And if you've set up the Zulip development environment on a remote
machine, take a look at our tips for

View File

@@ -16,12 +16,12 @@ need to.
The best way to connect to your server is using the command line tool `ssh`.
* On macOS and Linux/UNIX, `ssh` is a part of Terminal.
* On Windows, `ssh` comes with [Bash for Git][git-bash].
- On macOS and Linux/UNIX, `ssh` is a part of Terminal.
- On Windows, `ssh` comes with [Bash for Git][git-bash].
Open *Terminal* or *Bash for Git*, and connect with the following:
Open _Terminal_ or _Bash for Git_, and connect with the following:
```
```console
$ ssh username@host
```
@@ -37,14 +37,15 @@ to the next section.
You can create a new user with sudo privileges by running the
following commands as root:
* You can create a `zulipdev` user by running the command `adduser
zulipdev`. Run through the prompts to assign a password and user
information. (You can pick any username you like for this user
- You can create a `zulipdev` user by running the command
`adduser zulipdev`. Run through the prompts to assign a password and
user information. (You can pick any username you like for this user
account.)
* You can add the user to the sudo group by running the command
- You can add the user to the sudo group by running the command
`usermod -aG sudo zulipdev`.
* Finally, you can switch to the user by running the command `su -
zulipdev` (or just log in to that user using `ssh`).
- Finally, you can switch to the user by running the command
`su - zulipdev` (or just log in to that user using `ssh`).
## Setting up the development environment
@@ -75,7 +76,7 @@ Once you have set up the development environment, you can start up the
development server with the following command in the directory where
you cloned Zulip:
```
```bash
./tools/run-dev.py --interface=''
```
@@ -98,7 +99,7 @@ such as a DigitalOcean Droplet or an AWS EC2 instance, you can set up
port-forwarding to access Zulip by running the following command in
your terminal:
```
```bash
ssh -L 3000:127.0.0.1:9991 <username>@<remote_server_ip> -N
```
@@ -112,10 +113,10 @@ environment][rtd-using-dev-env].
To see changes on your remote development server, you need to do one of the following:
* [Edit locally](#editing-locally): Clone Zulip code to your computer and
- [Edit locally](#editing-locally): Clone Zulip code to your computer and
then use your favorite editor to make changes. When you want to see changes
on your remote Zulip development instance, sync with Git.
* [Edit remotely](#editing-remotely): Edit code directly on your remote
- [Edit remotely](#editing-remotely): Edit code directly on your remote
Zulip development instance using a [Web-based IDE](#web-based-ide) (recommended for
beginners) or a [command line editor](#command-line-editors), or a
[desktop IDE](#desktop-gui-editors) using a plugin to sync your
@@ -126,12 +127,12 @@ To see changes on your remote development server, you need to do one of the foll
If you want to edit code locally install your favorite text editor. If you
don't have a favorite, here are some suggestions:
* [atom](https://atom.io/)
* [emacs](https://www.gnu.org/software/emacs/)
* [vim](https://www.vim.org/)
* [spacemacs](https://github.com/syl20bnr/spacemacs)
* [sublime](https://www.sublimetext.com/)
* [PyCharm](https://www.jetbrains.com/pycharm/)
- [atom](https://atom.io/)
- [emacs](https://www.gnu.org/software/emacs/)
- [vim](https://www.vim.org/)
- [spacemacs](https://github.com/syl20bnr/spacemacs)
- [sublime](https://www.sublimetext.com/)
- [PyCharm](https://www.jetbrains.com/pycharm/)
Next, follow our [Git and GitHub guide](../git/index.md) to clone and configure
your fork of zulip on your local computer.
@@ -149,7 +150,7 @@ guide][rtd-git-guide]. In brief, the steps are as follows.
On your **local computer**:
1. Open *Terminal* (macOS/Linux) or *Git for BASH*.
1. Open _Terminal_ (macOS/Linux) or _Git for BASH_.
2. Change directory to where you cloned Zulip (e.g. `cd zulip`).
3. Use `git add` and `git commit` to stage and commit your changes (if you
haven't already).
@@ -160,7 +161,7 @@ Be sure to replace `branchname` with the name of your actual feature branch.
Once `git push` has completed successfully, you are ready to fetch the commits
from your remote development instance:
1. In *Terminal* or *Git BASH*, connect to your remote development
1. In _Terminal_ or _Git BASH_, connect to your remote development
instance with `ssh user@host`.
2. Change to the zulip directory (e.g., `cd zulip`).
3. Fetch new commits from GitHub with `git fetch origin`.
@@ -172,10 +173,10 @@ from your remote development instance:
There are a few good ways to edit code in your remote development
environment:
* With a command-line editor like vim or emacs run over SSH.
* With a desktop GUI editor like VS Code or Atom and a plugin for
- With a command-line editor like vim or emacs run over SSH.
- With a desktop GUI editor like VS Code or Atom and a plugin for
syncing your changes to the remote server.
* With a web-based IDE like CodeAnywhere.
- With a web-based IDE like CodeAnywhere.
We document these options below; we recommend using whatever editor
you prefer for development in general.
@@ -199,10 +200,11 @@ developing locally.
[vscode-remote-ssh]: https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh
- [rmate](https://github.com/textmate/rmate) for TextMate + VS Code:
1. Install the extension
[Remote VSCode](https://marketplace.visualstudio.com/items?itemName=rafaelmaiolla.remote-vscode).
2. On your remote machine, run:
```
```console
$ mkdir -p ~/bin
$ curl -Lo ~/bin/rmate https://raw.githubusercontent.com/textmate/rmate/master/bin/rmate
$ chmod a+x ~/bin/rmate
@@ -210,11 +212,11 @@ developing locally.
3. Make sure the remote server is running in VS Code (you can
force-start through the Command Palette).
4. SSH to your remote machine using
```
```console
$ ssh -R 52698:localhost:52698 user@example.org
```
5. On your remote machine, run
```
```console
$ rmate [options] file
```
and the file should open up in VS Code. Any changes you make now will be saved remotely.
@@ -226,20 +228,20 @@ a command line text editor on the remote machine.
Two editors often available by default on Linux systems are:
* **Nano**: A very simple, beginner-friendly editor. However, it lacks a lot of
- **Nano**: A very simple, beginner-friendly editor. However, it lacks a lot of
features useful for programming, such as syntax highlighting, so we only
recommended it for quick edits to things like configuration files. Launch by
running command `nano <filename>`. Exit by pressing *Ctrl-X*.
running command `nano <filename>`. Exit by pressing _Ctrl-X_.
* **[Vim](https://www.vim.org/)**: A very powerful editor that can take a while
to learn. Launch by running `vim <filename>`. Quit Vim by pressing *Esc*,
typing `:q`, and then pressing *Enter*. Vim comes with a program to learn it
- **[Vim](https://www.vim.org/)**: A very powerful editor that can take a while
to learn. Launch by running `vim <filename>`. Quit Vim by pressing _Esc_,
typing `:q`, and then pressing _Enter_. Vim comes with a program to learn it
called `vimtutor` (just run that command to start it).
Other options include:
* [emacs](https://www.gnu.org/software/emacs/)
* [spacemacs](https://github.com/syl20bnr/spacemacs)
- [emacs](https://www.gnu.org/software/emacs/)
- [spacemacs](https://github.com/syl20bnr/spacemacs)
##### Web-based IDE
@@ -250,12 +252,12 @@ started working quickly, we recommend web-based IDE
To set up Codeanywhere for Zulip:
1. Create a [Codeanywhere][codeanywhere] account and log in.
2. Create a new **SFTP-SSH** project. Use *Public key* for authentication.
2. Create a new **SFTP-SSH** project. Use _Public key_ for authentication.
3. Click **GET YOUR PUBLIC KEY** to get the new public key that
Codeanywhere generates when you create a new project. Add this public key to
`~/.ssh/authorized_keys` on your remote development instance.
4. Once you've added the new public key to your remote development instance, click
*CONNECT*.
_CONNECT_.
Now your workspace should look similar this:
![Codeanywhere workspace][img-ca-workspace]
@@ -264,9 +266,9 @@ Now your workspace should look similar this:
Next, read the following to learn more about developing for Zulip:
* [Git & GitHub guide][rtd-git-guide]
* [Using the development environment][rtd-using-dev-env]
* [Testing][rtd-testing]
- [Git & GitHub guide][rtd-git-guide]
- [Using the development environment][rtd-using-dev-env]
- [Testing][rtd-testing]
[install-direct]: ../development/setup-advanced.html#installing-directly-on-ubuntu-debian-centos-or-fedora
[install-vagrant]: ../development/setup-vagrant.md
@@ -292,7 +294,8 @@ different.
1. First, get an SSL certificate; you can use
[our certbot wrapper script used for production](../production/ssl-certificates.html#certbot-recommended)
by running the following commands as root:
```
```bash
# apt install -y crudini
mkdir -p /var/lib/zulip/certbot-webroot/
# if nginx running this will fail and you need to run `service nginx stop`
@@ -303,7 +306,7 @@ different.
1. Install nginx configuration:
```
```bash
apt install -y nginx-full
cp -a /home/zulipdev/zulip/tools/droplets/zulipdev /etc/nginx/sites-available/
ln -nsf /etc/nginx/sites-available/zulipdev /etc/nginx/sites-enabled/
@@ -311,11 +314,11 @@ different.
service nginx reload # Actually enabled your nginx configuration
```
1. Edit `zproject/dev_settings.py` to set `EXTERNAL_URI_SCHEME =
"https://"`, so that URLs served by the development environment
will be HTTPS.
1. Edit `zproject/dev_settings.py` to set
`EXTERNAL_URI_SCHEME = "https://"`, so that URLs served by the
development environment will be HTTPS.
1. Start the Zulip development environment with the following command:
```
```bash
env EXTERNAL_HOST="hostname.example.com" ./tools/run-dev.py --interface=''
```

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -70,14 +70,14 @@ Once your remote dev instance is ready:
Once you've confirmed you can connect to your remote server, take a look at:
* [developing remotely](../development/remote.md) for tips on using the remote dev
- [developing remotely](../development/remote.md) for tips on using the remote dev
instance, and
* our [Git & GitHub guide](../git/index.md) to learn how to use Git with Zulip.
- our [Git & GitHub guide](../git/index.md) to learn how to use Git with Zulip.
Next, read the following to learn more about developing for Zulip:
* [Using the development environment](../development/using.md)
* [Testing](../testing/testing.md)
- [Using the development environment](../development/using.md)
- [Testing](../testing/testing.md)
[github-join]: https://github.com/join
[github-help-add-ssh-key]: https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account

View File

@@ -2,22 +2,22 @@
Contents:
* [Installing directly on Ubuntu, Debian, CentOS, or Fedora](#installing-directly-on-ubuntu-debian-centos-or-fedora)
* [Installing directly on Windows 10 with WSL 2](#installing-directly-on-windows-10-with-wsl-2)
* [Using the Vagrant Hyper-V provider on Windows](#using-the-vagrant-hyper-v-provider-on-windows-beta)
* [Newer versions of supported platforms](#newer-versions-of-supported-platforms)
* [Installing directly on cloud9](#installing-on-cloud9)
- [Installing directly on Ubuntu, Debian, CentOS, or Fedora](#installing-directly-on-ubuntu-debian-centos-or-fedora)
- [Installing directly on Windows 10 with WSL 2](#installing-directly-on-windows-10-with-wsl-2)
- [Using the Vagrant Hyper-V provider on Windows](#using-the-vagrant-hyper-v-provider-on-windows-beta)
- [Newer versions of supported platforms](#newer-versions-of-supported-platforms)
- [Installing directly on cloud9](#installing-on-cloud9)
## Installing directly on Ubuntu, Debian, CentOS, or Fedora
If you'd like to install a Zulip development environment on a computer
that's running one of:
* Ubuntu 20.04 Focal, 18.04 Bionic
* Debian 10 Buster, 11 Bullseye (beta)
* CentOS 7 (beta)
* Fedora 33 (beta)
* RHEL 7 (beta)
- Ubuntu 20.04 Focal, 18.04 Bionic
- Debian 10 Buster, 11 Bullseye (beta)
- CentOS 7 (beta)
- Fedora 33 (beta)
- RHEL 7 (beta)
You can just run the Zulip provision script on your machine.
@@ -26,23 +26,22 @@ If you are using a [remote server](../development/remote.md), see
the
[section on creating appropriate user accounts](../development/remote.html#setting-up-user-accounts).
```eval_rst
.. warning::
:::{warning}
There is no supported uninstallation process with this
method. If you want that, use the Vagrant environment, where you can
just do `vagrant destroy` to clean up the development environment.
```
:::
Start by [cloning your fork of the Zulip repository][zulip-rtd-git-cloning]
and [connecting the Zulip upstream repository][zulip-rtd-git-connect]:
```
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
```
```
```bash
# On CentOS/RHEL, you must first install epel-release, and then python36,
# and finally you must run `sudo ln -nsf /usr/bin/python36 /usr/bin/python3`
# On Fedora, you must first install python3
@@ -71,20 +70,20 @@ installation method described here.
1. Launch the `Ubuntu 18.04` shell and run the following commands:
```
```bash
sudo apt update && sudo apt upgrade
sudo apt install rabbitmq-server memcached redis-server postgresql
```
1. Open `/etc/rabbitmq/rabbitmq-env.conf` using e.g.:
```
```bash
sudo vim /etc/rabbitmq/rabbitmq-env.conf
```
Add the following lines at the end of your file and save:
```
```ini
NODE_IP_ADDRESS=127.0.0.1
NODE_PORT=5672
```
@@ -92,14 +91,15 @@ installation method described here.
1. Make sure you are inside the WSL disk and not in a Windows mounted disk.
You will run into permission issues if you run `provision` from `zulip`
in a Windows mounted disk.
```
```bash
cd ~ # or cd /home/USERNAME
```
1. [Clone your fork of the Zulip repository][zulip-rtd-git-cloning]
and [connecting the Zulip upstream repository][zulip-rtd-git-connect]:
```
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git ~/zulip
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
@@ -109,7 +109,7 @@ installation method described here.
start it (click `Allow access` if you get popups for Windows Firewall
blocking some services)
```
```bash
# Start database, cache, and other services
./tools/wsl/start_services
# Install/update the Zulip development environment
@@ -120,11 +120,10 @@ installation method described here.
./tools/run-dev.py
```
```eval_rst
.. note::
:::{note}
If you shut down WSL, after starting it again, you will have to manually start
the services using ``./tools/wsl/start_services``.
```
the services using `./tools/wsl/start_services`.
:::
1. If you are facing problems or you see error messages after running `./tools/run-dev.py`,
you can try running `./tools/provision` again.
@@ -154,7 +153,7 @@ expected.
1. Start by [cloning your fork of the Zulip repository][zulip-rtd-git-cloning]
and [connecting the Zulip upstream repository][zulip-rtd-git-connect]:
```
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
@@ -169,7 +168,7 @@ expected.
You should get output like this:
```text
```console
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
@@ -203,36 +202,35 @@ expected.
1. Set the `EXTERNAL_HOST` environment variable.
```bash
```console
(zulip-py3-venv) vagrant@ubuntu-18:/srv/zulip$ export EXTERNAL_HOST="$(hostname -I | xargs):9991"
(zulip-py3-venv) vagrant@ubuntu-18:/srv/zulip$ echo $EXTERNAL_HOST
```
The output will be like:
```text
```console
172.28.122.156:9991
```
Make sure you note down this down. This is where your zulip development web
server can be accessed.
```eval_rst
.. important::
:::{important}
The output of the above command changes every time you restart the Vagrant
development machine. Thus, it will have to be run every time you bring one up.
This quirk is one reason this method is marked experimental.
```
:::
1. You should now be able to start the Zulip development server.
```bash
```console
(zulip-py3-venv) vagrant@ubuntu-18:/srv/zulip$ ./tools/run-dev.py
```
The output will look like:
```text
```console
Starting Zulip on:
http://172.30.24.235:9991/
@@ -254,10 +252,10 @@ expected.
1. If you get the error `Hyper-V could not initialize memory`, this is
likely because your system has insufficient free memory to start
the virtual machine. You can generally work around this error by
closing all other running programs and running `vagrant up
--provider=hyperv` again. You can reopen the other programs after
the provisioning is completed. If it still isn't enough, try
restarting your system and running the command again.
closing all other running programs and running
`vagrant up --provider=hyperv` again. You can reopen the other
programs after the provisioning is completed. If it still isn't
enough, try restarting your system and running the command again.
2. Be patient the first time you run `./tools/run-dev.py`.
@@ -294,17 +292,18 @@ This section documents how to set up the Zulip development environment
in a Cloud9 workspace. If you don't have an existing Cloud9 account,
you can sign up [here](https://aws.amazon.com/cloud9/).
* Create a Workspace, and select the blank template.
* Resize the workspace to be 1GB of memory and 4GB of disk
- Create a Workspace, and select the blank template.
- Resize the workspace to be 1GB of memory and 4GB of disk
space. (This is under free limit for both the old Cloud9 and the AWS
Free Tier).
* Clone the zulip repo: `git clone --config pull.rebase
https://github.com/<your-username>/zulip.git`
* Restart rabbitmq-server since its broken on Cloud9: `sudo service
rabbitmq-server restart`.
* And run provision `cd zulip && ./tools/provision`, once this is done.
* Activate the Zulip virtual environment by `source
/srv/zulip-py3-venv/bin/activate` or by opening a new terminal.
- Clone the zulip repo:
`git clone --config pull.rebase https://github.com/<your-username>/zulip.git`
- Restart rabbitmq-server since its broken on Cloud9:
`sudo service rabbitmq-server restart`.
- And run provision `cd zulip && ./tools/provision`, once this is done.
- Activate the Zulip virtual environment by
`source /srv/zulip-py3-venv/bin/activate` or by opening a new
terminal.
#### Install zulip-cloud9

View File

@@ -10,16 +10,17 @@ or a Linux container (for Ubuntu) inside which the Zulip server and
all related services will run.
Contents:
* [Requirements](#requirements)
* [Step 0: Set up Git & GitHub](#step-0-set-up-git-github)
* [Step 1: Install prerequisites](#step-1-install-prerequisites)
* [Step 2: Get Zulip code](#step-2-get-zulip-code)
* [Step 3: Start the development environment](#step-3-start-the-development-environment)
* [Step 4: Developing](#step-4-developing)
* [Troubleshooting and common errors](#troubleshooting-and-common-errors)
* [Specifying an Ubuntu mirror](#specifying-an-ubuntu-mirror)
* [Specifying a proxy](#specifying-a-proxy)
* [Customizing CPU and RAM allocation](#customizing-cpu-and-ram-allocation)
- [Requirements](#requirements)
- [Step 0: Set up Git & GitHub](#step-0-set-up-git-github)
- [Step 1: Install prerequisites](#step-1-install-prerequisites)
- [Step 2: Get Zulip code](#step-2-get-zulip-code)
- [Step 3: Start the development environment](#step-3-start-the-development-environment)
- [Step 4: Developing](#step-4-developing)
- [Troubleshooting and common errors](#troubleshooting-and-common-errors)
- [Specifying an Ubuntu mirror](#specifying-an-ubuntu-mirror)
- [Specifying a proxy](#specifying-a-proxy)
- [Customizing CPU and RAM allocation](#customizing-cpu-and-ram-allocation)
**If you encounter errors installing the Zulip development
environment,** check [troubleshooting and common
@@ -32,10 +33,10 @@ server](../contributing/chat-zulip-org.md) for real-time help or
When reporting your issue, please include the following information:
* host operating system
* installation method (Vagrant or direct)
* whether or not you are using a proxy
* a copy of Zulip's `vagrant` provisioning logs, available in
- host operating system
- installation method (Vagrant or direct)
- whether or not you are using a proxy
- a copy of Zulip's `vagrant` provisioning logs, available in
`/var/log/provision.log` on your virtual machine
### Requirements
@@ -73,10 +74,10 @@ GitHub account using
Jump to:
* [macOS](#macos)
* [Ubuntu](#ubuntu)
* [Debian](#debian)
* [Windows](#windows-10)
- [macOS](#macos)
- [Ubuntu](#ubuntu)
- [Debian](#debian)
- [Windows](#windows-10)
#### macOS
@@ -94,14 +95,14 @@ Now you are ready for [Step 2: Get Zulip code](#step-2-get-zulip-code).
##### 1. Install Vagrant, Docker, and Git
```
```console
christie@ubuntu-desktop:~
$ sudo apt install vagrant docker.io git
```
##### 2. Add yourself to the `docker` group:
```
```console
christie@ubuntu-desktop:~
$ sudo adduser $USER docker
Adding user `christie' to group `docker' ...
@@ -112,7 +113,7 @@ Done.
You will need to reboot for this change to take effect. If it worked,
you will see `docker` in your list of groups:
```
```console
christie@ubuntu-desktop:~
$ groups | grep docker
christie adm cdrom sudo dip plugdev lpadmin sambashare docker
@@ -126,7 +127,7 @@ bug](https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1844894)
may prevent Docker from being automatically enabled and started after
installation. You can check using the following:
```
```console
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
@@ -137,7 +138,7 @@ If the service is not running, you'll see `Active: inactive (dead)` on
the second line, and will need to enable and start the Docker service
using the following:
```
```bash
sudo systemctl unmask docker
sudo systemctl enable docker
sudo systemctl start docker
@@ -154,16 +155,15 @@ Debian](https://docs.docker.com/install/linux/docker-ce/debian/).
#### Windows 10
```eval_rst
.. note::
We recommend using `WSL 2 for Windows development <../development/setup-advanced.html#installing-directly-on-windows-10-with-wsl-2>`_.
```
:::{note}
We recommend using [WSL 2 for Windows development](../development/setup-advanced.html#installing-directly-on-windows-10-with-wsl-2).
:::
1. Install [Git for Windows][git-bash], which installs *Git BASH*.
1. Install [Git for Windows][git-bash], which installs _Git BASH_.
2. Install [VirtualBox][vbox-dl] (latest).
3. Install [Vagrant][vagrant-dl] (latest).
(Note: While *Git BASH* is recommended, you may also use [Cygwin][cygwin-dl].
(Note: While _Git BASH_ is recommended, you may also use [Cygwin][cygwin-dl].
If you do, make sure to **install default required packages** along with
**git**, **curl**, **openssh**, and **rsync** binaries.)
@@ -189,13 +189,13 @@ In **Git for BASH**:
Open **Git BASH as an administrator** and run:
```
```console
$ git config --global core.symlinks true
```
Now confirm the setting:
```
```console
$ git config core.symlinks
true
```
@@ -210,7 +210,7 @@ In **Cygwin**:
Open a Cygwin window **as an administrator** and do this:
```
```console
christie@win10 ~
$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
```
@@ -218,7 +218,7 @@ $ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
Next, close that Cygwin window and open another. If you `echo` $CYGWIN you
should see:
```
```console
christie@win10 ~
$ echo $CYGWIN
winsymlinks:native
@@ -244,7 +244,7 @@ projects and to instead follow these instructions exactly.)
[clone your fork of the Zulip repository](../git/cloning.html#step-1b-clone-to-your-machine) and
[connect the Zulip upstream repository](../git/cloning.html#step-1c-connect-your-fork-to-zulip-upstream):
```
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
@@ -255,7 +255,7 @@ This will create a 'zulip' directory and download the Zulip code into it.
Don't forget to replace YOURUSERNAME with your Git username. You will see
something like:
```
```console
christie@win10 ~
$ git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
Cloning into 'zulip'...
@@ -276,7 +276,7 @@ environment](#step-3-start-the-development-environment).
Change into the zulip directory and tell vagrant to start the Zulip
development environment with `vagrant up`:
```
```bash
# On Windows or macOS:
cd zulip
vagrant plugin install vagrant-vbguest
@@ -314,20 +314,21 @@ section. If that doesn't help, please visit
in the [Zulip development community server](../contributing/chat-zulip-org.md) for
real-time help.
On Windows, you will see the message `The system cannot find the path
specified.` several times. This is normal and is not a problem.
On Windows, you will see the message
`The system cannot find the path specified.` several times. This is
normal and is not a problem.
Once `vagrant up` has completed, connect to the development
environment with `vagrant ssh`:
```
```console
christie@win10 ~/zulip
$ vagrant ssh
```
You should see output that starts like this:
```
```console
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-54-generic x86_64)
```
@@ -340,14 +341,14 @@ provisioning failed and you should look at the
Next, start the Zulip server:
```
```console
(zulip-py3-venv) vagrant@ubuntu-bionic:/srv/zulip
$ ./tools/run-dev.py
```
You will see several lines of output starting with something like:
```
```console
2016-05-04 22:20:33,895 INFO: process_fts_updates starting
Recompiling templates
2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates
@@ -362,9 +363,10 @@ Quit the server with CTRL-C.
2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds
Performing system checks...
```
And ending with something similar to:
```
```console
http://localhost:9994/webpack-dev-server/
webpack result is served from http://localhost:9991/webpack/
content is served from /srv/zulip
@@ -385,7 +387,7 @@ The Zulip server will continue to run and send output to the terminal window.
When you navigate to Zulip in your browser, check your terminal and you
should see something like:
```
```console
2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) / (unauth@zulip via ?)
[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0
2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth@zulip via ?)
@@ -438,7 +440,7 @@ guide][rtd-git-guide].
If after rebasing onto a new version of the Zulip server, you receive
new errors while starting the Zulip server or running tests, this is
probably not because Zulip's master branch is broken. Instead, this
probably not because Zulip's `main` branch is broken. Instead, this
is likely because we've recently merged changes to the development
environment provisioning process that you need to apply to your
development environment. To update your environment, you'll need to
@@ -484,7 +486,7 @@ can halt vagrant from another Terminal/Git BASH window.
From the window where run-dev.py is running:
```
```console
2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth@zulip via ?)
^C
KeyboardInterrupt
@@ -493,9 +495,10 @@ logout
Connection to 127.0.0.1 closed.
christie@win10 ~/zulip
```
Now you can suspend the development environment:
```
```console
christie@win10 ~/zulip
$ vagrant suspend
==> default: Saving VM state and suspending execution...
@@ -503,7 +506,7 @@ $ vagrant suspend
If `vagrant suspend` doesn't work, try `vagrant halt`:
```
```console
christie@win10 ~/zulip
$ vagrant halt
==> default: Attempting graceful shutdown of VM...
@@ -520,7 +523,7 @@ pass the `--provider` option required above). You will also need to
connect to the virtual machine with `vagrant ssh` and re-start the
Zulip server:
```
```console
christie@win10 ~/zulip
$ vagrant up
$ vagrant ssh
@@ -533,9 +536,9 @@ $ ./tools/run-dev.py
Next, read the following to learn more about developing for Zulip:
* [Git & GitHub guide][rtd-git-guide]
* [Using the development environment][rtd-using-dev-env]
* [Testing][rtd-testing] (and [Configuring CI][ci] to
- [Git & GitHub guide][rtd-git-guide]
- [Using the development environment][rtd-using-dev-env]
- [Testing][rtd-testing] (and [Configuring CI][ci] to
run the full test suite against any branches you push to your fork,
which can help you optimize your development workflow).
@@ -549,16 +552,16 @@ equivalently `vagrant provision` from outside).
If these solutions aren't working for you or you encounter an issue not
documented below, there are a few ways to get further help:
* Ask in [#provision help](https://chat.zulip.org/#narrow/stream/21-provision-help)
- Ask in [#provision help](https://chat.zulip.org/#narrow/stream/21-provision-help)
in the [Zulip development community server](../contributing/chat-zulip-org.md).
* [File an issue](https://github.com/zulip/zulip/issues).
- [File an issue](https://github.com/zulip/zulip/issues).
When reporting your issue, please include the following information:
* host operating system
* installation method (Vagrant or direct)
* whether or not you are using a proxy
* a copy of Zulip's `vagrant` provisioning logs, available in
- host operating system
- installation method (Vagrant or direct)
- whether or not you are using a proxy
- a copy of Zulip's `vagrant` provisioning logs, available in
`/var/log/provision.log` on your virtual machine. If you choose to
post just the error output, please include the **beginning of the
error output**, not just the last few lines.
@@ -572,7 +575,7 @@ This is caused by provisioning failing to complete successfully. You
can see the errors in `var/log/provision.log`; it should end with
something like this:
```
```text
ESC[94mZulip development environment setup succeeded!ESC[0m
```
@@ -589,7 +592,8 @@ shell and run `vagrant ssh` again to get the virtualenv setup properly.
#### Vagrant was unable to mount VirtualBox shared folders
For the following error:
```
```console
Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel
@@ -603,7 +607,7 @@ was:
If this error starts happening unexpectedly, then just run:
```
```bash
vagrant halt
vagrant up
```
@@ -615,7 +619,7 @@ to reboot the guest. After this, you can do `vagrant provision` and
If you receive the following error while running `vagrant up`:
```
```console
SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
```
@@ -627,14 +631,14 @@ better network connection).
When running `vagrant up` or `provision`, if you see the following error:
```
```console
==> default: E:unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
```
It means that your local apt repository has been corrupted, which can
usually be resolved by executing the command:
```
```bash
apt-get -f install
```
@@ -642,7 +646,7 @@ apt-get -f install
On running `vagrant ssh`, if you see the following error:
```
```console
ssh_exchange_identification: Connection closed by remote host
```
@@ -655,7 +659,7 @@ for more details.
If you receive the following error while running `vagrant up`:
```
```console
==> default: Traceback (most recent call last):
==> default: File "./emoji_dump.py", line 75, in <module>
==> default:
@@ -697,7 +701,7 @@ Get the name of your virtual machine by running `vboxmanage list vms` and
then print out the custom settings for this virtual machine with
`vboxmanage getextradata YOURVMNAME enumerate`:
```
```console
christie@win10 ~/zulip
$ vboxmanage list vms
"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
@@ -716,7 +720,7 @@ If `vboxmanage enumerate` prints nothing, or shows a value of 0 for
VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable
symbolic links by running this command in Terminal/Git BASH/Cygwin:
```
```bash
vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip 1
```
@@ -730,7 +734,7 @@ Windows is incorrectly attempting to use Hyper-V rather than
Virtualbox as the virtualization provider. You can fix this by
explicitly passing the virtualbox provider to `vagrant up`:
```
```console
christie@win10 ~/zulip
$ vagrant up --provide=virtualbox
```
@@ -739,15 +743,15 @@ $ vagrant up --provide=virtualbox
If you see the following error after running `vagrant up`:
```
```console
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Error: Connection timeout. Retrying...
default: Error: Connection timeout. Retrying...
default: Error: Connection timeout. Retrying...
```
A likely cause is that hardware virtualization is not enabled for your
computer. This must be done via your computer's BIOS settings. Look for a
setting called VT-x (Intel) or (AMD-V).
@@ -762,7 +766,7 @@ this post](https://stackoverflow.com/questions/22575261/vagrant-stuck-connection
If you see the following error when you run `vagrant up`:
```
```console
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
@@ -789,19 +793,19 @@ by rebooting the guest via `vagrant halt; vagrant up`.
The `vagrant up` command basically does the following:
* Downloads an Ubuntu image and starts it using a Vagrant provider.
* Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs
- Downloads an Ubuntu image and starts it using a Vagrant provider.
- Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs
`tools/provision`, which has a lot of subcommands that are
executed via Python's `subprocess` module. These errors mean that
one of those subcommands failed.
To debug such errors, you can log in to the Vagrant guest machine by
running `vagrant ssh`, which should present you with a standard shell
prompt. You can debug interactively by using e.g. `cd zulip &&
./tools/provision`, and then running the individual subcommands
that failed. Once you've resolved the problem, you can rerun
`tools/provision` to proceed; the provisioning system is designed
to recover well from failures.
prompt. You can debug interactively by using e.g.
`cd zulip && ./tools/provision`, and then running the individual
subcommands that failed. Once you've resolved the problem, you can
rerun `tools/provision` to proceed; the provisioning system is
designed to recover well from failures.
The Zulip provisioning system is generally highly reliable; the most common
cause of issues here is a poor network connection (or one where you need a
@@ -809,7 +813,8 @@ proxy to access the Internet and haven't [configured the development
environment to use it](#specifying-a-proxy).
Once you've provisioned successfully, you'll get output like this:
```
```console
Zulip development environment setup succeeded!
(zulip-py3-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$
```
@@ -836,7 +841,7 @@ the VM.
##### yarn install warnings
```
```console
$ yarn install
yarn install v0.24.5
[1/4] Resolving packages...
@@ -853,7 +858,7 @@ It is okay to proceed and start the Zulip server.
#### VBoxManage errors related to VT-x or WHvSetupPartition
```
```console
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
@@ -866,7 +871,7 @@ VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap,
or
```
```console
Stderr: VBoxManage.exe: error: Call to WHvSetupPartition failed: ERROR_SUCCESS (Last=0xc000000d/87) (VERR_NEM_VM_CREATE_FAILED)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole
```
@@ -876,13 +881,14 @@ enabled in your BIOS.
If the error persists, you may have run into an incompatibility
between VirtualBox and Hyper-V on Windows. To disable Hyper-V, open
command prompt as administrator, run `bcdedit /set
hypervisorlaunchtype off`, and reboot. If you need to enable it
later, run `bcdedit /deletevalue hypervisorlaunchtype`, and reboot.
command prompt as administrator, run
`bcdedit /set hypervisorlaunchtype off`, and reboot. If you need to
enable it later, run `bcdedit /deletevalue hypervisorlaunchtype`, and
reboot.
#### OSError: [Errno 26] Text file busy
```
```console
default: Traceback (most recent call last):
default: File "/srv/zulip-py3-venv/lib/python3.6/shutil.py", line 426, in _rmtree_safe_fd
@@ -896,7 +902,7 @@ the VirtualBox Guest Additions for Linux on Windows hosts. You can
check the running version of VirtualBox Guest Additions with this
command:
```
```bash
vagrant ssh -- 'modinfo -F version vboxsf'
```
@@ -905,13 +911,13 @@ able to work around it by downgrading VirtualBox Guest Additions to
6.0.4. To do this, create a `~/.zulip-vagrant-config` file and add
this line:
```
```text
VBOXADD_VERSION 6.0.4
```
Then run these commands (yes, reload is needed twice):
```
```bash
vagrant plugin install vagrant-vbguest
vagrant reload
vagrant reload --provision
@@ -927,7 +933,7 @@ a local mirror closer to your location. To do this, create
`~/.zulip-vagrant-config` and add a line like this, replacing the URL
as appropriate:
```
```text
UBUNTU_MIRROR http://us.archive.ubuntu.com/ubuntu/
```
@@ -937,14 +943,14 @@ If you need to use a proxy server to access the Internet, you will
need to specify the proxy settings before running `Vagrant up`.
First, install the Vagrant plugin `vagrant-proxyconf`:
```
```bash
vagrant plugin install vagrant-proxyconf
```
Then create `~/.zulip-vagrant-config` and add the following lines to
it (with the appropriate values in it for your proxy):
```
```text
HTTP_PROXY http://proxy_host:port
HTTPS_PROXY http://proxy_host:port
NO_PROXY localhost,127.0.0.1,.example.com,.zulipdev.com
@@ -953,7 +959,7 @@ NO_PROXY localhost,127.0.0.1,.example.com,.zulipdev.com
For proxies that require authentication, the config will be a bit more
complex, e.g.:
```
```text
HTTP_PROXY http://userName:userPassword@192.168.1.1:8080
HTTPS_PROXY http://userName:userPassword@192.168.1.1:8080
NO_PROXY localhost,127.0.0.1,.example.com,.zulipdev.com
@@ -978,7 +984,7 @@ then do a `vagrant reload`.
You can also change the port on the host machine that Vagrant uses by
adding to your `~/.zulip-vagrant-config` file. E.g. if you set:
```
```text
HOST_PORT 9971
```
@@ -989,7 +995,7 @@ If you'd like to be able to connect to your development environment from other
machines than the VM host, you can manually set the host IP address in the
'~/.zulip-vagrant-config' file as well. For example, if you set:
```
```text
HOST_IP_ADDR 0.0.0.0
```
@@ -1015,14 +1021,14 @@ more resources.
To do so, create a `~/.zulip-vagrant-config` file containing the
following lines:
```
```text
GUEST_CPUS <number of cpus>
GUEST_MEMORY_MB <system memory (in MB)>
```
For example:
```
```text
GUEST_CPUS 4
GUEST_MEMORY_MB 8192
```

View File

@@ -1,12 +1,12 @@
# Testing the installer
Zulip's install process is tested as part of [its continuous
integrations suite][CI], but that only tests the most common
integrations suite][ci], but that only tests the most common
configurations; when making changes to more complicated [installation
options][installer-docs], Zulip provides tooling to repeatedly test
the installation process in a clean environment each time.
[CI]: https://github.com/zulip/zulip/actions/workflows/production-suite.yml?query=branch%3Amaster
[ci]: https://github.com/zulip/zulip/actions/workflows/production-suite.yml?query=branch%3Amain
[installer-docs]: ../production/install.md
## Configuring
@@ -17,7 +17,8 @@ RAM, in order to accommodate the VMs and the steps which build the
release assets.
To begin, install the LXC toolchain:
```
```bash
sudo apt-get install lxc lxc-utils
```
@@ -32,7 +33,8 @@ You only need to do this step once per time you work on a set of
changes, to refresh the package that the installer uses. The installer
doesn't work cleanly out of a source checkout; it wants a release
checkout, so we build a tarball of one of those first:
```
```bash
./tools/build-release-tarball test-installer
```
@@ -46,7 +48,8 @@ directory. The test installer needs the release directory to be named
`zulip-server`, so we rename it and move it appropriately. In the
first line, you'll need to substitute the actual path that you got for
the tarball, above:
```
```bash
tar xzf /tmp/tmp.fepqqNBWxp/zulip-server-test-installer.tar.gz
mkdir zulip-test-installer
mv zulip-server-test-installer zulip-test-installer/zulip-server
@@ -65,7 +68,8 @@ into the installer.
For example, to test an install onto Ubuntu 20.04 "Focal", we might
call:
```
```bash
sudo ./tools/test-install/install \
-r focal \
./zulip-test-installer/ \
@@ -82,7 +86,8 @@ take a while.
Regardless of if the install succeeds or fails, it will stay running
so you can inspect it. You can see all of the containers which are
running, and their randomly-generated names, by running:
```
```bash
sudo lxc-ls -f
```
@@ -90,7 +95,8 @@ sudo lxc-ls -f
After using `lxc-ls` to list containers, you can choose one of them
and connect to its terminal:
```
```bash
sudo lxc-attach --clear-env -n zulip-install-focal-PUvff
```
@@ -98,24 +104,25 @@ sudo lxc-attach --clear-env -n zulip-install-focal-PUvff
To destroy all containers (but leave the base containers, which speed
up the initial install):
```
```bash
sudo ./tools/test-install/destroy-all -f
```
To destroy just one container:
```
```bash
sudo lxc-destroy -f -n zulip-install-focal-PUvff
```
### Iterating on the installer
Iterate on the installer by making changes to your source tree,
copying them into the release directory, and re-running the installer,
which will start up a new container. Here, we update just the
`scripts` and `puppet` directories of the release directory:
```
```bash
rsync -az scripts puppet zulip-test-installer/zulip-server/
sudo ./tools/test-install/install \
@@ -124,4 +131,3 @@ sudo ./tools/test-install/install \
--hostname=zulip.example.net \
--email=username@example.net
```

View File

@@ -1,5 +1,4 @@
Using the development environment
=================================
# Using the development environment
This page describes the basic edit/refresh workflows for working with
the Zulip development environment. Generally, the development
@@ -13,79 +12,80 @@ the development environment][authentication-dev-server].
## Common
* Zulip's master branch moves quickly, and you should rebase
constantly with e.g. `git fetch upstream; git rebase
upstream/master` to avoid developing on an old version of the Zulip
codebase (leading to unnecessary merge conflicts).
* Remember to run `tools/provision` to update your development
- Zulip's `main` branch moves quickly, and you should rebase
constantly with e.g.
`git fetch upstream; git rebase upstream/main` to avoid developing
on an old version of the Zulip codebase (leading to unnecessary
merge conflicts).
- Remember to run `tools/provision` to update your development
environment after switching branches; it will run in under a second
if no changes are required.
* After making changes, you'll often want to run the
- After making changes, you'll often want to run the
[linters](../testing/linters.md) and relevant [test
suites](../testing/testing.md). Consider using our [Git pre-commit
hook](../git/zulip-tools.html#set-up-git-repo-script) to
automatically lint whenever you make a commit.
* All of our test suites are designed to support quickly testing just
- All of our test suites are designed to support quickly testing just
a single file or test case, which you should take advantage of to
save time.
* Many useful development tools, including tools for rebuilding the
- Many useful development tools, including tools for rebuilding the
database with different test data, are documented in-app at
`https://localhost:9991/devtools`.
* If you want to restore your development environment's database to a
- If you want to restore your development environment's database to a
pristine state, you can use `./tools/rebuild-dev-database`.
## Server
* For changes that don't affect the database model, the Zulip
- For changes that don't affect the database model, the Zulip
development environment will automatically detect changes and
restart:
* The main Django/Tornado server processes are run on top of
- The main Django/Tornado server processes are run on top of
Django's [manage.py runserver][django-runserver], which will
automatically restart them when you save changes to Python code
they use. You can watch this happen in the `run-dev.py` console
to make sure the backend has reloaded.
* The Python queue workers will also automatically restart when you
- The Python queue workers will also automatically restart when you
save changes, as long as they haven't crashed (which can happen if
they reloaded into a version with a syntax error).
* If you change the database schema (`zerver/models.py`), you'll need
- If you change the database schema (`zerver/models.py`), you'll need
to use the [Django migrations
process](../subsystems/schema-migrations.md); see also the [new
feature tutorial][new-feature-tutorial] for an example.
* While testing server changes, it's helpful to watch the `run-dev.py`
- While testing server changes, it's helpful to watch the `run-dev.py`
console output, which will show tracebacks for any 500 errors your
Zulip development server encounters (which are probably caused by
bugs in your code).
* To manually query Zulip's database interactively, use `./manage.py
shell` or `manage.py dbshell`.
* The database(s) used for the automated tests are independent from
- To manually query Zulip's database interactively, use
`./manage.py shell` or `manage.py dbshell`.
- The database(s) used for the automated tests are independent from
the one you use for manual testing in the UI, so changes you make to
the database manually will never affect the automated tests.
## Web
* Once the development server (`run-dev.py`) is running, you can visit
- Once the development server (`run-dev.py`) is running, you can visit
<http://localhost:9991/> in your browser.
* By default, the development server homepage just shows a list of the
- By default, the development server homepage just shows a list of the
users that exist on the server and you can log in as any of them by
just clicking on a user.
* This setup saves time for the common case where you want to test
- This setup saves time for the common case where you want to test
something other than the login process.
* You can test the login or registration process by clicking the
- You can test the login or registration process by clicking the
links for the normal login page.
* Most changes will take effect automatically. Details:
* If you change CSS files, your changes will appear immediately via
- Most changes will take effect automatically. Details:
- If you change CSS files, your changes will appear immediately via
webpack hot module replacement.
* If you change JavaScript code (`static/js`) or Handlebars
- If you change JavaScript code (`static/js`) or Handlebars
templates (`static/templates`), the browser window will be
reloaded automatically.
* For Jinja2 backend templates (`templates/*`), you'll need to reload
- For Jinja2 backend templates (`templates/*`), you'll need to reload
the browser window to see your changes.
* Any JavaScript exceptions encountered while using the webapp in a
- Any JavaScript exceptions encountered while using the webapp in a
development environment will be displayed as a large notice, so you
don't need to watch the JavaScript console for exceptions.
* Both Chrome and Firefox have great debuggers, inspectors, and
- Both Chrome and Firefox have great debuggers, inspectors, and
profilers in their built-in developer tools.
* `debug.js` has some occasionally useful JavaScript profiling code.
- `debug.js` has some occasionally useful JavaScript profiling code.
## Mobile

View File

@@ -26,30 +26,30 @@ the validation Zulip has today.
Our API documentation is defined by a few sets of files:
* Most data describing API endpoints and examples is stored in our
- Most data describing API endpoints and examples is stored in our
[OpenAPI configuration](../documentation/openapi.md) at
`zerver/openapi/zulip.yaml`.
* The top-level templates live under `templates/zerver/api/*`, and are
- The top-level templates live under `templates/zerver/api/*`, and are
written using the Markdown framework that powers our [user
docs](../documentation/user.md), with some special extensions for
rendering nice code blocks and example responses. We expect to
eventually remove most of these files where it is possible to
fully generate the documentation from the OpenAPI files.
* The text for the Python examples comes from a test suite for the
- The text for the Python examples comes from a test suite for the
Python API documentation (`zerver/openapi/python_examples.py`; run via
`tools/test-api`). The `generate_code_example` macro will magically
read content from that test suite and render it as the code example.
This structure ensures that Zulip's API documentation is robust to a
wide range of possible typos and other bugs in the API
documentation.
* The JavaScript examples are similarly generated and tested using
- The JavaScript examples are similarly generated and tested using
`zerver/openapi/javascript_examples.js`.
* The cURL examples are generated and tested using
- The cURL examples are generated and tested using
`zerver/openapi/curl_param_value_generators.py`.
* The REST API index
- The REST API index
(`templates/zerver/help/include/rest-endpoints.md`) in the broader
/api left sidebar (`templates/zerver/api/sidebar_index.md`).
* We have an extensive set of tests designed to validate that the data
- We have an extensive set of tests designed to validate that the data
in this file is correct, `zerver/tests/test_openapi.py` compares
every endpoint's accepted parameters in `views` code with those
declared in `zulip.yaml`. And [backend test
@@ -74,10 +74,10 @@ We highly recommend looking at those resources while reading this page.
If you look at the documentation for existing endpoints, you'll notice
that a typical endpoint's documentation is divided into four sections:
* The top-level **Description**
* **Usage examples**
* **Arguments**
* **Responses**
- The top-level **Description**
- **Usage examples**
- **Arguments**
- **Responses**
The rest of this guide describes how each of these sections works.
@@ -101,7 +101,7 @@ defined using a special Markdown extension
(`zerver/openapi/markdown_extension.py`). To use this extension, one
writes a Markdown file block that looks something like this:
```
```md
{start_tabs}
{tab|python}
@@ -169,7 +169,7 @@ an API endpoint supports. You'll see this in files like
directive (implemented in
`zerver/lib/markdown/api_arguments_table_generator.py`):
```
```md
{generate_api_arguments_table|zulip.yaml|/messages/render:post}
```
@@ -186,7 +186,7 @@ You can use the following Markdown directive to render the fixtures
defined in the OpenAPI `zulip.yaml` for a given endpoint and status
code:
```
```md
{generate_code_example|/messages/render:post|fixture(200)}
```
@@ -215,11 +215,12 @@ above.
declared using `REQ`.
You can check your formatting using these helpful tools.
* `tools/check-openapi` will verify the syntax of `zerver/openapi/zulip.yaml`.
* `tools/test-backend zerver/tests/test_openapi.py`; this test compares
- `tools/check-openapi` will verify the syntax of `zerver/openapi/zulip.yaml`.
- `tools/test-backend zerver/tests/test_openapi.py`; this test compares
your documentation against the code and can find many common
mistakes in how arguments are declared.
* `test-backend`: The full Zulip backend test suite will fail if
- `test-backend`: The full Zulip backend test suite will fail if
any actual API responses generated by the tests don't match your
defined OpenAPI schema. Use `test-backend --rerun` for a fast
edit/refresh cycle when debugging.
@@ -237,7 +238,7 @@ above.
bindings don't have a dedicated method for a specific API call,
you may either use `client.call_endpoint` or add a dedicated
function to the [zulip PyPI
package](https://github.com/zulip/python-zulip-api/tree/master/zulip).
package](https://github.com/zulip/python-zulip-api/tree/main/zulip).
Ultimately, the goal is for every endpoint to be documented the
latter way, but it's useful to be able to write working
documentation for an endpoint that isn't supported by
@@ -285,7 +286,7 @@ above.
in `zerver/openapi/zulip.yaml`, which mentions the API feature level
at which they were added.
[javascript-examples]: https://github.com/zulip/zulip-js/tree/master/examples
[javascript-examples]: https://github.com/zulip/zulip-js/tree/main/examples
## Why a custom system?
@@ -293,14 +294,14 @@ Given that our documentation is written in large part using the
OpenAPI format, why maintain a custom Markdown system for displaying
it? There's several major benefits to this system:
* It is extremely common for API documentation to become out of date
- It is extremely common for API documentation to become out of date
as an API evolves; this automated testing system helps make it
possible for Zulip to maintain accurate documentation without a lot
of manual management.
* Every Zulip server can host correct API documentation for its
- Every Zulip server can host correct API documentation for its
version, with the key variables (like the Zulip server URL) already
pre-substituted for the user.
* We're able to share implementation language and visual styling with
- We're able to share implementation language and visual styling with
our Help Center, which is especially useful for the extensive
non-REST API documentation pages (e.g. our bot framework).

View File

@@ -10,14 +10,14 @@ integrations).
Usually, this involves a few steps:
* Add text explaining all of the steps required to set up the
- Add text explaining all of the steps required to set up the
integration, including what URLs to use, etc. See
[Writing guidelines](#writing-guidelines) for detailed writing guidelines.
Zulip's pre-defined Markdown macros can be used for some of these steps.
See [Markdown macros](#markdown-macros) for further details.
* Make sure you've added your integration to
- Make sure you've added your integration to
`zerver/lib/integrations.py` in both the `WEBHOOK_INTEGRATIONS`
section (or `INTEGRATIONS` if not a webhook), and the
`DOC_SCREENSHOT_CONFIG` sections. These registries configure your
@@ -26,7 +26,7 @@ Usually, this involves a few steps:
message (which is important for the screenshots to be updated as
Zulip's design changes).
* You'll need to add an SVG graphic
- You'll need to add an SVG graphic
of your integration's logo under the
`static/images/integrations/logos/<name>.svg`, where `<name>` is the
name of the integration, all in lower case; you can usually find them in the
@@ -40,13 +40,13 @@ Usually, this involves a few steps:
If you cannot find an SVG graphic of the logo, please find and include a PNG
image of the logo instead.
* Finally, generate a message sent by the integration and take a screenshot of
- Finally, generate a message sent by the integration and take a screenshot of
the message to provide an example message in the documentation.
If your new integration is an incoming webhook integration, you can generate
the screenshot using `tools/generate-integration-docs-screenshot`:
```sh
```bash
./tools/generate-integration-docs-screenshot --integration integrationname
```
@@ -71,18 +71,18 @@ always create a new macro by adding a new file to that folder.
Here are a few common macros used to document Zulip's integrations:
* `{!create-stream.md!}` macro - Recommends that users create a dedicated
- `{!create-stream.md!}` macro - Recommends that users create a dedicated
stream for a given integration. Usually the first step is setting up an
integration or incoming webhook. For an example rendering, see **Step 1** of
[the docs for Zulip's GitHub integration][GitHub].
[the docs for Zulip's GitHub integration][github-integration].
* `{!create-bot-construct-url-indented.md!}` macro - Instructs users to create a bot
- `{!create-bot-construct-url-indented.md!}` macro - Instructs users to create a bot
for a given integration and construct a webhook URL using the bot API key
and stream name. The URL is generated automatically for every incoming webhook
by using attributes in the `WebhookIntegration` class in
[zerver/lib/integrations.py][integrations-file].
This macro is usually used right after `{!create-stream!}`. For an example
rendering, see **Step 2** of [the docs for Zulip's GitHub integration][GitHub].
rendering, see **Step 2** of [the docs for Zulip's GitHub integration][github-integration].
**Note:** If special configuration is
required to set up the URL and you can't use this macro, be sure to use the
@@ -91,62 +91,62 @@ Here are a few common macros used to document Zulip's integrations:
deployed on. If special configuration is required to set the `SITE`
variable, you should document that too.
* `{!append-stream-name.md!}` macro - Recommends appending `&stream=stream_name`
- `{!append-stream-name.md!}` macro - Recommends appending `&stream=stream_name`
to a URL in cases where supplying a stream name in the URL is optional.
Supplying a stream name is optional for most Zulip integrations. If you use
`{!create-bot-construct-url-indented.md!}`, this macro need not be used.
* `{!append-topic.md!}` macro - Recommends appending `&topic=my_topic` to a URL
- `{!append-topic.md!}` macro - Recommends appending `&topic=my_topic` to a URL
to supply a custom topic for webhook notification messages. Supplying a custom
topic is optional for most Zulip integrations. If you use
`{!create-bot-construct-url-indented.md!}`, this macro need not be used.
* `{!congrats.md!}` macro - Inserts congratulatory lines signifying the
- `{!congrats.md!}` macro - Inserts congratulatory lines signifying the
successful setup of a given integration. This macro is usually used at
the end of the documentation, right before the sample message screenshot.
For an example rendering, see the end of
[the docs for Zulip's GitHub integration][GitHub].
[the docs for Zulip's GitHub integration][github-integration].
* `{!download-python-bindings.md!}` macro - Links to Zulip's
- `{!download-python-bindings.md!}` macro - Links to Zulip's
[API page](https://zulip.com/api/) to download and install Zulip's
API bindings. This macro is usually used in non-webhook integration docs under
`templates/zerver/integrations/<integration_name>.md`. For an example
rendering, see **Step 2** of
[the docs for Zulip's Codebase integration][codebase].
* `{!change-zulip-config-file.md!}` macro - Instructs users to create a bot and
- `{!change-zulip-config-file.md!}` macro - Instructs users to create a bot and
specify said bot's credentials in the config file for a given non-webhook
integration. This macro is usually used in non-webhook integration docs under
`templates/zerver/integrations/<integration_name>.md`. For an example
rendering, see **Step 4** of
[the docs for Zulip's Codebase integration][codebase].
* `{!git-append-branches.md!}` and `{!git-webhook-url-with-branches.md!}` -
- `{!git-append-branches.md!}` and `{!git-webhook-url-with-branches.md!}` -
These two macros explain how to specify a list of branches in the webhook URL
to filter notifications in our Git-related webhooks. For an example rendering,
see the last paragraph of **Step 2** in
[the docs for Zulip's GitHub integration][GitHub].
[the docs for Zulip's GitHub integration][github-integration].
* `{!webhook-url.md!}` - Used internally by `{!create-bot-construct-url-indented.md!}`
- `{!webhook-url.md!}` - Used internally by `{!create-bot-construct-url-indented.md!}`
to generate the webhook URL.
* `{!zulip-config.md!}` - Used internally by `{!change-zulip-config-file.md!}`
- `{!zulip-config.md!}` - Used internally by `{!change-zulip-config-file.md!}`
to specify the lines in the config file for a non-webhook integration.
* `{!webhook-url-with-bot-email.md!}` - Used in certain non-webhook integrations
- `{!webhook-url-with-bot-email.md!}` - Used in certain non-webhook integrations
to generate URLs of the form:
```
```text
https://bot_email:bot_api_key@yourZulipDomain.zulipchat.com/api/v1/external/beanstalk
```
For an example rendering, see
[Zulip's Beanstalk integration](https://zulip.com/integrations/doc/beanstalk).
[GitHub]: https://zulip.com/integrations/doc/github
[github-integration]: https://zulip.com/integrations/doc/github
[codebase]: https://zulip.com/integrations/doc/codebase
[beanstalk]: https://zulip.com/integrations/doc/beanstalk
[integrations-file]: https://github.com/zulip/zulip/blob/master/zerver/lib/integrations.py
[integrations-file]: https://github.com/zulip/zulip/blob/main/zerver/lib/integrations.py
## Writing guidelines
@@ -189,7 +189,6 @@ concrete guidelines.
- Follow the organization and wording of existing docs as much as possible.
### Guidelines for specific steps
Most doc files should start with a generic sentence about the

View File

@@ -40,7 +40,8 @@ types of authentication, and configure other settings. Once defined,
information in this section rarely changes.
For example, the `swagger` and `info` objects look like this:
```
```yaml
# Basic Swagger UI info
openapi: 3.0.1
info:
@@ -79,7 +80,7 @@ expects a GET request with one
Basic authentication, and returns a JSON response containing `msg`,
`result`, and `presence` values.
```
```yaml
/users/{user}/presence:
get:
description: Get presence data for another user.
@@ -119,7 +120,7 @@ contains schemas referenced by other objects. For example,
contains three required parameters. Two are strings, and one is an
integer.
```
```yaml
MessageResponse:
type: object
required:
@@ -144,7 +145,7 @@ You can find more examples, including GET requests and nested objects, in
We're collecting decisions we've made on how our Swagger YAML files
should be organized here:
* Use shared definitions and YAML anchors to avoid duplicating content
- Use shared definitions and YAML anchors to avoid duplicating content
where possible.
## Tips for working with YAML:
@@ -169,13 +170,13 @@ correct.
### Formatting help:
* Comments begin with a # character.
- Comments begin with a # character.
* Descriptions do not need to be in quotes, and may use common
- Descriptions do not need to be in quotes, and may use common
Markdown format options like inline code \` (backtick) and `#`
headings.
* A single `|` (pipe) character begins a multi-line description on the
- A single `|` (pipe) character begins a multi-line description on the
next line. Single spaced lines (one newline at the end of each) are
joined. Use an extra blank line for a paragraph break. We prefer
to use this format for all descriptions because it doesn't require
@@ -183,7 +184,7 @@ correct.
### Examples:
```
```yaml
Description: |
This description has multiple lines.
Sometimes descriptions can go on for

View File

@@ -2,26 +2,26 @@
Zulip has three major documentation systems:
* Developer and sysadmin documentation: Documentation for people
- Developer and sysadmin documentation: Documentation for people
actually interacting with the Zulip codebase (either by developing
it or installing it), and written in Markdown.
* Core website documentation: Complete webpages for complex topics,
- Core website documentation: Complete webpages for complex topics,
written in HTML, JavaScript, and CSS (using the Django templating
system). These roughly correspond to the documentation someone
might look at when deciding whether to use Zulip. We don't expect
to ever have more than about 10 pages written using this system.
* User-facing documentation: Our scalable system for documenting
- User-facing documentation: Our scalable system for documenting
Zulip's huge collection of specific features without a lot of
overhead or duplicated code/syntax, written in Markdown. We have
several hundred pages written using this system. There are 3
branches of this documentation:
* User documentation (with a target audience of individual Zulip
- User documentation (with a target audience of individual Zulip
users),
* Integrations documentation (with a target audience of IT folks
- Integrations documentation (with a target audience of IT folks
setting up integrations), and
* API documentation (with a target audience of developers writing
- API documentation (with a target audience of developers writing
code to extend Zulip).
These three systems are documented in detail.
@@ -43,7 +43,7 @@ your changes), the dependencies are automatically installed as part of
Zulip development environment provisioning, and you can build the
documentation using:
```
```bash
./tools/build-docs
```
@@ -123,12 +123,12 @@ details on how to contribute to this documentation.
Zulip has several automated test suites that we run in CI and
recommend running locally when making significant edits:
* `tools/lint` catches a number of common mistakes, and we highly
- `tools/lint` catches a number of common mistakes, and we highly
recommend
[using our linter pre-commit hook](../git/zulip-tools.html#set-up-git-repo-script).
See the [main linter doc](../testing/linters.md) for more details.
* The ReadTheDocs docs are built and the links tested by
- The ReadTheDocs docs are built and the links tested by
`tools/test-documentation`, which runs `build-docs` and then checks
all the links.
@@ -136,17 +136,17 @@ There's an exclude list for the link testing at this horrible path:
`tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py`,
which is relevant for flaky links.
* The API docs are tested by `tools/test-api`, which does some basic
- The API docs are tested by `tools/test-api`, which does some basic
payload verification. Note that this test does not check for broken
links (those are checked by `test-help-documentation`).
* `tools/test-help-documentation` checks `/help/`, `/api/`,
- `tools/test-help-documentation` checks `/help/`, `/api/`,
`/integrations/`, and the core website ("portico") documentation for
broken links. Note that the "portico" documentation check has a
manually maintained whitelist of pages, so if you add a new page to
this site, you will need to edit `PorticoDocumentationSpider` to add it.
* `tools/test-backend test_docs.py` tests various internal details of
- `tools/test-backend test_docs.py` tests various internal details of
the variable substitution logic, as well as rendering. It's
essential when editing the documentation framework, but not
something you'll usually need to interact with when editing

View File

@@ -8,12 +8,13 @@ There are two types of documents: articles about specific features, and a
handful of longer guides.
The feature articles serve a few different purposes:
* Feature discovery, for someone browsing the `/help` page, and looking at
- Feature discovery, for someone browsing the `/help` page, and looking at
the set of titles.
* Public documentation of our featureset, for someone googling "can zulip do .."
* Canned responses to support questions; if someone emails a Zulip admin
- Public documentation of our featureset, for someone googling "can zulip do .."
- Canned responses to support questions; if someone emails a Zulip admin
asking "how do I change my name", they can reply with a link to the doc.
* Feature explanations for new Zulip users and admins, especially for
- Feature explanations for new Zulip users and admins, especially for
organization settings.
This system is designed to make writing and maintaining such documentation
@@ -52,22 +53,22 @@ experience with.
Tips for adding a new article:
* Find an existing article in the same section of the help documentation,
- Find an existing article in the same section of the help documentation,
and copy the format, wording, style, etc as closely as you can.
* If the feature exists in other team chat products, check out their
- If the feature exists in other team chat products, check out their
documentation for inspiration.
* Fewer words is better than more. Many Zulip users have English as a second
- Fewer words is better than more. Many Zulip users have English as a second
language.
* Try to put yourself in the shoes of a new Zulip user. What would you want
- Try to put yourself in the shoes of a new Zulip user. What would you want
to know?
* The goal of user-facing documentation is not to be comprehensive. The goal
- The goal of user-facing documentation is not to be comprehensive. The goal
is to give the right bits of information for the intended audience.
* Real estate in the left sidebar is somewhat precious. Minor features
- Real estate in the left sidebar is somewhat precious. Minor features
should rarely get their own article.
An anti-pattern is trying to make up for bad UX by adding user
@@ -98,17 +99,17 @@ allows .." rather than "we also allow ..". `You` is ok and used liberally.
Zulip's Markdown processor allows you to include several special features in
your documentation to help improve its readability:
* Since raw HTML is supported in Markdown, you can include arbitrary
- Since raw HTML is supported in Markdown, you can include arbitrary
HTML/CSS in your documentation as needed.
* Code blocks allow you to highlight syntax, similar to Zulip's own Markdown.
* Anchor tags can be used to link to headers in other documents.
* [Images](#images) of Zulip UI can be added to documentation.
* Inline [icons](#icons) used to refer to features in the Zulip UI.
* You can utilize [macros](#macros) to limit repeated content in the
- Code blocks allow you to highlight syntax, similar to Zulip's own Markdown.
- Anchor tags can be used to link to headers in other documents.
- [Images](#images) of Zulip UI can be added to documentation.
- Inline [icons](#icons) used to refer to features in the Zulip UI.
- You can utilize [macros](#macros) to limit repeated content in the
documentation.
* You can create special highlight warning blocks using
- You can create special highlight warning blocks using
[tips and warnings](#tips-and-warnings).
* You can create tabs using [Markdown tab switcher](#tab-switcher).
- You can create tabs using [Markdown tab switcher](#tab-switcher).
### Images
@@ -127,7 +128,7 @@ instructions for something simple look long and complicated.
When taking screenshots, the image should never include the whole
Zulip browser window in a screenshot; instead, it should only show
relevant parts of the app. In addition, the screenshot should always
come *after* the text that describes it, never before.
come _after_ the text that describes it, never before.
Images are often a part of a numbered step and must be indented four
spaces to be formatted correctly.
@@ -141,39 +142,39 @@ base class `icon-vector` and have dropped support for it. We now only support
icons from [FontAwesome](https://fontawesome.com/v4.7.0/) (version 4.7.0) which
make use of `fa` as a base class.
* cog (<i class="fa fa-cog"></i>) icon — `cog (<i
class="fa fa-cog"></i>) icon`
* down chevron (<i class="fa fa-chevron-down"></i>) icon —
- cog (<i class="fa fa-cog"></i>) icon —
`cog (<i class="fa fa-cog"></i>) icon`
- down chevron (<i class="fa fa-chevron-down"></i>) icon —
`down chevron (<i class="fa fa-chevron-down"></i>) icon`
* eye (<i class="fa fa-eye"></i>) icon — `eye (<i
class="fa fa-eye"></i>) icon`
* file (<i class="fa fa-file-code-o"></i>) icon — `file (<i
class="fa fa-file-code-o"></i>) icon`
* filled star (<i class="fa fa-star"></i>) icon —
- eye (<i class="fa fa-eye"></i>) icon —
`eye (<i class="fa fa-eye"></i>) icon`
- file (<i class="fa fa-file-code-o"></i>) icon —
`file (<i class="fa fa-file-code-o"></i>) icon`
- filled star (<i class="fa fa-star"></i>) icon —
`filled star (<i class="fa fa-star"></i>) icon`
* formatting (<i class="fa fa-font"></i>) icon —
- formatting (<i class="fa fa-font"></i>) icon —
`formatting (<i class="fa fa-font"></i>) icon`
* menu (<i class="fa fa-bars"></i>) icon — `menu (<i
class="fa fa-bars"></i>) icon`
* overflow ( <i class="fa fa-ellipsis-v"></i> ) icon —
- menu (<i class="fa fa-bars"></i>) icon —
`menu (<i class="fa fa-bars"></i>) icon`
- overflow ( <i class="fa fa-ellipsis-v"></i> ) icon —
`overflow ( <i class="fa fa-ellipsis-v"></i> ) icon`
* paperclip (<i class="fa fa-paperclip"></i>) icon —
- paperclip (<i class="fa fa-paperclip"></i>) icon —
`paperclip (<i class="fa fa-paperclip"></i>) icon`
* pencil (<i class="fa fa-pencil"></i>) icon —
- pencil (<i class="fa fa-pencil"></i>) icon —
`pencil (<i class="fa fa-pencil"></i>) icon`
* pencil and paper (<i class="fa fa-pencil-square-o"></i>) icon —
- pencil and paper (<i class="fa fa-pencil-square-o"></i>) icon —
`pencil and paper (<i class="fa fa-pencil-square-o"></i>) icon`
* plus (<i class="fa fa-plus"></i>) icon —
- plus (<i class="fa fa-plus"></i>) icon —
`plus (<i class="fa fa-plus"></i>) icon`
* smiley face (<i class="fa fa-smile-o"></i>) icon —
- smiley face (<i class="fa fa-smile-o"></i>) icon —
`smiley face (<i class="fa fa-smile-o"></i>) icon`
* star (<i class="fa fa-star-o"></i>) icon —
- star (<i class="fa fa-star-o"></i>) icon —
`star (<i class="fa fa-star-o"></i>) icon`
* trash (<i class="fa fa-trash-o"></i>) icon —
- trash (<i class="fa fa-trash-o"></i>) icon —
`trash (<i class="fa fa-trash-o"></i>) icon`
* video-camera (<i class="fa fa-video-camera"></i>) icon —
- video-camera (<i class="fa fa-video-camera"></i>) icon —
`video-camera (<i class="fa fa-video-camera"></i>) icon`
* x (<i class="fa fa-times"></i>) icon —
- x (<i class="fa fa-times"></i>) icon —
`x (<i class="fa fa-times"></i>) icon`
### Macros
@@ -186,22 +187,22 @@ The source for macros is the Markdown files under
`templates/zerver/help/include` in the
[main Zulip server repository](https://github.com/zulip/zulip).
* **Administrator only feature** `{!admin-only.md!}`: Notes that the feature
- **Administrator only feature** `{!admin-only.md!}`: Notes that the feature
is only available to organization administrators.
* **Message actions** `{!message-actions.md!}`: First step to navigating to
- **Message actions** `{!message-actions.md!}`: First step to navigating to
the on-hover message actions.
* **Message actions menu** `{!message-actions-menu.md!}`: Navigate to the
- **Message actions menu** `{!message-actions-menu.md!}`: Navigate to the
message actions menu.
* **Save changes** `{!save-changes.md!}`: Save changes after modifying
- **Save changes** `{!save-changes.md!}`: Save changes after modifying
organization settings.
* **Stream actions** `{!stream-actions.md!}`: Navigate to the stream actions
- **Stream actions** `{!stream-actions.md!}`: Navigate to the stream actions
menu from the left sidebar.
* **Start composing** `{!start-composing.md!}`: Open the compose box.
- **Start composing** `{!start-composing.md!}`: Open the compose box.
### Tips and warnings
@@ -210,7 +211,7 @@ instructions. For instance, it may address a common problem users may
encounter while following the instructions, or point to an option for power
users.
```
```md
!!! tip ""
If you've forgotten your password, see the
[Change your password](/help/change-your-password) page for
@@ -220,7 +221,7 @@ users.
A **warning** is a note on what happens when there is some kind of problem.
Tips are more common than warnings.
```
```md
!!! warn ""
**Note:** If you attempt to input a nonexistent stream name, an error
message will appear.
@@ -237,6 +238,7 @@ design to easily show the instructions for different
[platforms](https://zulip.com/help/logging-out) in user docs,
languages in API docs, etc. To create a tab switcher, write:
```md
{start_tabs}
{tab|desktop-web}
# First tab's content
@@ -245,12 +247,13 @@ languages in API docs, etc. To create a tab switcher, write:
{tab|android}
# Third tab's content
{end_tabs}
```
The tab identifiers (e.g. `desktop-web` above) and their mappings to
the tabs' labels are declared in
[zerver/lib/markdown/tabbed_sections.py][tabbed-sections-code].
[tabbed-sections-code]: https://github.com/zulip/zulip/blob/master/zerver/lib/markdown/tabbed_sections.py
[tabbed-sections-code]: https://github.com/zulip/zulip/blob/main/zerver/lib/markdown/tabbed_sections.py
This widget can also be used just to create a nice box around a set of
instructions

View File

@@ -8,7 +8,7 @@ See also [fixing commits][fix-commit]
- `git add foo.py`
- checkout
- `git checkout -b new-branch-name`
- `git checkout master`
- `git checkout main`
- `git checkout old-branch-name`
- commit
- `git commit -m "topic: Commit message title."`
@@ -36,8 +36,8 @@ See also [fixing commits][fix-commit]
- `git push origin +branch-name`
- rebase
- `git rebase -i HEAD~3`
- `git rebase -i master`
- `git rebase upstream/master`
- `git rebase -i main`
- `git rebase upstream/main`
- reflog
- `git reflog | head -10`
- remote
@@ -49,7 +49,7 @@ See also [fixing commits][fix-commit]
- show
- `git show HEAD`
- `git show HEAD~~~`
- `git show master`
- `git show main`
- status
- `git status`
@@ -61,7 +61,7 @@ See also [fixing commits][fix-commit]
- `git add -u`: Adds all tracked files to the staging area.
- checkout
- `git checkout -b new-branch-name`: create branch `new-branch-name` and switch to/check out that new branch
- `git checkout master`: switch to your `master` branch
- `git checkout main`: switch to your `main` branch
- `git checkout old-branch-name`: switch to an existing branch `old-branch-name`
- commit
- `git commit -m "commit message"`: It is recommended to type a
@@ -84,18 +84,18 @@ See also [fixing commits][fix-commit]
- `git log`: show commit logs
- `git log --oneline | head`: To quickly see the latest ten commits on a branch.
- pull
- `git pull --rebase`: rebase your changes on top of master.
- `git pull --rebase`: rebase your changes on top of `main`.
- `git pull` (with no options): Will either create a merge commit
(which you don't want) or do the same thing as `git pull --rebase`,
depending on [whether you've configured Git properly][git-config-clone]
- push
- `git push origin branch-name`: push you commits to the origin repository *only if* there are no conflicts.
- `git push origin branch-name`: push you commits to the origin repository _only if_ there are no conflicts.
Use this when collaborating with others to prevent overwriting their work.
- `git push origin +branch-name`: force push your commits to your origin repository.
- rebase
- `git rebase -i HEAD~3`: interactive rebasing current branch with first three items on HEAD
- `git rebase -i master`: interactive rebasing current branch with master branch
- `git rebase upstream/master`: rebasing current branch with master branch from upstream repository
- `git rebase -i main`: interactive rebasing current branch with `main` branch
- `git rebase upstream/main`: rebasing current branch with `main` branch from upstream repository
- reflog
- `git reflog | head -10`: manage reference logs for the past 10 commits
- remote
@@ -107,7 +107,7 @@ See also [fixing commits][fix-commit]
- show
- `git show HEAD`: display most recent commit
- `git show HEAD~~~`: display third most recent commit
- `git show master`: display most recent commit on `master`
- `git show main`: display most recent commit on `main`
- status
- `git status`: show the working tree status, unstaged and staged files

View File

@@ -20,7 +20,7 @@ the main server app, this is [zulip/zulip][github-zulip-zulip].
Next, clone your fork to your local machine:
```
```console
$ git clone --config pull.rebase https://github.com/YOUR_USERNAME/zulip.git
Cloning into 'zulip'
remote: Counting objects: 86768, done.
@@ -32,12 +32,12 @@ Checking connectivity... done.
```
(The `--config pull.rebase` option configures Git so that `git pull`
will behave like `git pull --rebase` by default. Using `git pull
--rebase` to update your changes to resolve merge conflicts is
expected by essentially all of open source projects, including Zulip.
You can also set that option after cloning using `git config --add
pull.rebase true`, or just be careful to always run `git pull
--rebase`, never `git pull`).
will behave like `git pull --rebase` by default. Using
`git pull --rebase` to update your changes to resolve merge conflicts
is expected by essentially all of open source projects, including
Zulip. You can also set that option after cloning using
`git config --add pull.rebase true`, or just be careful to always run
`git pull --rebase`, never `git pull`).
Note: If you receive an error while cloning, you may not have [added your ssh
key to GitHub][github-help-add-ssh-key].
@@ -56,7 +56,7 @@ your fork.
First, show the currently configured remote repository:
```
```console
$ git remote -v
origin git@github.com:YOUR_USERNAME/zulip.git (fetch)
origin git@github.com:YOUR_USERNAME/zulip.git (push)
@@ -65,10 +65,10 @@ origin git@github.com:YOUR_USERNAME/zulip.git (push)
Note: If you've cloned the repository using a graphical client, you may already
have the upstream remote repository configured. For example, when you clone
[zulip/zulip][github-zulip-zulip] with the GitHub desktop client it configures
the remote repository `zulip` and you see the following output from `git remote
-v`:
the remote repository `zulip` and you see the following output from
`git remote -v`:
```
```console
origin git@github.com:YOUR_USERNAME/zulip.git (fetch)
origin git@github.com:YOUR_USERNAME/zulip.git (push)
zulip https://github.com/zulip/zulip.git (fetch)
@@ -78,13 +78,13 @@ zulip https://github.com/zulip/zulip.git (push)
If your client hasn't automatically configured a remote for zulip/zulip, you'll
need to with:
```
```console
$ git remote add -f upstream https://github.com/zulip/zulip.git
```
Finally, confirm that the new remote repository, upstream, has been configured:
```
```console
$ git remote -v
origin git@github.com:YOUR_USERNAME/zulip.git (fetch)
origin git@github.com:YOUR_USERNAME/zulip.git (push)

View File

@@ -6,7 +6,7 @@ What happens when you would like to collaborate with another contributor and
they have work-in-progress on their own fork of Zulip? No problem! Just add
their fork as a remote and pull their changes.
```
```console
$ git remote add <username> https://github.com/<username>/zulip.git
$ git fetch <username>
```
@@ -15,12 +15,13 @@ Now you can check out their branch just like you would any other. You can name
the branch anything you want, but using both the username and branch name will
help you keep things organized.
```
```console
$ git checkout -b <username>/<branchname>
```
You can choose to rename the branch if you prefer:
```
```bash
git checkout -b <custombranchname> <username>/<branchname>
```
@@ -31,27 +32,28 @@ pull request locally. GitHub provides a special syntax
([details][github-help-co-pr-locally]) for this since pull requests are
specific to GitHub rather than Git.
First, fetch and create a branch for the pull request, replacing *ID* and
*BRANCHNAME* with the ID of the pull request and your desired branch name:
First, fetch and create a branch for the pull request, replacing _ID_ and
_BRANCHNAME_ with the ID of the pull request and your desired branch name:
```
```console
$ git fetch upstream pull/ID/head:BRANCHNAME
```
Now switch to the branch:
```
```console
$ git checkout BRANCHNAME
```
Now you work on this branch as you would any other.
Note: you can use the scripts provided in the tools/ directory to fetch pull
requests. You can read more about what they do [here][tools-PR].
```
requests. You can read more about what they do [here][tools-pr].
```bash
tools/fetch-rebase-pull-request <PR-number>
tools/fetch-pull-request <PR-number>
```
[github-help-co-pr-locally]: https://help.github.com/en/articles/checking-out-pull-requests-locally
[tools-PR]: ../git/zulip-tools.html#fetch-a-pull-request-and-rebase
[tools-pr]: ../git/zulip-tools.html#fetch-a-pull-request-and-rebase

View File

@@ -1,35 +1,45 @@
# Fixing commits
This is mostly from
[here](https://help.github.com/en/articles/changing-a-commit-message#rewriting-the-most-recent-commit-message).
## Fixing the last commit
### Changing the last commit message
1. `git commit --amend -m "New message"`
### Changing the last commit
1. Make your changes to the files
2. Run `git add <filename>` to add one file or `git add <filename1> <filename2> ...` to add multiple files
3. `git commit --amend`
## Fixing older commits
### Changing commit messages
1. `git rebase -i HEAD~5` (if, for example, you are editing some of the last five commits)
2. For each commit that you want to change the message, change `pick` to `reword`, and save
3. Change the commit messages
### Deleting old commits
1. `git rebase -i HEAD~n` where `n` is the number of commits you are looking at
2. For each commit that you want to delete, change `pick` to `drop`, and save
## Squashing commits
Sometimes, you want to make one commit out of a bunch of commits. To do this,
1. `git rebase -i HEAD~n` where `n` is the number of commits you are interested in
2. Change `pick` to `squash` on the lines containing the commits you want to squash and save
## Reordering commits
1. `git rebase -i HEAD~n` where `n` is the number of commits you are interested in
2. Reorder the lines containing the commits and save
## Pushing commits after tidying them
1. `git push origin +my-feature-branch` (Note the `+` there and substitute your actual branch name.)

View File

@@ -10,7 +10,7 @@ with these details in mind:
[repository][github-zulip], if you are working on something else besides
Zulip server) to your own account and then create feature/issue branches.
When you're ready to get feedback, submit a work-in-progress (WIP) pull
request. *We encourage you to submit WIP pull requests early and often.*
request. _We encourage you to submit WIP pull requests early and often._
- We use a **[rebase][gitbook-rebase]-oriented workflow.** We do not use merge
commits. This means you should use `git fetch` followed by `git rebase`
@@ -24,13 +24,13 @@ with these details in mind:
when another branch is merged, that clutter the commit history (it's
popular with other large projects such as Django). This makes
Zulip's commit history more readable, but a side effect is that many
pull requests we merge will be reported by GitHub's UI as *closed*
instead of *merged*, since GitHub has poor support for
pull requests we merge will be reported by GitHub's UI as _closed_
instead of _merged_, since GitHub has poor support for
rebase-oriented workflows.
- We have a **[code style guide][zulip-rtd-code-style]**, a **[commit message
guide][zulip-rtd-commit-messages]**, and strive for each commit to be *a
minimal coherent idea* (see **[commit
guide][zulip-rtd-commit-messages]**, and strive for each commit to be _a
minimal coherent idea_ (see **[commit
discipline][zulip-rtd-commit-discipline]** for details).
- We provide **many tools to help you submit quality code.** These include
@@ -48,7 +48,7 @@ with these details in mind:
Finally, install the [Zulip developer environment][zulip-rtd-dev-overview], and then
[configure continuous integration for your fork][zulip-git-guide-fork-ci].
***
---
The following sections will help you be awesome with Zulip and Git/GitHub in a
rebased-based workflow. Read through it if you're new to Git, to a rebase-based

View File

@@ -3,7 +3,7 @@
When you're ready for feedback, submit a pull request. Pull requests
are a feature specific to GitHub. They provide a simple, web-based way
to submit your work (often called "patches") to a project. It's called
a *pull request* because you're asking the project to *pull changes*
a _pull request_ because you're asking the project to _pull changes_
from your fork.
If you're unfamiliar with how to create a pull request, you can check
@@ -26,7 +26,7 @@ work from being merged before you're confident in it.
## Create a pull request
### Step 0: Make sure you're on a feature branch (not `master`)
### Step 0: Make sure you're on a feature branch (not `main`)
It is important to [work on a feature
branch](using.html#work-on-a-feature-branch) when creating a pull
@@ -35,7 +35,7 @@ branch while it is open, so you will need to reserve your branch only
for changes related to your issue, and avoid introducing extraneous
changes for other issues or from upstream.
If you are working on a branch named `master`, you need to create and
If you are working on a branch named `main`, you need to create and
switch to a feature branch before proceeding.
### Step 1: Update your branch with git rebase
@@ -44,9 +44,9 @@ The best way to update your branch is with `git fetch` and `git rebase`. Do not
use `git pull` or `git merge` as this will create merge commits. See [keep your
fork up to date][keep-up-to-date] for details.
Here's an example (you would replace *issue-123* with the name of your feature branch):
Here's an example (you would replace _issue-123_ with the name of your feature branch):
```
```console
$ git checkout issue-123
Switched to branch 'issue-123'
@@ -56,9 +56,9 @@ remote: Compressing objects: 100% (23/23), done.
remote: Total 69 (delta 49), reused 39 (delta 39), pack-reused 7
Unpacking objects: 100% (69/69), done.
From https://github.com/zulip/zulip
69fa600..43e21f6 master -> upstream/master
69fa600..43e21f6 main -> upstream/main
$ git rebase upstream/master
$ git rebase upstream/main
First, rewinding head to replay your work on top of it...
Applying: troubleshooting tip about provisioning
@@ -68,7 +68,7 @@ Applying: troubleshooting tip about provisioning
Once you've updated your local feature branch, push the changes to GitHub:
```
```console
$ git push origin issue-123
Counting objects: 6, done.
Delta compression using up to 4 threads.
@@ -83,7 +83,7 @@ To git@github.com:christi3k/zulip.git
If your push is rejected with error **failed to push some refs** then you need
to prefix the name of your branch with a `+`:
```
```console
$ git push origin +issue-123
Counting objects: 6, done.
Delta compression using up to 4 threads.
@@ -117,7 +117,7 @@ pull request** button.
Alternatively, if you've recently pushed to your fork, you will see a green
**Compare & pull request** button.
You'll see the *Open a pull request* page:
You'll see the _Open a pull request_ page:
![images-create-pr]

View File

@@ -8,20 +8,20 @@ on reviewing changes by other contributors.
Display changes between index and working tree (what is not yet staged for commit):
```
```console
$ git diff
```
Display changes between index and last commit (what you have staged for commit):
```
```console
$ git diff --cached
```
Display changes in working tree since last commit (changes that are staged as
well as ones that are not):
```
```console
$ git diff HEAD
```
@@ -31,34 +31,34 @@ Use any git-ref to compare changes between two commits on the current branch.
Display changes between commit before last and last commit:
```
```console
$ git diff HEAD^ HEAD
```
Display changes between two commits using their hashes:
```
```console
$ git diff e2f404c 7977169
```
## Changes between branches
Display changes between tip of topic branch and tip of master branch:
Display changes between tip of `topic` branch and tip of `main` branch:
```
$ git diff topic master
```console
$ git diff topic main
```
Display changes that have occurred on master branch since topic branch was created:
Display changes that have occurred on `main` branch since `topic` branch was created:
```
$ git diff topic...master
```console
$ git diff topic...main
```
Display changes you've committed so far since creating a branch from upstream/master:
Display changes you've committed so far since creating a branch from `upstream/main`:
```
$ git diff upstream/master...HEAD
```console
$ git diff upstream/main...HEAD
```
[zulip-rtd-review]: ../contributing/code-reviewing.md

View File

@@ -2,9 +2,10 @@
When you install Git, it adds a manual entry for `gitglossary`. You can view
this glossary by running `man gitglossary`. Below we've included the Git terms
you'll encounter most often along with their definitions from *gitglossary*.
you'll encounter most often along with their definitions from _gitglossary_.
## branch
A "branch" is an active line of development. The most recent commit
on a branch is referred to as the tip of that branch. The tip of
the branch is referenced by a branch head, which moves forward as
@@ -14,14 +15,17 @@ working tree is associated with just one of them (the "current" or
"checked out" branch), and HEAD points to that branch.
## cache
Obsolete for: index
## checkout
The action of updating all or part of the working tree with a tree
object or blob from the object database, and updating the index and
HEAD if the whole working tree has been pointed at a new branch.
## commit
As a noun: A single point in the Git history; the entire history of
a project is represented as a set of interrelated commits. The word
"commit" is often used by Git in the same places other revision
@@ -33,6 +37,7 @@ state in the Git history, by creating a new commit representing the
current state of the index and advancing HEAD to point at the new
## fast-forward
A fast-forward is a special type of merge where you have a revision
and you are "merging" another branch's changes that happen to be a
descendant of what you have. In such these cases, you do not make a
@@ -41,19 +46,23 @@ happen frequently on a remote-tracking branch of a remote
repository.
## fetch
Fetching a branch means to get the branch's head ref from a remote
repository, to find out which objects are missing from the local
object database, and to get them, too. See also [git-fetch(1)](https://git-scm.com/docs/git-fetch)
## hash
In Git's context, synonym for object name.
## head
A named reference to the commit at the tip of a branch. Heads are
stored in a file in $GIT_DIR/refs/heads/ directory, except when
using packed refs. See also [git-pack-refs(1)](https://git-scm.com/docs/git-pack-refs).
## HEAD
The current branch. In more detail: Your working tree is normally
derived from the state of the tree referred to by HEAD. HEAD is a
reference to one of the heads in your repository, except when using
@@ -61,15 +70,18 @@ a detached HEAD, in which case it directly references an arbitrary
commit.
## index
A collection of files with stat information, whose contents are
stored as objects. The index is a stored version of your working
tree. Truth be told, it can also contain a second, and even a third
version of a working tree, which are used when merging.
## pull
Pulling a branch means to fetch it and merge it. See also [git-pull(1)](https://git-scm.com/docs/git-pull)
## push
Pushing a branch means to get the branch's head ref from a remote
repository, find out if it is a direct ancestor to the branch's
local head ref, and in that case, putting all objects, which are
@@ -79,5 +91,6 @@ the remote head ref. If the remote head is not an ancestor to the
local head, the push fails.
## rebase
To reapply a series of changes from a branch to a different base,
and reset the head of that branch to the result.

View File

@@ -2,22 +2,22 @@
Whether you're new to Git or have experience with another version control
system (VCS), it's a good idea to learn a bit about how Git works. We recommend
this excellent presentation *[Understanding Git][understanding-git]* from
this excellent presentation _[Understanding Git][understanding-git]_ from
Nelson Elhage and Anders Kaseorg and the [Git Basics][gitbook-basics] chapter
from *Pro Git* by Scott Chacon and Ben Straub.
from _Pro Git_ by Scott Chacon and Ben Straub.
Here are the top things to know:
- **Git works on snapshots.** Unlike other version control systems (e.g.,
Subversion, Perforce, Bazaar), which track files and changes to those files
made over time, Git tracks *snapshots* of your project. Each time you commit
made over time, Git tracks _snapshots_ of your project. Each time you commit
or otherwise make a change to your repository, Git takes a snapshot of your
project and stores a reference to that snapshot. If a file hasn't changed,
Git creates a link to the identical file rather than storing it again.
- **Most Git operations are local.** Git is a distributed version control
system, so once you've cloned a repository, you have a complete copy of that
repository's *entire history*. Staging, committing, branching, and browsing
repository's _entire history_. Staging, committing, branching, and browsing
history are all things you can do locally without network access and without
immediately affecting any remote repositories. To make or receive changes
from remote repositories, you need to `git fetch`, `git pull`, or `git push`.
@@ -45,9 +45,9 @@ Here are the top things to know:
- **Cloning a repository creates a working copy.** Every working copy has a
`.git` subdirectory, which contains its own Git repository. The `.git`
subdirectory also tracks the *index*, a staging area for changes that will
become part of the next commit. All files outside of `.git` is the *working
tree*.
subdirectory also tracks the _index_, a staging area for changes that will
become part of the next commit. All files outside of `.git` is the _working
tree_.
- **Files tracked with Git have possible three states: committed, modified, and
staged.** Committed files are those safely stored in your local `.git`
@@ -56,8 +56,8 @@ Here are the top things to know:
changes but have not yet been marked for inclusion in the next commit; they
have not been added to the index.
- **Git commit workflow is as follows.** Edit files in your *working tree*. Add
to the *index* (that is *stage*) with `git add`. *Commit* to the HEAD of the
- **Git commit workflow is as follows.** Edit files in your _working tree_. Add
to the _index_ (that is _stage_) with `git add`. _Commit_ to the HEAD of the
current branch with `git commit`.
[gitbook-basics]: https://git-scm.com/book/en/v2/Getting-Started-Git-Basics

View File

@@ -26,8 +26,8 @@ A merge commit is usually created when you've run `git pull` or `git merge`.
You'll know you're creating a merge commit if you're prompted for a commit
message and the default is something like this:
```
Merge branch 'master' of https://github.com/zulip/zulip
```text
Merge branch 'main' of https://github.com/zulip/zulip
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
@@ -38,13 +38,13 @@ Merge branch 'master' of https://github.com/zulip/zulip
And the first entry for `git log` will show something like:
```
```console
commit e5f8211a565a5a5448b93e98ed56415255546f94
Merge: 13bea0e e0c10ed
Author: Christie Koehler <ck@christi3k.net>
Date: Mon Oct 10 13:25:51 2016 -0700
Merge branch 'master' of https://github.com/zulip/zulip
Merge branch 'main' of https://github.com/zulip/zulip
```
Some graphical Git clients may also create merge commits.
@@ -52,10 +52,10 @@ Some graphical Git clients may also create merge commits.
To undo a merge commit, first run `git reflog` to identify the commit you want
to roll back to:
```
```console
$ git reflog
e5f8211 HEAD@{0}: pull upstream master: Merge made by the 'recursive' strategy.
e5f8211 HEAD@{0}: pull upstream main: Merge made by the 'recursive' strategy.
13bea0e HEAD@{1}: commit: test commit for docs.
```
@@ -67,19 +67,18 @@ by `git pull` and `13bea0e HEAD@{1}:` is the last commit I made before running
Once you'd identified the ref you want to revert to, you can do so with [git
reset][gitbook-reset]:
```
```console
$ git reset --hard 13bea0e
HEAD is now at 13bea0e test commit for docs.
```
```eval_rst
.. important::
``git reset --hard <commit>`` will discard all changes in your
:::{important}
`git reset --hard <commit>` will discard all changes in your
working directory and index since the commit you're resetting to with
``<commit>``. *This is the main way you can lose work in Git*. If you need
`<commit>`. _This is the main way you can lose work in Git_. If you need
to keep any changes that are in your working directory or that you have
committed, use ``git reset --merge <commit>`` instead.
```
committed, use `git reset --merge <commit>` instead.
:::
You can also use the relative reflog `HEAD@{1}` instead of the commit hash,
just keep in mind that this changes as you run git commands.
@@ -87,17 +86,17 @@ just keep in mind that this changes as you run git commands.
Now when you look at the output of `git reflog`, you should see that the tip of your branch points to your
last commit `13bea0e` before the merge:
```
```console
$ git reflog
13bea0e HEAD@{2}: reset: moving to HEAD@{1}
e5f8211 HEAD@{3}: pull upstream master: Merge made by the 'recursive' strategy.
e5f8211 HEAD@{3}: pull upstream main: Merge made by the 'recursive' strategy.
13bea0e HEAD@{4}: commit: test commit for docs.
```
And the first entry `git log` shows is this:
```
```console
commit 13bea0e40197b1670e927a9eb05aaf50df9e8277
Author: Christie Koehler <ck@christi3k.net>
Date: Mon Oct 10 13:25:38 2016 -0700
@@ -115,32 +114,32 @@ with `git cherry-pick` ([docs][gitbook-git-cherry-pick]).
For example, let's say you just committed "some work" and your `git log` looks
like this:
```
* 67aea58 (HEAD -> master) some work
```console
* 67aea58 (HEAD -> main) some work
* 13bea0e test commit for docs.
```
You then mistakenly run `git reset --hard 13bea0e`:
```
```console
$ git reset --hard 13bea0e
HEAD is now at 13bea0e test commit for docs.
$ git log
* 13bea0e (HEAD -> master) test commit for docs.
* 13bea0e (HEAD -> main) test commit for docs.
```
And then realize you actually needed to keep commit 67aea58. First, use `git
reflog` to confirm that commit you want to restore and then run `git
cherry-pick <commit>`:
And then realize you actually needed to keep commit 67aea58. First, use
`git reflog` to confirm that commit you want to restore and then run
`git cherry-pick <commit>`:
```
```console
$ git reflog
13bea0e HEAD@{0}: reset: moving to 13bea0e
67aea58 HEAD@{1}: commit: some work
$ git cherry-pick 67aea58
[master 67aea58] some work
[main 67aea58] some work
Date: Thu Oct 13 11:51:19 2016 -0700
1 file changed, 1 insertion(+)
create mode 100644 test4.txt
@@ -154,13 +153,13 @@ which ever branch you are rebasing on top of, is to code that has been changed
by those new commits.
For example, while I'm working on a file, another contributor makes a change to
that file, submits a pull request and has their code merged into master.
that file, submits a pull request and has their code merged into `main`.
Usually this is not a problem, but in this case the other contributor made a
change to a part of the file I also want to change. When I try to bring my
branch up to date with `git fetch` and then `git rebase upstream/master`, I see
branch up to date with `git fetch` and then `git rebase upstream/main`, I see
the following:
```
```console
First, rewinding head to replay your work on top of it...
Applying: test change for docs
Using index info to reconstruct a base tree...
@@ -178,11 +177,11 @@ To check out the original branch and stop rebasing, run "git rebase --abort".
```
This message tells me that Git was not able to apply my changes to README.md
after bringing in the new commits from upstream/master.
after bringing in the new commits from upstream/main.
Running `git status` also gives me some information:
```
```console
rebase in progress; onto 5ae56e6
You are currently rebasing branch 'docs-test' on '5ae56e6'.
(fix conflicts and then run "git rebase --continue")
@@ -204,10 +203,12 @@ and `>>>>>>>`) markers to indicate where in files there are conflicts.
Tip: You can see recent changes made to a file by running the following
commands:
```
```bash
git fetch upstream
git log -p upstream/master -- /path/to/file
git log -p upstream/main -- /path/to/file
```
You can use this to compare the changes that you have made to a file with the
ones in upstream, helping you avoid undoing changes from a previous commit when
you are rebasing.
@@ -215,7 +216,7 @@ you are rebasing.
Once you've done that, save the file(s), stage them with `git add` and then
continue the rebase with `git rebase --continue`:
```
```console
$ git add README.md
$ git rebase --continue
@@ -234,14 +235,14 @@ pay attention and do a bit of work to ensure all of your work is readily
available.
Recall that most Git operations are local. When you commit your changes with
`git commit` they are safely stored in your *local* Git database only. That is,
until you *push* the commits to GitHub, they are only available on the computer
`git commit` they are safely stored in your _local_ Git database only. That is,
until you _push_ the commits to GitHub, they are only available on the computer
where you committed them.
So, before you stop working for the day, or before you switch computers, push
all of your commits to GitHub with `git push`:
```
```console
$ git push origin <branchname>
```
@@ -254,7 +255,7 @@ But if you're switching to another computer on which you have already cloned
Zulip, you need to update your local Git database with new refs from your
GitHub fork. You do this with `git fetch`:
```
```console
$ git fetch <usermame>
```
@@ -262,11 +263,11 @@ Ideally you should do this before you have made any commits on the same branch
on the second computer. Then you can `git merge` on whichever branch you need
to update:
```
```console
$ git checkout <my-branch>
Switched to branch '<my-branch>'
$ git merge origin/master
$ git merge origin/main
```
**If you have already made commits on the second computer that you need to

View File

@@ -8,7 +8,7 @@ determine the currently checked out branch several ways.
One way is with [git status][gitbook-git-status]:
```
```console
$ git status
On branch issue-demo
nothing to commit, working directory clean
@@ -17,23 +17,23 @@ nothing to commit, working directory clean
Another is with [git branch][gitbook-git-branch] which will display all local
branches, with a star next to the current branch:
```
```console
$ git branch
* issue-demo
master
main
```
To see even more information about your branches, including remote branches,
use `git branch -vva`:
```
```console
$ git branch -vva
* issue-123 517468b troubleshooting tip about provisioning
master f0eaee6 [origin/master] bug: Fix traceback in get_missed_message_token_from_address().
remotes/origin/HEAD -> origin/master
main f0eaee6 [origin/main] bug: Fix traceback in get_missed_message_token_from_address().
remotes/origin/HEAD -> origin/main
remotes/origin/issue-1234 4aeccb7 Another test commit, with longer message.
remotes/origin/master f0eaee6 bug: Fix traceback in get_missed_message_token_from_address().
remotes/upstream/master dbeab6a Optimize checks of test database state by moving into Python.
remotes/origin/main f0eaee6 bug: Fix traceback in get_missed_message_token_from_address().
remotes/upstream/main dbeab6a Optimize checks of test database state by moving into Python.
```
You can also configure [Bash][gitbook-other-envs-bash] and
@@ -46,48 +46,48 @@ from Zulip's main repositories.
**Note about git pull**: You might be used to using `git pull` on other
projects. With Zulip, because we don't use merge commits, you'll want to avoid
it. Rather than using `git pull`, which by default is a shortcut for `git fetch
&& git merge FETCH_HEAD` ([docs][gitbook-git-pull]), you should use `git fetch`
and then `git rebase`.
it. Rather than using `git pull`, which by default is a shortcut for
`git fetch && git merge FETCH_HEAD` ([docs][gitbook-git-pull]), you
should use `git fetch` and then `git rebase`.
First, [fetch][gitbook-fetch] changes from Zulip's upstream repository you
configured in the step above:
```
```console
$ git fetch upstream
```
Next, check out your `master` branch and [rebase][gitbook-git-rebase] it on top
of `upstream/master`:
Next, check out your `main` branch and [rebase][gitbook-git-rebase] it on top
of `upstream/main`:
```
$ git checkout master
Switched to branch 'master'
```console
$ git checkout main
Switched to branch 'main'
$ git rebase upstream/master
$ git rebase upstream/main
```
This will rollback any changes you've made to master, update it from
`upstream/master`, and then re-apply your changes. Rebasing keeps the commit
This will rollback any changes you've made to `main`, update it from
`upstream/main`, and then re-apply your changes. Rebasing keeps the commit
history clean and readable.
When you're ready, [push your changes][github-help-push] to your remote fork.
Make sure you're in branch `master` and then run `git push`:
Make sure you're in branch `main` and then run `git push`:
```
$ git checkout master
$ git push origin master
```console
$ git checkout main
$ git push origin main
```
You can keep any branch up to date using this method. If you're working on a
feature branch (see next section), which we recommend, you would change the
command slightly, using the name of your `feature-branch` rather than `master`:
command slightly, using the name of your `feature-branch` rather than `main`:
```
```console
$ git checkout feature-branch
Switched to branch 'feature-branch'
$ git rebase upstream/master
$ git rebase upstream/main
$ git push origin feature-branch
```
@@ -99,25 +99,25 @@ feature. Recall from [how Git is different][how-git-is-different] that
**Git is designed for lightweight branching and merging.** You can and should
create as many branches as you'd like.
First, make sure your master branch is up-to-date with Zulip upstream ([see
First, make sure your `main` branch is up-to-date with Zulip upstream ([see
how][zulip-git-guide-up-to-date]).
Next, from your master branch, create a new tracking branch, providing a
Next, from your `main` branch, create a new tracking branch, providing a
descriptive name for your feature branch:
```
$ git checkout master
Switched to branch 'master'
```console
$ git checkout main
Switched to branch 'main'
$ git checkout -b issue-1755-fail2ban
Switched to a new branch 'issue-1755-fail2ban'
```
Alternatively, you can create a new branch explicitly based off
`upstream/master`:
`upstream/main`:
```
$ git checkout -b issue-1755-fail2ban upstream/master
```console
$ git checkout -b issue-1755-fail2ban upstream/main
Switched to a new branch 'issue-1755-fail2ban'
```
@@ -135,7 +135,7 @@ Recall that files tracked with Git have possible three states:
committed, modified, and staged.
To prepare a commit, first add the files with changes that you want
to include in your commit to your staging area. You *add* both new files and
to include in your commit to your staging area. You _add_ both new files and
existing ones. You can also remove files from staging when necessary.
### Get status of working directory
@@ -146,7 +146,7 @@ staged, use `git status`.
If you have no changes in the working directory, you'll see something like
this:
```
```console
$ git status
On branch issue-123
nothing to commit, working directory clean
@@ -154,7 +154,7 @@ nothing to commit, working directory clean
If you have unstaged changes, you'll see something like this:
```
```console
On branch issue-123
Untracked files:
(use "git add <file>..." to include in what will be committed)
@@ -166,14 +166,15 @@ nothing added to commit but untracked files present (use "git add" to track)
### Stage additions with git add
To add changes to your staging area, use `git add <filename>`. Because `git
add` is all about staging the changes you want to commit, you use it to add
*new files* as well as *files with changes* to your staging area.
To add changes to your staging area, use `git add <filename>`. Because
`git add` is all about staging the changes you want to commit, you use
it to add _new files_ as well as _files with changes_ to your staging
area.
Continuing our example from above, after we run `git add newfile.py`, we'll see
the following from `git status`:
```
```console
On branch issue-123
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
@@ -187,13 +188,12 @@ view changes to files you haven't yet staged, just use `git diff`.
If you want to add all changes in the working directory, use `git add -A`
([documentation][gitbook-add]).
You can also stage changes using your graphical Git client.
If you stage a file, you can undo it with `git reset HEAD <filename>`. Here's
an example where we stage a file `test3.txt` and then unstage it:
```
```console
$ git add test3.txt
On branch issue-1234
Changes to be committed:
@@ -222,7 +222,7 @@ stage the file for deletion and leave it in your working directory.
To stage a file for deletion and **remove** it from your working directory, use
`git rm <filename>`:
```
```console
$ git rm test.txt
rm 'test.txt'
@@ -240,7 +240,7 @@ ls: No such file or directory
To stage a file for deletion and **keep** it in your working directory, use
`git rm --cached <filename>`:
```
```console
$ git rm --cached test2.txt
rm 'test2.txt'
@@ -258,7 +258,7 @@ test2.txt
If you stage a file for deletion with the `--cached` option, and haven't yet
run `git commit`, you can undo it with `git reset HEAD <filename>`:
```
```console
$ git reset HEAD test2.txt
```
@@ -273,7 +273,7 @@ with `git commit -m "My commit message."` to include a commit message.
Here's an example of committing with the `-m` for a one-line commit message:
```
```console
$ git commit -m "Add a test commit for docs."
[issue-123 173e17a] Add a test commit for docs.
1 file changed, 1 insertion(+)
@@ -295,7 +295,7 @@ messages][zulip-rtd-commit-messages] for details.
Here's an example of a longer commit message that will be used for a pull request:
```
```text
Integrate Fail2Ban.
Updates Zulip logging to put an unambiguous entry into the logs such
@@ -317,7 +317,7 @@ testing in a more production-like environment.
The final paragraph indicates that this commit addresses and fixes issue #1755.
When you submit your pull request, GitHub will detect and link this reference
to the appropriate issue. Once your commit is merged into zulip/master, GitHub
to the appropriate issue. Once your commit is merged into `upstream/main`, GitHub
will automatically close the referenced issue. See [Closing issues via commit
messages][github-help-closing-issues] for details.
@@ -335,9 +335,9 @@ This ensures your work is backed up should something happen to your local
machine and allows others to follow your progress. It also allows you to
[work from multiple computers][self-multiple-computers] without losing work.
Pushing to a feature branch is just like pushing to master:
Pushing to a feature branch is just like pushing to `main`:
```
```console
$ git push origin <branch-name>
Counting objects: 6, done.
Delta compression using up to 4 threads.
@@ -367,7 +367,7 @@ your commit history be able to clearly understand your progression of work?
On the command line, you can use the `git log` command to display an easy to
read list of your commits:
```
```console
$ git log --all --graph --oneline --decorate
* 4f8d75d (HEAD -> 1754-docs-add-git-workflow) docs: Add details about configuring Travis CI.
@@ -376,7 +376,7 @@ $ git log --all --graph --oneline --decorate
* 985116b docs: Add graphic client recs to Git Guide.
* 3c40103 docs: Add stubs for remaining Git Guide sections.
* fc2c01e docs: Add git guide quickstart.
| * f0eaee6 (upstream/master) bug: Fix traceback in get_missed_message_token_from_address().
| * f0eaee6 (upstream/main) bug: Fix traceback in get_missed_message_token_from_address().
```
Alternatively, use your graphical client to view the history for your feature branch.
@@ -404,7 +404,7 @@ Any time you alter history for commits you have already pushed to GitHub,
you'll need to prefix the name of your branch with a `+`. Without this, your
updates will be rejected with a message such as:
```
```console
$ git push origin 1754-docs-add-git-workflow
To git@github.com:christi3k/zulip.git
! [rejected] 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (non-fast-forward)
@@ -413,13 +413,12 @@ hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```
Re-running the command with `+<branch>` allows the push to continue by
re-writing the history for the remote repository:
```
```console
$ git push origin +1754-docs-add-git-workflow
Counting objects: 12, done.
Delta compression using up to 4 threads.
@@ -429,7 +428,6 @@ Total 12 (delta 8), reused 0 (delta 0)
remote: Resolving deltas: 100% (8/8), completed with 2 local objects.
To git@github.com:christi3k/zulip.git
+ 2d49e2d...bfb2433 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (forced update)
```
This is perfectly okay to do on your own feature branches, especially if you're

View File

@@ -31,7 +31,7 @@ Sometimes you want to publish commits. Here are some scenarios:
Finally, the Zulip core team will occasionally want your changes!
- The Zulip core team can accept your changes and add them to
the official repo, usually on the master branch.
the official repo, usually on the `main` branch.
## Relevant Git commands
@@ -45,6 +45,6 @@ working copies:
- `git remote`: This helps you configure short names for remotes.
- `git pull`: This pulls code, but by default creates a merge commit
(which you definitely don't want). However, if you've followed our
[cloning documentation](../git/cloning.md), this will do `git pull
--rebase` instead, which is the only mode you'll want to use when
working on Zulip.
[cloning documentation](../git/cloning.md), this will do
`git pull --rebase` instead, which is the only mode you'll want to
use when working on Zulip.

View File

@@ -16,7 +16,7 @@ notices or warnings it displays.
It's simple to use. Make sure you're in the clone of zulip and run the following:
```
```console
$ ./tools/setup-git-repo
```
@@ -24,7 +24,7 @@ The script doesn't produce any output if successful. To check that the hook has
been installed, print a directory listing for `.git/hooks` and you should see
something similar to:
```
```console
$ ls -l .git/hooks
pre-commit -> ../../tools/pre-commit
```
@@ -44,13 +44,13 @@ checkout.
current branch using `git reset --hard`. Use with caution.**
First, make sure you are working in a branch you want to move (in this
example, we'll use the local `master` branch). Then run the script
example, we'll use the local `main` branch). Then run the script
with the ID number of the pull request as the first argument.
```
$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
```console
$ git checkout main
Switched to branch 'main'
Your branch is up-to-date with 'origin/main'.
$ ./tools/reset-to-pull-request 1900
+ request_id=1900
@@ -70,11 +70,11 @@ HEAD is now at 2bcd1d8 troubleshooting tip about provisioning
`tools/fetch-rebase-pull-request` is a short-cut for [checking out a pull
request locally][zulip-git-guide-fetch-pr] in its own branch and then updating it with any
changes from upstream/master with `git rebase`.
changes from `upstream/main` with `git rebase`.
Run the script with the ID number of the pull request as the first argument.
```
```console
$ tools/fetch-rebase-pull-request 1913
+ request_id=1913
+ git fetch upstream pull/1913/head
@@ -84,8 +84,8 @@ remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.
From https://github.com/zulip/zulip
* branch refs/pull/1913/head -> FETCH_HEAD
+ git checkout upstream/master -b review-1913
Branch review-1913 set up to track remote branch master from upstream.
+ git checkout upstream/main -b review-1913
Branch review-1913 set up to track remote branch main from upstream.
Switched to a new branch 'review-1913'
+ git reset --hard FETCH_HEAD
HEAD is now at 99aa2bf Add provision.py fails issue in common errors
@@ -96,12 +96,12 @@ Current branch review-1913 is up to date.
## Fetch a pull request without rebasing
`tools/fetch-pull-request` is a similar to `tools/fetch-rebase-pull-request`, but
it does not rebase the pull request against upstream/master, thereby getting
it does not rebase the pull request against `upstream/main`, thereby getting
exactly the same repository state as the commit author had.
Run the script with the ID number of the pull request as the first argument.
```
```console
$ tools/fetch-pull-request 5156
+ git diff-index --quiet HEAD
+ request_id=5156
@@ -123,13 +123,13 @@ are merging other users' commits into a Zulip repository. After doing
changes, you can push a branch back to a pull request with e.g.
`tools/push-to-pull-request 1234`. This is useful for a few things:
* Getting CI to run and enabling you to use the GitHub "Merge" buttons
- Getting CI to run and enabling you to use the GitHub "Merge" buttons
to merge a PR after you make some corrections to a PR, without
waiting for an extra round trip with the PR author.
* For commits that aren't ready to merge yet, communicating clearly
- For commits that aren't ready to merge yet, communicating clearly
any changes you'd like to see happen that are easier for you to
explain by just editing the code than in words.
* Saving a contributor from needing to duplicate any rebase work that
- Saving a contributor from needing to duplicate any rebase work that
you did as part of integrating parts of the PR.
You'll likely want to comment on the PR after doing so, to ensure that
@@ -146,16 +146,16 @@ achieve this by granting other users permission to write to your fork.
`tools/clean-branches` is a shell script that removes branches that are either:
1. Local branches that are ancestors of origin/master.
2. Branches in origin that are ancestors of origin/master and named like `$USER-*`.
1. Local branches that are ancestors of `origin/main`.
2. Branches in origin that are ancestors of `origin/main` and named like `$USER-*`.
3. Review branches created by `tools/fetch-rebase-pull-request` and `tools/fetch-pull-request`.
First, make sure you are working in branch `master`. Then run the script without any
First, make sure you are working in branch `main`. Then run the script without any
arguments for default behavior. Since removing review branches can inadvertently remove any
feature branches whose names are like `review-*`, it is not done by default. To
use it, run `tools/clean-branches --reviews`.
```
```console
$ tools/clean-branches --reviews
Deleting local branch review-original-5156 (was 5a1e982)
```
@@ -163,12 +163,13 @@ Deleting local branch review-original-5156 (was 5a1e982)
## Merge conflict on yarn.lock file
If there is a merge conflict on yarn.lock, yarn should be run to
regenerate the file. *Important* don't delete the yarn.lock file. Check out the
latest one from origin/master so that yarn knows the previous asset versions.
regenerate the file. _Important_ don't delete the yarn.lock file. Check out the
latest one from `origin/main` so that yarn knows the previous asset versions.
Run the following commands
```
git checkout origin/master -- yarn.lock
```bash
git checkout origin/main -- yarn.lock
yarn install
git add yarn.lock
git rebase --continue

View File

@@ -1,8 +1,6 @@
Zulip architectural overview
============================
# Zulip architectural overview
Key codebases
-------------
## Key codebases
The main Zulip codebase is at <https://github.com/zulip/zulip>. It
contains the Zulip backend (written in Python 3.x and Django), the
@@ -37,8 +35,7 @@ translations.
In this overview, we'll mainly discuss the core Zulip server and web
application.
Usage assumptions and concepts
------------------------------
## Usage assumptions and concepts
Zulip is a real-time team chat application meant to provide a great
experience for a wide range of organizations, from companies to
@@ -50,7 +47,7 @@ all modern web browsers, several cross-protocol chat clients, and
numerous dedicated [Zulip API](https://zulip.com/api) clients
(e.g. bots).
A server can host multiple Zulip *realms* (organizations), each on its
A server can host multiple Zulip _realms_ (organizations), each on its
own (sub)domain. While most installations host only one organization, some
such as zulip.com host thousands. Each organization is a private
chamber with its own users, streams, customizations, and so on. This
@@ -61,8 +58,7 @@ more on security considerations and options, see [the security model
section](../production/security-model.md) and the [Zulip Help
Center](https://zulip.com/help).
Components
----------
## Components
![architecture-simple](../images/architecture_simple.png)
@@ -116,8 +112,8 @@ For more details on the frontend, see our documentation on
[directory structure](../overview/directory-structure.md), and
[the static asset pipeline](../subsystems/html-css.html#static-asset-pipeline).
[Jinja2]: http://jinja.pocoo.org/
[Handlebars]: https://handlebarsjs.com/
[jinja2]: http://jinja.pocoo.org/
[handlebars]: https://handlebarsjs.com/
### nginx
@@ -181,8 +177,10 @@ Redis is configured in `zulip/puppet/zulip/files/redis` and it's a
pretty standard configuration except for the last line, which turns off
persistence:
```text
# Zulip-specific configuration: disable saving to disk.
save ""
```
People often wonder if we could replace memcached with Redis (or
replace RabbitMQ with Redis, with some loss of functionality).
@@ -194,12 +192,12 @@ perceived benefit for using Redis is usually to reduce memory
consumption by running fewer services, and no such benefit would
materialize:
* Our cache uses significant memory, but that memory usage would be
- Our cache uses significant memory, but that memory usage would be
essentially the same with Redis as it is with memcached.
* All of these services have low minimum memory requirements, and in
- All of these services have low minimum memory requirements, and in
fact our applications for Redis and RabbitMQ do not use significant
memory even at scale.
* We would likely need to run multiple Redis services (with different
- We would likely need to run multiple Redis services (with different
configurations) in order to ensure the pure LRU use case (memcached)
doesn't push out data that we want to persist until expiry
(Redis-based rate limiting) or until consumed (RabbitMQ-based
@@ -269,35 +267,35 @@ in Zulip development conversations. In general, our goal is to
minimize the set of terminology listed here by giving elements
self-explanatory names.
* **bankruptcy**: When a user has been off Zulip for several days and
- **bankruptcy**: When a user has been off Zulip for several days and
has hundreds of unread messages, they are prompted for whether
they want to mark all their unread messages as read. This is
called "declaring bankruptcy" (in reference to the concept in
finance).
* **chevron**: A small downward-facing arrow next to a message's
- **chevron**: A small downward-facing arrow next to a message's
timestamp, offering contextual options, e.g., "Reply", "Mute [this
topic]", or "Link to this conversation". To avoid visual clutter,
the chevron only appears in the web UI upon hover.
* **ellipsis**: A small vertical three dot icon (technically called
- **ellipsis**: A small vertical three dot icon (technically called
as ellipsis-v), present in sidebars as a menu icon.
It offers contextual options for global filters (All messages
and Starred messages), stream filters and topics in left
sidebar and users in right sidebar. To avoid visual clutter
ellipsis only appears in the web UI upon hover.
* **huddle**: What the codebase calls a "group private message".
- **huddle**: What the codebase calls a "group private message".
* **message editing**: If the realm admin allows it, then after a user
- **message editing**: If the realm admin allows it, then after a user
posts a message, the user has a few minutes to click "Edit" and
change the content of their message. If they do, Zulip adds a
marker such as "(EDITED)" at the top of the message, visible to
anyone who can see the message.
* **realm**: What the codebase calls an "organization" in the UI.
- **realm**: What the codebase calls an "organization" in the UI.
* **recipient bar**: A visual indication of the context of a message
- **recipient bar**: A visual indication of the context of a message
or group of messages, displaying the stream and topic or private
message recipient list, at the top of a group of messages. A
typical 1-line message to a new recipient shows to the user as
@@ -306,7 +304,7 @@ self-explanatory names.
the star and the chevron), and third the message content. The
recipient bar is or contains hyperlinks to help the user narrow.
* **star**: Zulip allows a user to mark any message they can see,
- **star**: Zulip allows a user to mark any message they can see,
public or private, as "starred". A user can easily access messages
they've starred through the "Starred messages" link in the
left sidebar, or use "is:starred" as a narrow or a search
@@ -314,4 +312,4 @@ self-explanatory names.
message is private; other users and realm admins don't know
whether a message has been starred, or by whom.
* **subject**: What the codebase calls a "topic" in many places.
- **subject**: What the codebase calls a "topic" in many places.

View File

@@ -7,6 +7,118 @@ up-to-date list of raw changes.
## Zulip 4.x series
### 4.7 -- 2021-10-04
- CVE-2021-41115: Prevent organization administrators from affecting
the server with a regular expression denial-of-service attack
through linkifier patterns.
### 4.6 -- 2021-09-23
- Documented official support for Debian 11 Bullseye, now that it is
officially released by Debian upstream.
- Fixed installation on Debian 10 Buster. Upstream infrastructure had
broken the Python `virtualenv` tool on this platform, which we've
worked around for this release.
- Zulip releases are now distributed from https://download.zulip.com/server/,
replacing the old `www.zulip.org` server.
- Added support for LDAP synchronization of the `is_realm_owner` and
`is_moderator` flags.
- `upgrade-zulip-from-git` now uses `git fetch --prune`; this ensures
`upgrade-zulip-from-git master` with return an error rather than
using a stale cached version of the `master` branch, which was
renamed to `main` this month.
- Added a new `reset_authentication_attempt_count` management command
to allow sysadmins to manually reset authentication rate limits.
- Fixed a bug that caused the `upgrade-postgresql` tool to
incorrectly remove `supervisord` configuration for `process-fts-updates`.
- Fixed a rare migration bug when upgrading from Zulip versions 2.1 and older.
- Fixed a subtle bug where the left sidebar would show both old and
new names for some topics that had been renamed.
- Fixed incoming email gateway support for configurations
with the `http_only` setting enabled.
- Fixed issues where Zulip's outgoing webhook, with the
Slack-compatible interface, had a different format from Slack's
documented interface.
- The installation and upgrade documentations now show the latest
release's version number.
- Backported many improvements to the ReadTheDocs documentation.
- Updated translation data from Transifex.
### 4.5 -- 2021-07-25
- Added a tool to fix potential database corruption caused by host OS
upgrades (was listed in 4.4 release notes, but accidentally omitted).
### 4.4 -- 2021-07-22
- Fixed a possible denial-of-service attack in Markdown fenced code
block parsing.
- Smokescreen, if installed, now defaults to only listening on
127.0.0.1; this prevents it from being used as an open HTTP proxy if
it did not have other firewalls protecting incoming port 4750.
- Fixed a performance/scalability issue for installations using the S3
file uploads backend.
- Fixed a bug where users could turn other users messages they could
read into widgets (e.g. polls).
- Fixed a bug where emoji and avatar image requests were sent through
Camo; doing so does not add any security benefit, and broke custom
emoji that had been imported from Slack in Zulip 1.8.1 or earlier.
- Changed to log just a warning, instead of an exception, in the case
that the `embed_links` worker cannot fetch previews for all links in
a message within the 30-second timeout. Each preview request within
a message already has a 15-second timeout.
- Ensured `psycopg2` is installed before starting
`process_fts_updates`; otherwise, it might fail to start several
times before the package was installed.
- Worked around a bug in supervisor where, when using SysV init,
`/etc/init.d/supervisor restart` would only have stopped, not
restarted, the process.
- Modified upgrade scripts to better handle failure, and suggest next
steps and point to logs.
- Zulip now hides the “show password” eye icon that IE and Edge
browsers place in password inputs; this duplicated the
already-present JavaScript-based functionality.
- Fixed “OR” glitch on login page if SAML authentication is enabled
but not configured.
- The `send_test_email` management command now shows the full SMTP
conversation on failure.
- Provided a `change_password` management command which takes a
`--realm` option.
- Fixed `upgrade-zulip-from-git` crashing in CSS source map generation
on 1-CPU systems.
- Added an `auto_signup` field in SAML configuration to auto-create
accounts upon first login attempt by users which are authenticated
by SAML.
- Provided better error messages when `puppet_classes` in `zulip.conf`
are mistakenly space-separated instead of comma-separated.
- Updated translations for many languages.
### 4.3 -- 2021-06-02
- Fixed exception when upgrading older servers with the
`JITSI_SERVER_URL` setting set to `None` to disable Jitsi.
- Fixed GIPHY integration dropdown appearing when the server
doesn't have a GIPHY API key configured.
- The GIPHY API library is no longer loaded for users who are not
actively using the GIPHY integration.
- Improved formatting for Grafana integration.
- Fixed previews of Dropbox image links.
- Fixed support for storing avatars/emoji in non-S3 upload backends.
- Fixed an overly strict database constaint for code playgrounds.
- Tagged user status strings for translation.
- Updated translation data from Transifex.
### 4.2 -- 2021-05-13
- Fixed exception in purge-old-deployments when upgrading on
a system that has never upgraded using Git.
- Fixed installation from a directory readable only by root.
### 4.1 -- 2021-05-13
- Fixed exception upgrading to the 4.x series from older releases.
### 4.0 -- 2021-05-13
#### Highlights
@@ -223,8 +335,9 @@ up-to-date list of raw changes.
- Fixed Postfix configuration error which would prevent outgoing email
to any email address containing `.`, `+`, or starting with `mm`, when
configured to use the local Postfix to deliver outgoing email.
- Fixed a backporting error which caused the `manage.py
change_user_role` tool to not work for `admin`, `member`, or `guest` roles.
- Fixed a backporting error which caused the
`manage.py change_user_role` tool to not work for `admin`, `member`,
or `guest` roles.
- Add support for logout events sent from modern versions of the
desktop application.
- Upgraded minor python dependencies.
@@ -288,7 +401,7 @@ up-to-date list of raw changes.
was actually an integration for canarytokens.org).
- Reformatted the frontend codebase using prettier. This change was
included in this maintenance release to ensure backporting patches
from master remains easy.
from `main` remains easy.
### 3.0 -- July 16, 2020
@@ -344,7 +457,7 @@ up-to-date list of raw changes.
that will fix this bug. The new migration will fail if any such
duplicate accounts already exist; you can check whether this will
happen be running the following in a [management shell][manage-shell]:
```
```python
from django.db.models.functions import Lower
UserProfile.objects.all().annotate(email_lower=Lower("delivery_email"))
.values('realm_id', 'email_lower').annotate(Count('id')).filter(id__count__gte=2)
@@ -363,8 +476,8 @@ up-to-date list of raw changes.
[our new PostgreSQL upgrade guide][postgresql-upgrade].
- The format of the `JWT_AUTH_KEYS` setting has changed to include an
[algorithms](https://pyjwt.readthedocs.io/en/latest/algorithms.html)
list: `{"subdomain": "key"}` becomes `{"subdomain": {"key": "key",
"algorithms": ["HS256"]}}`.
list: `{"subdomain": "key"}` becomes
`{"subdomain": {"key": "key", "algorithms": ["HS256"]}}`.
- Added a new organization owner permission above the previous
organization administrator. All existing organization
administrators are automatically converted into organization owners.
@@ -688,11 +801,11 @@ lose the setting and need to re-enable it.
still work, but users should update the following setting names in
their configuration as we will desupport the old names in a future
release:
* In `/etc/zulip/zulip-secrets.conf`, `google_oauth2_client_secret`
- In `/etc/zulip/zulip-secrets.conf`, `google_oauth2_client_secret`
is now called with `social_auth_google_secret`.
* In `/etc/zulip/settings.py`, `GOOGLE_OAUTH2_CLIENT_ID` should be
- In `/etc/zulip/settings.py`, `GOOGLE_OAUTH2_CLIENT_ID` should be
replaced with `SOCIAL_AUTH_GOOGLE_KEY`.
* In `/etc/zulip/settings.py`, `GoogleMobileOauth2Backend` should
- In `/etc/zulip/settings.py`, `GoogleMobileOauth2Backend` should
be replaced with called `GoogleAuthBackend`.
- Installations using Zulip's LDAP integration without
`LDAP_APPEND_DOMAIN` will need to configure two new settings telling
@@ -732,6 +845,7 @@ lose the setting and need to re-enable it.
downtime, and then upgrade to the new release.
#### Full feature changelog
- Added sortable columns to all tables in settings pages.
- Added webapp support for self-service public data exports.
- Added 'e' keyboard shortcut for editing currently selected message.
@@ -939,6 +1053,7 @@ lose the setting and need to re-enable it.
### 2.0.0 -- 2019-03-01
#### Highlights
- Added automation for synchronizing user avatars, custom profile
fields, disabled status, and more from LDAP/active directory.
- Added support for explicitly setting oneself as "away" and "user
@@ -968,6 +1083,7 @@ and is enabled by default in that case. To disable it, set
[mpns-statistics-docs]: ../production/mobile-push-notifications.html#submitting-statistics
#### Full feature changelog
- Added support for CentOS 7 in the development environment
provisioning process. This is an important step towards production
CentOS/RHEL 7 support.
@@ -1105,7 +1221,7 @@ Zulip installations; it has minimal changes for existing servers.
#### Upgrade notes for 1.9.0
* Zulip 1.9 contains a significant database migration that can take
- Zulip 1.9 contains a significant database migration that can take
several minutes to run. The upgrade process automatically minimizes
disruption by running this migration first, before beginning the
user-facing downtime. However, if you'd like to watch the downtime
@@ -1115,6 +1231,7 @@ Zulip installations; it has minimal changes for existing servers.
doing an apt upgrade first.
#### Full feature changelog
- Added an organization setting for message deletion time limits.
- Added an organization setting to control who can edit topics.
- Added Ctrl+K keyboard shortcut for getting to search (same as /, but
@@ -1210,6 +1327,7 @@ Zulip installations; it has minimal changes for existing servers.
### 1.8.0 -- 2018-04-17
#### Highlights
- Dramatically simplified the server installation process; it's now possible
to install Zulip without first setting up outgoing email.
- Added experimental support for importing an organization's history
@@ -1219,8 +1337,8 @@ Zulip installations; it has minimal changes for existing servers.
- Lots of visual polish improvements.
- Countless small bugfixes both in the backend and the UI.
**Security and privacy:**
- Several important security fixes since 1.7.0, which were released
already in 1.7.1 and 1.7.2.
- The security model for private streams has changed. Now
@@ -1237,8 +1355,8 @@ Zulip installations; it has minimal changes for existing servers.
- Added a user setting to control whether email notifications include
message content (or just the fact that there are new messages).
**Visual and UI:**
- Added a user setting to translate emoticons/smileys to emoji.
- Added a user setting to choose the emoji set used in Zulip: Google,
Twitter, Apple, or Emoji One.
@@ -1254,8 +1372,8 @@ Zulip installations; it has minimal changes for existing servers.
- Improved the descriptions and UI for many settings.
- Improved visual design of the help center (/help/).
**Core chat experience:**
- Added support for mentioning groups of users.
- Added a setting to allow users to delete their messages.
- Added support for uploading files in the message-edit UI.
@@ -1315,8 +1433,8 @@ Zulip installations; it has minimal changes for existing servers.
included in email subject lines.
- Fixed uploading user avatars encoded using the CMYK mode.
**User accounts and invites:**
- Added support for users in multiple realms having the same email.
- Added a display for whether the user is logged-in in logged-out
pages.
@@ -1331,8 +1449,8 @@ Zulip installations; it has minimal changes for existing servers.
- Split the Notifications Stream setting in two settings, one for new
users, the other for new streams.
**Stream subscriptions and settings:**
- Added traffic statistics (messages/week) to the "Manage streams" UI.
- Fixed numerous issues in the "stream settings" UI.
- Fixed numerous subtle bugs with the stream creation UI.
@@ -1340,8 +1458,8 @@ Zulip installations; it has minimal changes for existing servers.
so that they can be robust to streams being renamed. The change is
backwards-compatible; existing narrow URLs still work.
**API, bots, and integrations:**
- Rewrote our API documentation to be much more friendly and
expansive; it now covers most important endpoints, with nice examples.
- New integrations: ErrBot, GoCD, Google Code-In, Opbeat, Groove,
@@ -1359,8 +1477,8 @@ Zulip installations; it has minimal changes for existing servers.
numbered-list format.
- APIs for fetching messages now provide more metadata to help clients.
**Keyboard shortcuts:**
- Added new "basics" section to keyboard shortcuts documentation.
- Added a new ">" keyboard shortcut for quote-and-reply.
- Added a new "p" keyboard shortcut to jump to next unread PM thread.
@@ -1368,16 +1486,16 @@ Zulip installations; it has minimal changes for existing servers.
- Changed the hotkey for compose-private-message from "C" to "x".
- Improve keyboard navigation of left and right sidebars with arrow keys.
**Mobile apps backend:**
- Added support for logging into the mobile apps with RemoteUserBackend.
- Improved mobile notifications to support narrowing when one clicks a
mobile push notification.
- Statistics on the fraction of strings that are translated now
include strings in the mobile apps as well.
**For server admins:**
- Added certbot support to the installer for getting certificates.
- Added support for hosting multiple domains, not all as subdomains of
the same base domain.
@@ -1390,8 +1508,8 @@ Zulip installations; it has minimal changes for existing servers.
- Improved Tornado retry logic for connecting to RabbitMQ.
- Added a server setting to control whether digest emails are sent.
**For Zulip developers:**
- Migrated the codebase to use the nice Python 3 typing syntax.
- Added a new /team/ page explaining the team, with a nice
visualization of our contributors.
@@ -1493,7 +1611,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1.
#### Upgrade notes for 1.7.0
* Zulip 1.7 contains some significant database migrations that can
- Zulip 1.7 contains some significant database migrations that can
take several minutes to run. The upgrade process automatically
minimizes disruption by running these first, before beginning the
user-facing downtime. However, if you'd like to watch the downtime
@@ -1502,7 +1620,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1.
as the usual trick of
doing an apt upgrade first.
* We've removed support for an uncommon legacy deployment model where
- We've removed support for an uncommon legacy deployment model where
a Zulip server served multiple organizations on the same domain.
Installs with multiple organizations now require each organization
to have its own subdomain.
@@ -1512,7 +1630,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1.
that hosts multiple organizations, you'll want to read [our guide on
multiple organizations](../production/multiple-organizations.md).
* We simplified the configuration for our password strength checker to
- We simplified the configuration for our password strength checker to
be much more intuitive. If you were using the
`PASSWORD_MIN_ZXCVBN_QUALITY` setting,
[it has been replaced](https://github.com/zulip/zulip/commit/a116303604e362796afa54b5d923ea5312b2ea23) by
@@ -1666,91 +1784,91 @@ Zulip apps.
#### Full feature changelog
* Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and
- Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and
WordPress webhook integrations.
* Added LaTeX support to the Markdown processor.
* Added support for filtering branches to all Git integrations.
* Added read-only access to organization-level settings for all users.
* Added UI for managing muted topics and uploaded files.
* Added UI for displaying message edit history.
* Added support for various features needed by new mobile app.
* Added deep links for settings/subscriptions interfaces.
* Added an animation when messages are edited.
* Added support for registration with GitHub auth (not just login).
* Added tracking of uploaded file quotas.
* Added option to display emoji as their alt codes.
* Added new audit log table, to eventually support an auditing UI.
* Added several new permissions-related organization settings.
* Added new endpoint for fetching presence data, useful in employee directories.
* Added typeahead for language for syntax highlighting in code blocks.
* Added support for basic Markdown in stream descriptions.
* Added email notifications on new Zulip logins.
* Added security hardening before serving uploaded files.
* Added new PRIVACY_POLICY setting to provide a Markdown privacy policy.
* Added an icon to distinguish bot users as message senders.
* Added a command-line Slack importer tool using the API.
* Added new announcement notifications on stream creation.
* Added support for some newer Unicode emoji code points.
* Added support for users deleting realm emoji they themselves uploaded.
* Added support for organization administrators deleting messages.
* Extended data available to mobile apps to cover the entire API.
* Redesigned bots UI. Now can change owners and reactivate bots.
* Redesigned the visuals of code blocks to be prettier.
* Changed right sidebar presence UI to only show recently active users
- Added LaTeX support to the Markdown processor.
- Added support for filtering branches to all Git integrations.
- Added read-only access to organization-level settings for all users.
- Added UI for managing muted topics and uploaded files.
- Added UI for displaying message edit history.
- Added support for various features needed by new mobile app.
- Added deep links for settings/subscriptions interfaces.
- Added an animation when messages are edited.
- Added support for registration with GitHub auth (not just login).
- Added tracking of uploaded file quotas.
- Added option to display emoji as their alt codes.
- Added new audit log table, to eventually support an auditing UI.
- Added several new permissions-related organization settings.
- Added new endpoint for fetching presence data, useful in employee directories.
- Added typeahead for language for syntax highlighting in code blocks.
- Added support for basic Markdown in stream descriptions.
- Added email notifications on new Zulip logins.
- Added security hardening before serving uploaded files.
- Added new PRIVACY_POLICY setting to provide a Markdown privacy policy.
- Added an icon to distinguish bot users as message senders.
- Added a command-line Slack importer tool using the API.
- Added new announcement notifications on stream creation.
- Added support for some newer Unicode emoji code points.
- Added support for users deleting realm emoji they themselves uploaded.
- Added support for organization administrators deleting messages.
- Extended data available to mobile apps to cover the entire API.
- Redesigned bots UI. Now can change owners and reactivate bots.
- Redesigned the visuals of code blocks to be prettier.
- Changed right sidebar presence UI to only show recently active users
in large organizations. This has a huge performance benefit.
* Changed color for private messages to look better.
* Converted realm emoji to be uploaded, not links, for better robustness.
* Switched the default password hasher for new passwords to Argon2.
* Increased the paragraph spacing, making multi-paragraph.
* Improved formatting of all Git integrations.
* Improved the UI of the /stats analytics pages.
* Improved search typeahead to support group private messages.
* Improved logic for when the compose box should open/close.
* Improved lightbox to support scrolling through images.
* Improved Markdown support for bulleted lists.
* Improved copy-to-clipboard support in various places.
* Improved subject lines of missed message emails.
* Improved handling of users trying to log in with OAuth without an account.
* Improved UI of off-the-Internet errors to not be hidden in narrow windows.
* Improved rate-limiting errors to be more easily machine-readable.
* Parallelized the backend test suite; now runs 1600 tests in <30s.
* Fixed numerous bugs and performance issues with stream management.
* Fixed an issue with the fake emails assigned to bot users.
* Fixed a major performance issue in stream creation.
* Fixed numerous minor accessibility issues.
* Fixed a subtle interaction between click-to-reply and copy-paste.
* Fixed various formatting issues with /me messages.
* Fixed numerous real-time sync issues involving users changing their
- Changed color for private messages to look better.
- Converted realm emoji to be uploaded, not links, for better robustness.
- Switched the default password hasher for new passwords to Argon2.
- Increased the paragraph spacing, making multi-paragraph.
- Improved formatting of all Git integrations.
- Improved the UI of the /stats analytics pages.
- Improved search typeahead to support group private messages.
- Improved logic for when the compose box should open/close.
- Improved lightbox to support scrolling through images.
- Improved Markdown support for bulleted lists.
- Improved copy-to-clipboard support in various places.
- Improved subject lines of missed message emails.
- Improved handling of users trying to log in with OAuth without an account.
- Improved UI of off-the-Internet errors to not be hidden in narrow windows.
- Improved rate-limiting errors to be more easily machine-readable.
- Parallelized the backend test suite; now runs 1600 tests in <30s.
- Fixed numerous bugs and performance issues with stream management.
- Fixed an issue with the fake emails assigned to bot users.
- Fixed a major performance issue in stream creation.
- Fixed numerous minor accessibility issues.
- Fixed a subtle interaction between click-to-reply and copy-paste.
- Fixed various formatting issues with /me messages.
- Fixed numerous real-time sync issues involving users changing their
name, avatar, or email address and streams being renamed.
* Fixed numerous performance issues across the project.
* Fixed various left sidebar ordering and live-updated bugs.
* Fixed numerous bugs with the message editing widget.
* Fixed missing logging / rate limiting on browser endpoints.
* Fixed regressions in Zulip's browser state preservation on reload logic.
* Fixed support for Unicode characters in the email mirror system.
* Fixed load spikes when email mirror is receiving a lot of traffic.
* Fixed the ugly grey flicker when scrolling fast on Macs.
* Fixed previews of GitHub image URLs.
* Fixed narrowing via clicking on desktop notifications.
* Fixed Subscribed/Unsubscribed bookends appearing incorrectly.
* Eliminated the idea of a realm having a canonical domain; now
- Fixed numerous performance issues across the project.
- Fixed various left sidebar ordering and live-updated bugs.
- Fixed numerous bugs with the message editing widget.
- Fixed missing logging / rate limiting on browser endpoints.
- Fixed regressions in Zulip's browser state preservation on reload logic.
- Fixed support for Unicode characters in the email mirror system.
- Fixed load spikes when email mirror is receiving a lot of traffic.
- Fixed the ugly grey flicker when scrolling fast on Macs.
- Fixed previews of GitHub image URLs.
- Fixed narrowing via clicking on desktop notifications.
- Fixed Subscribed/Unsubscribed bookends appearing incorrectly.
- Eliminated the idea of a realm having a canonical domain; now
there's simply the list of allowed domains for new users.
* Migrated avatars to a user-id-based storage setup (not email-based).
* Trailing whitespace is now stripped in code blocks, avoiding
- Migrated avatars to a user-id-based storage setup (not email-based).
- Trailing whitespace is now stripped in code blocks, avoiding
unnecessary scrollbars.
* Most API payloads now refer to users primarily by user ID, with
- Most API payloads now refer to users primarily by user ID, with
email available for backwards-compatibility. In the future, we may
remove email support.
* Cleaned up Zulip's supervisord configuration. A side effect is the
- Cleaned up Zulip's supervisord configuration. A side effect is the
names of the log files have changed for all the queue workers.
* Refactored various endpoints to use a single code path for security
- Refactored various endpoints to use a single code path for security
hardening.
* Removed support for the `MANDRILL_CLIENT` setting. It hadn't been
- Removed support for the `MANDRILL_CLIENT` setting. It hadn't been
used in years.
* Changed `NOREPLY_EMAIL_ADDRESS` setting to `Name <user@example.com>`
- Changed `NOREPLY_EMAIL_ADDRESS` setting to `Name <user@example.com>`
format.
* Disabled the web tutorial on mobile.
* Backend test coverage is now 93%, with 100% in views code.
- Disabled the web tutorial on mobile.
- Backend test coverage is now 93%, with 100% in views code.
### 1.5.2 -- 2017-06-01
@@ -1993,6 +2111,7 @@ Zulip apps.
open source.
### 1.3.13 - 2016-06-21
- Added nearly complete internationalization of the Zulip UI.
- Added warning when using @all/@everyone.
- Added button offering to subscribe at bottom of narrows to streams
@@ -2030,6 +2149,7 @@ Zulip apps.
- Removed most of the remaining JavaScript global variables.
### 1.3.12 - 2016-05-10
- CVE-2016-4426: Bot API keys were accessible to other users in the same realm.
- CVE-2016-4427: Deactivated users could access messages if SSO was enabled.
- Fixed a RabbitMQ configuration bug that resulted in reordered messages.
@@ -2037,6 +2157,7 @@ Zulip apps.
- Added an option to logout_all_users to delete only sessions for deactivated users.
### 1.3.11 - 2016-05-02
- Moved email digest support into the default Zulip production configuration.
- Added options for configuring PostgreSQL, RabbitMQ, Redis, and memcached
in settings.py.
@@ -2065,6 +2186,7 @@ Zulip apps.
- Fixed Jira integration incorrectly not @-mentioning assignee.
### 1.3.10 - 2016-01-21
- Added new integration for Travis CI.
- Added settings option to control maximum file upload size.
- Added support for running Zulip development environment in Docker.
@@ -2087,9 +2209,11 @@ Zulip apps.
- Substantially cleaned up console logging from run-dev.py.
### 1.3.9 - 2015-11-16
- Fixed buggy #! lines in upgrade scripts.
### 1.3.8 - 2015-11-15
- Added options to the Python API for working with untrusted server certificates.
- Added a lot of documentation on the development environment and testing.
- Added partial support for translating the Zulip UI.
@@ -2105,6 +2229,7 @@ Zulip apps.
- Major preliminary progress towards supporting Python 3.
### 1.3.7 - 2015-10-19
- Turn off desktop and audible notifications for streams by default.
- Added support for the LDAP authentication integration creating new users.
- Added new endpoint to support Google auth on mobile.
@@ -2117,15 +2242,15 @@ Zulip apps.
This section links to the upgrade notes from past releases, so you can
easily read them all when upgrading across multiple releases.
* [Upgrade notes for 4.0](#upgrade-notes-for-4-0)
* [Upgrade notes for 3.0](#upgrade-notes-for-3-0)
* [Upgrade notes for 2.1.5](#upgrade-notes-for-2-1-5)
* [Upgrade notes for 2.1.0](#upgrade-notes-for-2-1-0)
* [Upgrade notes for 2.0.0](#upgrade-notes-for-2-0-0)
* [Upgrade notes for 1.9.0](#upgrade-notes-for-1-9-0)
* [Upgrade notes for 1.8.0](#upgrade-notes-for-1-8-0)
* [Upgrade notes for 1.7.0](#upgrade-notes-for-1-7-0)
- [Upgrade notes for 4.0](#upgrade-notes-for-4-0)
- [Upgrade notes for 3.0](#upgrade-notes-for-3-0)
- [Upgrade notes for 2.1.5](#upgrade-notes-for-2-1-5)
- [Upgrade notes for 2.1.0](#upgrade-notes-for-2-1-0)
- [Upgrade notes for 2.0.0](#upgrade-notes-for-2-0-0)
- [Upgrade notes for 1.9.0](#upgrade-notes-for-1-9-0)
- [Upgrade notes for 1.8.0](#upgrade-notes-for-1-8-0)
- [Upgrade notes for 1.7.0](#upgrade-notes-for-1-7-0)
[docker-zulip]: https://github.com/zulip/docker-zulip
[commit-log]: https://github.com/zulip/zulip/commits/master
[commit-log]: https://github.com/zulip/zulip/commits/main
[latest-changelog]: https://zulip.readthedocs.io/en/latest/overview/changelog.html

View File

@@ -13,173 +13,172 @@ Zulip uses the [Django web
framework](https://docs.djangoproject.com/en/1.8/), so a lot of these
paths will be familiar to Django developers.
* `zproject/urls.py` Main
- `zproject/urls.py` Main
[Django routes file](https://docs.djangoproject.com/en/1.8/topics/http/urls/).
Defines which URLs are handled by which view functions or templates.
* `zerver/models.py` Main
- `zerver/models.py` Main
[Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/)
file. Defines Zulip's database tables.
* `zerver/lib/*.py` Most library code.
- `zerver/lib/*.py` Most library code.
* `zerver/lib/actions.py` Most code doing writes to user-facing
- `zerver/lib/actions.py` Most code doing writes to user-facing
database tables lives here. In particular, we have a policy that
all code calling `send_event` to trigger [pushing data to
clients](../subsystems/events-system.md) must live here.
* `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/).
- `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/).
* `zerver/webhooks/` Webhook views and tests for [Zulip's incoming webhook integrations](
https://zulip.com/api/incoming-webhooks-overview).
- `zerver/webhooks/` Webhook views and tests for [Zulip's incoming webhook integrations](https://zulip.com/api/incoming-webhooks-overview).
* `zerver/tornado/views.py` Tornado views.
- `zerver/tornado/views.py` Tornado views.
* `zerver/worker/queue_processors.py` [Queue workers](../subsystems/queuing.md).
- `zerver/worker/queue_processors.py` [Queue workers](../subsystems/queuing.md).
* `zerver/lib/markdown/` [Backend Markdown processor](../subsystems/markdown.md).
- `zerver/lib/markdown/` [Backend Markdown processor](../subsystems/markdown.md).
* `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/).
- `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/).
-------------------------------------------------------------------
---
### HTML templates
See [our docs](../subsystems/html-css.md) for details on Zulip's
templating systems.
* `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates
- `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates
for the backend (for zerver app; logged-in content is in `templates/zerver/app`).
* `static/templates/` [Handlebars](https://handlebarsjs.com/) templates for the frontend.
- `static/templates/` [Handlebars](https://handlebarsjs.com/) templates for the frontend.
----------------------------------------
---
### JavaScript, TypeScript, and other static assets
* `static/js/` Zulip's own JavaScript and TypeScript sources.
- `static/js/` Zulip's own JavaScript and TypeScript sources.
* `static/styles/` Zulip's own CSS.
- `static/styles/` Zulip's own CSS.
* `static/images/` Zulip's images.
- `static/images/` Zulip's images.
* `static/third/` Third-party JavaScript and CSS that has been vendored.
- `static/third/` Third-party JavaScript and CSS that has been vendored.
* `node_modules/` Third-party JavaScript installed via `yarn`.
- `node_modules/` Third-party JavaScript installed via `yarn`.
* `static/assets/` For assets not to be served to the web (e.g. the system to
- `static/assets/` For assets not to be served to the web (e.g. the system to
generate our favicons).
-----------------------------------------------------------------------
---
### Tests
* `zerver/tests/` Backend tests.
- `zerver/tests/` Backend tests.
* `frontend_tests/node_tests/` Node Frontend unit tests.
- `frontend_tests/node_tests/` Node Frontend unit tests.
* `frontend_tests/puppeteer_tests/` Puppeteer frontend integration tests.
- `frontend_tests/puppeteer_tests/` Puppeteer frontend integration tests.
* `tools/test-*` Developer-facing test runner scripts.
- `tools/test-*` Developer-facing test runner scripts.
-----------------------------------------------------
---
### Management commands
These are distinguished from scripts, below, by needing to run a
Django context (i.e. with database access).
* `zerver/management/commands/`
- `zerver/management/commands/`
[Management commands](../subsystems/management-commands.md) one might run at a
production deployment site (e.g. scripts to change a value or
deactivate a user properly).
* `zilencer/management/commands/` includes some dev-specific
- `zilencer/management/commands/` includes some dev-specific
commands such as `populate_db`, which are not included in
the production distribution.
---------------------------------------------------------------
---
### Scripts
* `scripts/` Scripts that production deployments might run manually
- `scripts/` Scripts that production deployments might run manually
(e.g., `restart-server`).
* `scripts/lib/` Scripts that are needed on production deployments but
- `scripts/lib/` Scripts that are needed on production deployments but
humans should never run directly.
* `scripts/setup/` Scripts that production deployments will only run
- `scripts/setup/` Scripts that production deployments will only run
once, during installation.
* `tools/` Scripts used only in a Zulip development environment.
- `tools/` Scripts used only in a Zulip development environment.
These are not included in production release tarballs for Zulip, so
that we can include scripts here one wouldn't want someone to run in
production accidentally (e.g. things that delete the Zulip database
without prompting).
* `tools/setup/` Subdirectory of `tools/` for things only used during
- `tools/setup/` Subdirectory of `tools/` for things only used during
the development environment setup process.
* `tools/ci/` Subdirectory of `tools/` for things only used to
- `tools/ci/` Subdirectory of `tools/` for things only used to
set up and run our tests in CI. Actual test suites should
go in `tools/`.
---------------------------------------------------------
---
### API and bots
* See the [Zulip API repository](https://github.com/zulip/python-zulip-api).
- See the [Zulip API repository](https://github.com/zulip/python-zulip-api).
Zulip's Python API bindings, a number of Zulip integrations and
bots, and a framework for running and testing Zulip bots, used to be
developed in the main Zulip server repo but are now in their own repo.
* `templates/zerver/integrations/` (within `templates/zerver/`, above).
- `templates/zerver/integrations/` (within `templates/zerver/`, above).
Documentation for these integrations.
-------------------------------------------------------------------------
---
### Production Puppet configuration
This is used to deploy essentially all configuration in production.
* `puppet/zulip/` For configuration for production deployments.
- `puppet/zulip/` For configuration for production deployments.
* `puppet/zulip/manifests/profile/standalone.pp` Main manifest for Zulip standalone deployments.
- `puppet/zulip/manifests/profile/standalone.pp` Main manifest for Zulip standalone deployments.
-----------------------------------------------------------------------
---
### Additional Django apps
* `confirmation` Email confirmation system.
- `confirmation` Email confirmation system.
* `analytics` Analytics for the Zulip server administrator (needs work to
- `analytics` Analytics for the Zulip server administrator (needs work to
be useful to normal Zulip sites).
* `corporate` The old Zulip.com website. Not included in production
- `corporate` The old Zulip.com website. Not included in production
distribution.
* `zilencer` Primarily used to hold management commands that aren't
- `zilencer` Primarily used to hold management commands that aren't
used in production. Not included in production distribution.
-----------------------------------------------------------------------
---
### Jinja2 compatibility files
* `zproject/jinja2/__init__.py` Jinja2 environment.
- `zproject/jinja2/__init__.py` Jinja2 environment.
-----------------------------------------------------------------------
---
### Translation files
* `locale/` Backend (Django) and frontend translation data files.
- `locale/` Backend (Django) and frontend translation data files.
-----------------------------------------------------------------------
---
### Documentation
* `docs/` Source for this documentation.
- `docs/` Source for this documentation.
--------------------------------------------------------------
---
You can consult the repository's `.gitattributes` file to see exactly
which components are excluded from production releases (release

View File

@@ -4,20 +4,20 @@ This page details the release lifecycle for the Zulip server and
client-apps, well as our policies around backwards-compatibility and
security support policies. In short:
* We recommend always running the latest releases of the Zulip clients
- We recommend always running the latest releases of the Zulip clients
and servers. Server upgrades are designed to Just Work; mobile and
desktop client apps update automatically.
* The server and client apps are backwards and forwards compatible
- The server and client apps are backwards and forwards compatible
across a wide range of versions. So while it's important to upgrade
the server to get security updates, bug fixes, and new features, the
mobile and desktop apps will continue working for at least 18 months
if you don't do so.
* New server releases are announced via the low-traffic
- New server releases are announced via the low-traffic
[zulip-announce email
list](https://groups.google.com/forum/#!forum/zulip-announce). We
highly recommend subscribing so that you are notified about new
security releases.
* Zulip Cloud runs the branch that will become the next major
- Zulip Cloud runs the branch that will become the next major
server/webapp release, so it is always "newer" than the latest
stable release.
@@ -28,16 +28,16 @@ server repository][zulip-server].
### Stable releases
* Zulip Server **stable releases**, such as Zulip 4.5.
- Zulip Server **stable releases**, such as Zulip 4.5.
Organizations self-hosting Zulip primarily use stable releases.
* The numbering scheme is simple: the first digit indicates the major
- The numbering scheme is simple: the first digit indicates the major
release series (which we'll refer to as "4.x"). (Before Zulip 3.0,
Zulip versions had another digit, e.g. 1.9.2 was a bug fix release
in the Zulip 1.9.x major release series).
* [New major releases][blog-major-releases], like Zulip 4.0, are
- [New major releases][blog-major-releases], like Zulip 4.0, are
published every 3-6 months, and contain hundreds of features, bug
fixes, and improvements to Zulip's internals.
* New maintenance releases, like 4.3, are published roughly once a
- New maintenance releases, like 4.3, are published roughly once a
month. Maintenance releases are designed to have no risky changes
and be easy to reverse, to minimize stress for administrators. When
upgrading to a new major release series, We recommend always
@@ -60,25 +60,25 @@ the Zulip server itself (E.g. `https://zulip.example.com/help/`).
Many Zulip servers run versions from Git that have not been published
in a stable release.
* [Zulip Cloud](https://zulip.com) essentially runs the master
branch. It is usually a few days behind master (with some
- [Zulip Cloud](https://zulip.com) essentially runs the `main`
branch. It is usually a few days behind `main` (with some
cherry-picked bug fixes), but can fall up to 2 weeks behind when
major UI or internals changes mean we'd like to bake changes longer
on chat.zulip.org before exposing them to the full Zulip Cloud
userbase.
* [chat.zulip.org][chat-zulip-org], the bleeding-edge server for the
Zulip development community, is upgraded to master several times
every week. We also often "test deploy" changes not yet in master
- [chat.zulip.org][chat-zulip-org], the bleeding-edge server for the
Zulip development community, is upgraded to `main` several times
every week. We also often "test deploy" changes not yet in `main`
to chat.zulip.org to facilitate design feedback.
* We maintain Git branches with names like `4.x` containing backported
commits from master that we plan to include in the next maintenance
- We maintain Git branches with names like `4.x` containing backported
commits from `main` that we plan to include in the next maintenance
release. Self hosters can [upgrade][upgrade-from-git] to these
stable release branches to get bug fixes staged for the next stable
release (which is very useful when you reported a bug whose fix we
choose to backport). We support these branches as though they were a
stable release.
* Self-hosters who want new features not yet present in a major
release can [upgrade to master][upgrading-to-master] or run [a fork
- Self-hosters who want new features not yet present in a major
release can [upgrade to `main`][upgrading-to-main] or run [a fork
of Zulip][fork-zulip].
### Compatibility and upgrading
@@ -91,13 +91,13 @@ process](../production/upgrade-or-modify.md) Just Works.
The Zulip server and clients apps are all carefully engineered to
ensure compatibility with old versions. In particular:
* The Zulip mobile and desktop apps maintain backwards-compatibility
- The Zulip mobile and desktop apps maintain backwards-compatibility
code to support any Zulip server since 2.1.0. (They may also work
with older versions, with a degraded experience).
* Zulip maintains an [API changelog](https://zulip.com/api/changelog)
- Zulip maintains an [API changelog](https://zulip.com/api/changelog)
detailing all changes to the API to make it easy for client
developers to do this correctly.
* The Zulip server preserves backwards-compatibility in its API to
- The Zulip server preserves backwards-compatibility in its API to
support versions of the mobile and desktop apps released in roughly
the last year. Because these clients auto-update, generally there
are only a handful of active clients left by the time we desupport a
@@ -117,7 +117,7 @@ bug fix release, transparently documenting the issue(s) using the
industry-standard [CVE advisory process](https://cve.mitre.org/).
When new security releases are published, we simultaneously publish
the fixes to the `master` and stable release branches (E.g. `4.x`), so
the fixes to the `main` and stable release branches (E.g. `4.x`), so
that anyone using those branches can immediately upgrade as well.
See also our [security model][security-model] documentation.
@@ -130,9 +130,9 @@ Starting with Zulip 4.0, the Zulip webapp will display a banner
warning users of a server running a Zulip release that is more than 18
months old. We do this for a few reasons:
* It is unlikely that a server of that age is not vulnerable to
- It is unlikely that a server of that age is not vulnerable to
a security bug in Zulip or one of its dependencies.
* The Zulip mobile and desktop apps are only guaranteed to support
- The Zulip mobile and desktop apps are only guaranteed to support
server versions less than 18 months old.
The nag will appear only to organization administrators starting a
@@ -163,11 +163,11 @@ releases, and do not support them in production.
The Zulip server project uses several GitHub labels to structure
communication within the project about priorities:
* The [high priority][label-high] label tags issues that we consider
- The [high priority][label-high] label tags issues that we consider
important. This label is meant to be a determination of importance
that can be done quickly and then used as an input to planning
processes.
* The [release goal][label-release-goal] label is used for work that
- The [release goal][label-release-goal] label is used for work that
we hope to include in the next major release. The related [post
release][label-post-release] label is used to track work we want to
focus on shortly after the next major release.
@@ -192,12 +192,12 @@ Zulip's client apps officially support all Zulip server versions (and
Git commits) released in the previous 18 months, matching the behavior
of our [upgrade nag](#upgrade-nag).
* The Zulip mobile apps release new versions from the development
- The Zulip mobile apps release new versions from the development
branch frequently (usually every couple weeks). Except when fixing a
critical bug, releases are first published to our [beta
channels][mobile-beta].
* The Zulip desktop apps are implemented in [Electron][electron], the
- The Zulip desktop apps are implemented in [Electron][electron], the
browser-based desktop application framework used by essentially all
modern chat applications. The Zulip UI in these apps is served from
the Zulip server (and thus can vary between tabs when it is
@@ -227,19 +227,13 @@ core community, like the Python and JavaScript bindings, are released
independently as needed.
[electron]: https://www.electronjs.org/
[upgrading-to-master]: ../production/upgrade-or-modify.html#upgrading-to-master
[upgrading-to-main]: ../production/upgrade-or-modify.html#upgrading-to-main
[os-upgrade]: ../production/upgrade-or-modify.html#upgrading-the-operating-system
[chat-zulip-org]: ../contributing/chat-zulip-org.md
[fork-zulip]: ../production/upgrade-or-modify.html#modifying-zulip
[zulip-server]: https://github.com/zulip/zulip
[mobile-beta]: https://github.com/zulip/zulip-mobile#using-the-beta
[label-blocker]:
https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+blocker%22
[label-high]:
https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+high%22
[label-release-goal]:
https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22release+goal%22
[label-post-release]:
https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22post+release%22
[label-blocker]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+blocker%22
[label-high]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+high%22
[label-release-goal]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22release+goal%22
[label-post-release]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22post+release%22

View File

@@ -26,15 +26,18 @@ creating the initial realm and user. You can disable it after that.
With just a few lines of configuration, your Zulip server can
authenticate users with any of several single-sign-on (SSO)
authentication providers:
* Google accounts, with `GoogleAuthBackend`
* GitHub accounts, with `GitHubAuthBackend`
* GitLab accounts, with `GitLabAuthBackend`
* Microsoft Azure Active Directory, with `AzureADAuthBackend`
- Google accounts, with `GoogleAuthBackend`
- GitHub accounts, with `GitHubAuthBackend`
- GitLab accounts, with `GitLabAuthBackend`
- Microsoft Azure Active Directory, with `AzureADAuthBackend`
Each of these requires one to a handful of lines of configuration in
`settings.py`, as well as a secret in `zulip-secrets.conf`. Details
are documented in your `settings.py`.
(ldap)=
## LDAP (including Active Directory)
Zulip supports retrieving information about users via LDAP, and
@@ -48,20 +51,22 @@ In either configuration, you will need to do the following:
organization using LDAP authentication.
1. Tell Zulip how to connect to your LDAP server:
* Fill out the section of your `/etc/zulip/settings.py` headed "LDAP
- Fill out the section of your `/etc/zulip/settings.py` headed "LDAP
integration, part 1: Connecting to the LDAP server".
* If a password is required, put it in
- If a password is required, put it in
`/etc/zulip/zulip-secrets.conf` by setting
`auth_ldap_bind_password`. For example: `auth_ldap_bind_password
= abcd1234`.
`auth_ldap_bind_password`. For example:
`auth_ldap_bind_password = abcd1234`.
1. Decide how you want to map the information in your LDAP database to
users' account data in Zulip. For each Zulip user, two closely
related concepts are:
* their **email address**. Zulip needs this in order to send, for
- their **email address**. Zulip needs this in order to send, for
example, a notification when they're offline and another user
sends a PM.
* their **Zulip username**. This means the name the user types into the
- their **Zulip username**. This means the name the user types into the
Zulip login form. You might choose for this to be the user's
email address (`sam@example.com`), or look like a traditional
"username" (`sam`), or be something else entirely, depending on
@@ -76,35 +81,38 @@ In either configuration, you will need to do the following:
(A) Using email addresses as Zulip usernames, if LDAP has each
user's email address:
* Make `AUTH_LDAP_USER_SEARCH` a query by email address.
* Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to the same query with
- Make `AUTH_LDAP_USER_SEARCH` a query by email address.
- Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to the same query with
`%(email)s` rather than `%(user)s` as the search parameter.
* Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
- Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
attribute for the user's LDAP username in the search result
for `AUTH_LDAP_REVERSE_EMAIL_SEARCH`.
(B) Using LDAP usernames as Zulip usernames, with email addresses
formed consistently like `sam` -> `sam@example.com`:
* Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
* Set `LDAP_APPEND_DOMAIN = "example.com"`.
- Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
- Set `LDAP_APPEND_DOMAIN = "example.com"`.
(C) Using LDAP usernames as Zulip usernames, with email addresses
taken from some other attribute in LDAP (for example, `mail`):
* Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
* Set `LDAP_EMAIL_ATTR = "mail"`.
* Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find
- Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
- Set `LDAP_EMAIL_ATTR = "mail"`.
- Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find
an LDAP user given their email address (i.e. a search by
`LDAP_EMAIL_ATTR`). For example:
```
```python
AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
ldap.SCOPE_SUBTREE, "(mail=%(email)s)")
```
* Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
- Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
attribute for the user's LDAP username in that search result.
You can quickly test whether your configuration works by running:
```
```bash
/home/zulip/deployments/current/manage.py query_ldap username
```
@@ -115,8 +123,9 @@ email address, if it isn't the same as the "Zulip username").
**Active Directory**: Most Active Directory installations will use one
of the following configurations:
* To access by Active Directory username:
```
- To access by Active Directory username:
```python
AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
@@ -124,8 +133,8 @@ of the following configurations:
AUTH_LDAP_USERNAME_ATTR = "sAMAccountName"
```
* To access by Active Directory email address:
```
- To access by Active Directory email address:
```python
AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
ldap.SCOPE_SUBTREE, "(mail=%(user)s)")
AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
@@ -154,7 +163,7 @@ Zulip can automatically synchronize data declared in
`AUTH_LDAP_USER_ATTR_MAP` from LDAP into Zulip, via the following
management command:
```
```bash
/home/zulip/deployments/current/manage.py sync_ldap_user_data
```
@@ -165,11 +174,12 @@ We recommend running this command in a **regular cron job**, to pick
up changes made on your LDAP server.
All of these data synchronization options have the same model:
* New users will be populated automatically with the
- New users will be populated automatically with the
name/avatar/etc. from LDAP (as configured) on account creation.
* The `manage.py sync_ldap_user_data` cron job will automatically
- The `manage.py sync_ldap_user_data` cron job will automatically
update existing users with any changes that were made in LDAP.
* You can easily test your configuration using `manage.py query_ldap`.
- You can easily test your configuration using `manage.py query_ldap`.
Once you're happy with the configuration, remember to restart the
Zulip server with
`/home/zulip/deployments/current/scripts/restart-server` so that
@@ -208,11 +218,11 @@ to the `AUTH_LDAP_USER_ATTR_MAP`.
Starting with Zulip 2.0, Zulip supports synchronizing the
disabled/deactivated status of users from Active Directory. You can
configure this by uncommenting the sample line `"userAccountControl":
"userAccountControl",` in `AUTH_LDAP_USER_ATTR_MAP` (and restarting
the Zulip server). Zulip will then treat users that are disabled via
the "Disable Account" feature in Active Directory as deactivated in
Zulip.
configure this by uncommenting the sample line
`"userAccountControl": "userAccountControl",` in
`AUTH_LDAP_USER_ATTR_MAP` (and restarting the Zulip server). Zulip
will then treat users that are disabled via the "Disable Account"
feature in Active Directory as deactivated in Zulip.
Users disabled in active directory will be immediately unable to log in
to Zulip, since Zulip queries the LDAP/Active Directory server on
@@ -242,7 +252,7 @@ the next time your `manage.py sync_ldap_user_data` cron job runs.
Other fields you may want to sync from LDAP include:
* Boolean flags; `is_realm_admin` (the organization's administrator
- Boolean flags; `is_realm_admin` (the organization's administrator
permission) is the main one. You can use the
[AUTH_LDAP_USER_FLAGS_BY_GROUP][django-auth-booleans] feature of
`django-auth-ldap` to configure a group to get this permissions.
@@ -250,9 +260,9 @@ Other fields you may want to sync from LDAP include:
`is_active` because deactivating a user this way would not disable
any active sessions the user might have; see the above discussion of
automatic deactivation for how to do that properly).
* String fields like `default_language` (e.g. `en`) or `timezone`, if
- String fields like `default_language` (e.g. `en`) or `timezone`, if
you have that data in the right format in your LDAP database.
* [Coming soon][custom-profile-fields-ldap]: Support for syncing
- [Coming soon][custom-profile-fields-ldap]: Support for syncing
[custom profile fields](https://zulip.com/help/add-custom-profile-fields)
from your LDAP database.
@@ -260,14 +270,15 @@ You can look at the [full list of fields][models-py] in the Zulip user
model; search for `class UserProfile`, but the above should cover all
the fields that would be useful to sync from your LDAP databases.
[models-py]: https://github.com/zulip/zulip/blob/master/zerver/models.py
[models-py]: https://github.com/zulip/zulip/blob/main/zerver/models.py
[django-auth-booleans]: https://django-auth-ldap.readthedocs.io/en/latest/users.html#easy-attributes
[custom-profile-fields-ldap]: https://github.com/zulip/zulip/issues/10976
### Multiple LDAP searches
To do the union of multiple LDAP searches, use `LDAPSearchUnion`. For example:
```
```python
AUTH_LDAP_USER_SEARCH = LDAPSearchUnion(
LDAPSearch("ou=users,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(uid=%(user)s)"),
LDAPSearch("ou=otherusers,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(uid=%(user)s)"),
@@ -297,7 +308,8 @@ For the root subdomain, `www` in the list will work, or any other of
For example, with `org_membership` set to `department`, a user with
the following attributes will have access to the root and `engineering` subdomains:
```
```text
...
department: engineering
department: www
@@ -318,12 +330,11 @@ organization list with multiple maps, that contain a map with an attribute, and
value for that attribute. If for any of the attribute maps, all user's
LDAP attributes match what is configured, access is granted.
```eval_rst
.. warning::
:::{warning}
Restricting access using these mechanisms only affects authentication via LDAP,
and won't prevent users from accessing the organization using any other
authentication backends that are enabled for the organization.
```
:::
### Troubleshooting
@@ -331,16 +342,16 @@ Most issues with LDAP authentication are caused by misconfigurations of
the user and email search settings. Some things you can try to get to
the bottom of the problem:
* Review the instructions for the LDAP configuration type you're
- Review the instructions for the LDAP configuration type you're
using: (A), (B) or (C) (described above), and that you have
configured all of the required settings documented in the
instructions for that configuration type.
* Use the `manage.py query_ldap` tool to verify your configuration.
- Use the `manage.py query_ldap` tool to verify your configuration.
The output of the command will usually indicate the cause of any
configuration problem. For the LDAP integration to work, this
command should be able to successfully fetch a complete, correct set
of data for the queried user.
* You can find LDAP-specific logs in `/var/log/zulip/ldap.log`. If
- You can find LDAP-specific logs in `/var/log/zulip/ldap.log`. If
you're asking for help with your setup, please provide logs from
this file (feel free to anonymize any email addresses to
`username@example.com`) in your report.
@@ -361,15 +372,14 @@ it as follows:
1. Tell your IdP how to find your Zulip server:
* **SP Entity ID**: `https://yourzulipdomain.example.com`.
- **SP Entity ID**: `https://yourzulipdomain.example.com`.
The `Entity ID` should match the value of
`SOCIAL_AUTH_SAML_SP_ENTITY_ID` computed in the Zulip settings.
You can get the correct value by running the following:
`/home/zulip/deployments/current/scripts/get-django-setting
SOCIAL_AUTH_SAML_SP_ENTITY_ID`.
`/home/zulip/deployments/current/scripts/get-django-setting SOCIAL_AUTH_SAML_SP_ENTITY_ID`.
* **SSO URL**:
- **SSO URL**:
`https://yourzulipdomain.example.com/complete/saml/`. This is
the "SAML ACS url" in SAML terminology.
@@ -379,13 +389,14 @@ it as follows:
if `SOCIAL_AUTH_SUBDOMAIN="auth"` and `EXTERNAL_HOST=zulip.example.com`,
this should be `https://auth.zulip.example.com/complete/saml/`.
2. Tell Zulip how to connect to your SAML provider(s) by filling
1. Tell Zulip how to connect to your SAML provider(s) by filling
out the section of `/etc/zulip/settings.py` on your Zulip server
with the heading "SAML Authentication".
* You will need to update `SOCIAL_AUTH_SAML_ORG_INFO` with your
- You will need to update `SOCIAL_AUTH_SAML_ORG_INFO` with your
organization name (`displayname` may appear in the IdP's
authentication flow; `name` won't be displayed to humans).
* Fill out `SOCIAL_AUTH_SAML_ENABLED_IDPS` with data provided by
- Fill out `SOCIAL_AUTH_SAML_ENABLED_IDPS` with data provided by
your identity provider. You may find [the python-social-auth
SAML
docs](https://python-social-auth.readthedocs.io/en/latest/backends/saml.html)
@@ -406,8 +417,10 @@ it as follows:
user ID) and name for the user.
5. The `display_name` and `display_icon` fields are used to
display the login/registration buttons for the IdP.
6. The `auto_signup` field determines how Zulip should handle
login attempts by users who don't have an account yet.
3. Install the certificate(s) required for SAML authentication. You
1. Install the certificate(s) required for SAML authentication. You
will definitely need the public certificate of your IdP. Some IdP
providers also support the Zulip server (Service Provider) having
a certificate used for encryption and signing. We detail these
@@ -423,13 +436,13 @@ it as follows:
trust, which consists of multiple certificates.
4. Set the proper permissions on these files and directories:
```
```bash
chown -R zulip.zulip /etc/zulip/saml/
find /etc/zulip/saml/ -type f -exec chmod 644 -- {} +
chmod 640 /etc/zulip/saml/zulip-private-key.key
```
4. (Optional) If you configured the optional public and private server
1. (Optional) If you configured the optional public and private server
certificates above, you can enable the additional setting
`"authnRequestsSigned": True` in `SOCIAL_AUTH_SAML_SECURITY_CONFIG`
to have the SAMLRequests the server will be issuing to the IdP
@@ -438,15 +451,15 @@ it as follows:
assertions in the SAMLResponses the IdP will send about
authenticated users.
5. Enable the `zproject.backends.SAMLAuthBackend` auth backend, in
1. Enable the `zproject.backends.SAMLAuthBackend` auth backend, in
`AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`.
6. [Restart the Zulip server](../production/settings.md) to ensure
1. [Restart the Zulip server](../production/settings.md) to ensure
your settings changes take effect. The Zulip login page should now
have a button for SAML authentication that you can use to log in or
create an account (including when creating a new organization).
7. If the configuration was successful, the server's metadata can be
1. If the configuration was successful, the server's metadata can be
found at `https://yourzulipdomain.example.com/saml/metadata.xml`. You
can use this for verifying your configuration or provide it to your
IdP.
@@ -465,15 +478,11 @@ correctly authenticate the user to Zulip.
If you're hosting multiple organizations and thus using the
`SOCIAL_AUTH_SUBDOMAIN` setting, you'll need to configure a custom
`RelayState` in your IdP of the form `{"subdomain":
"yourzuliporganization"}` to let Zulip know which organization to
authenticate the user to when they visit your SSO URL from the IdP.
(If the organization is on the root domain, use the empty string:
`{"subdomain": ""}`.).
```eval_rst
.. _ldap:
```
`RelayState` in your IdP of the form
`{"subdomain": "yourzuliporganization"}` to let Zulip know which
organization to authenticate the user to when they visit your SSO URL
from the IdP. (If the organization is on the root domain, use the
empty string: `{"subdomain": ""}`.).
### Restricting access to specific organizations
@@ -491,7 +500,7 @@ For example, with `attr_org_membership` set to `member`, a user with
the following attribute in their `AttributeStatement` will have access
to the root and `engineering` subdomains:
```
```xml
<saml2:Attribute Name="member" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
<saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">
www
@@ -502,7 +511,6 @@ to the root and `engineering` subdomains:
</saml2:Attribute>
```
## Apache-based SSO with `REMOTE_USER`
If you have any existing SSO solution where a preferred way to deploy
@@ -514,17 +522,17 @@ straightforward way to deploy that SSO solution with Zulip.
1. In `/etc/zulip/settings.py`, configure two settings:
* `AUTHENTICATION_BACKENDS`: `'zproject.backends.ZulipRemoteUserBackend'`,
- `AUTHENTICATION_BACKENDS`: `'zproject.backends.ZulipRemoteUserBackend'`,
and no other entries.
* `SSO_APPEND_DOMAIN`: see documentation in `settings.py`.
- `SSO_APPEND_DOMAIN`: see documentation in `settings.py`.
Make sure that you've restarted the Zulip server since making this
configuration change.
2. Edit `/etc/zulip/zulip.conf` and change the `puppet_classes` line to read:
```
```ini
puppet_classes = zulip::profile::standalone, zulip::apache_sso
```
@@ -542,7 +550,8 @@ straightforward way to deploy that SSO solution with Zulip.
using the `htpasswd` example configuration and demonstrate that
working end-to-end, before returning later to configure your SSO
solution. You can do that with the following steps:
```
```bash
/home/zulip/deployments/current/scripts/restart-server
cd /etc/apache2/sites-available/
cp zulip-sso.example zulip-sso.conf
@@ -552,8 +561,8 @@ straightforward way to deploy that SSO solution with Zulip.
5. Run `a2ensite zulip-sso` to enable the SSO integration within Apache.
6. Run `service apache2 reload` to use your new configuration. If
Apache isn't already running, you may need to run `service apache2
start` instead.
Apache isn't already running, you may need to run
`service apache2 start` instead.
Now you should be able to visit your Zulip server in a browser (e.g.,
at `https://zulip.example.com/`) and log in via the SSO solution.
@@ -564,23 +573,23 @@ Most issues with this setup tend to be subtle issues with the
hostname/DNS side of the configuration. Suggestions for how to
improve this SSO setup documentation are very welcome!
* For example, common issues have to do with `/etc/hosts` not mapping
- For example, common issues have to do with `/etc/hosts` not mapping
`settings.EXTERNAL_HOST` to the Apache listening on
`127.0.0.1`/`localhost`.
* While debugging, it can often help to temporarily change the Apache
- While debugging, it can often help to temporarily change the Apache
config in `/etc/apache2/sites-available/zulip-sso` to listen on all
interfaces rather than just `127.0.0.1`.
* While debugging, it can also be helpful to change `proxy_pass` in
- While debugging, it can also be helpful to change `proxy_pass` in
`/etc/nginx/zulip-include/app.d/external-sso.conf` to point to a
more explicit URL, possibly not over HTTPS.
* The following log files can be helpful when debugging this setup:
- The following log files can be helpful when debugging this setup:
* `/var/log/zulip/{errors.log,server.log}` (the usual places)
* `/var/log/nginx/access.log` (nginx access logs)
* `/var/log/apache2/zulip_auth_access.log` (from the
- `/var/log/zulip/{errors.log,server.log}` (the usual places)
- `/var/log/nginx/access.log` (nginx access logs)
- `/var/log/apache2/zulip_auth_access.log` (from the
`zulip-sso.conf` Apache config file; you may want to change
`LogLevel` in that file to "debug" to make this more verbose)
@@ -591,7 +600,7 @@ assuming you're using the example configuration with HTTP basic auth.
This summary should help with understanding what's going on as you try
to debug.
* Since you've configured `/etc/zulip/settings.py` to only define the
- Since you've configured `/etc/zulip/settings.py` to only define the
`zproject.backends.ZulipRemoteUserBackend`,
`zproject/computed_settings.py` configures `/accounts/login/sso/` as
`HOME_NOT_LOGGED_IN`. This makes `https://zulip.example.com/`
@@ -599,12 +608,12 @@ to debug.
nginx) redirect to `/accounts/login/sso/` for a user that isn't
logged in.
* nginx proxies requests to `/accounts/login/sso/` to an Apache
- nginx proxies requests to `/accounts/login/sso/` to an Apache
instance listening on `localhost:8888`, via the config in
`/etc/nginx/zulip-include/app.d/external-sso.conf` (using the
upstream `localhost_sso`, defined in `/etc/nginx/zulip-include/upstreams`).
* The Apache `zulip-sso` site which you've enabled listens on
- The Apache `zulip-sso` site which you've enabled listens on
`localhost:8888` and (in the example config) presents the `htpasswd`
dialogue. (In a real configuration, it takes the user through
whatever more complex interaction your SSO solution performs.) The
@@ -615,7 +624,7 @@ to debug.
variable and either logs the user in or, if they don't have an
account already, registers them. The login sets a cookie.
* After succeeding, that redirects the user back to `/` on port 443.
- After succeeding, that redirects the user back to `/` on port 443.
This request is sent by nginx to the main Zulip Django app, which
sees the cookie, treats them as logged in, and proceeds to serve
them the main app page normally.
@@ -636,23 +645,24 @@ domain for your server).
`/etc/zulip/apple-auth-key.p8`. Be sure to set
permissions correctly:
```
```bash
chown zulip:zulip /etc/zulip/apple-auth-key.p8
chmod 640 /etc/zulip/apple-auth-key.p8
```
1. Configure Apple authentication in `/etc/zulip/settings.py`:
* `SOCIAL_AUTH_APPLE_TEAM`: Your Team ID from Apple, which is a
- `SOCIAL_AUTH_APPLE_TEAM`: Your Team ID from Apple, which is a
string like "A1B2C3D4E5".
* `SOCIAL_AUTH_APPLE_SERVICES_ID`: The Services ID you created in
- `SOCIAL_AUTH_APPLE_SERVICES_ID`: The Services ID you created in
step 1, which might look like "com.example.services".
* `SOCIAL_AUTH_APPLE_APP_ID`: The App ID, or Bundle ID, of your
- `SOCIAL_AUTH_APPLE_APP_ID`: The App ID, or Bundle ID, of your
app that you used in step 1 to configure your Services ID.
This might look like "com.example.app".
* `SOCIAL_AUTH_APPLE_KEY`: Despite the name this is not a key, but
- `SOCIAL_AUTH_APPLE_KEY`: Despite the name this is not a key, but
rather the Key ID of the key you created in step 2. This looks
like "F6G7H8I9J0".
* `AUTHENTICATION_BACKENDS`: Uncomment (or add) a line like
- `AUTHENTICATION_BACKENDS`: Uncomment (or add) a line like
`'zproject.backends.AppleAuthBackend',` to enable Apple auth
using the created configuration.

View File

@@ -11,7 +11,7 @@ something more complicated. This page documents the options for doing so.
To install a development version of Zulip from Git, just clone the Git
repository from GitHub:
```
```bash
# First, install Git if you don't have it installed already
sudo apt install git
git clone https://github.com/zulip/zulip.git zulip-server-git
@@ -21,19 +21,19 @@ and then
[continue the normal installation instructions](../production/install.html#step-2-install-zulip).
You can also [upgrade Zulip from Git](../production/upgrade-or-modify.html#upgrading-from-a-git-repository).
The most common use case for this is upgrading to `master` to get a
The most common use case for this is upgrading to `main` to get a
feature that hasn't made it into an official release yet (often
support for a new base OS release). See [upgrading to
master][upgrade-to-master] for notes on how `master` works and the
main][upgrade-to-main] for notes on how `main` works and the
support story for it, and [upgrading to future
releases][upgrade-to-future-release] for notes on upgrading Zulip
afterwards.
In particular, we are always very glad to investigate problems with
installing Zulip from `master`; they are rare and help us ensure that
installing Zulip from `main`; they are rare and help us ensure that
our next major release has a reliable install experience.
[upgrade-to-master]: ../production/upgrade-or-modify.html#upgrading-to-master
[upgrade-to-main]: ../production/upgrade-or-modify.html#upgrading-to-main
[upgrade-to-future-release]: ../production/upgrade-or-modify.html#upgrading-to-future-releases
## Zulip in Docker
@@ -51,21 +51,21 @@ The Zulip installer supports the following advanced installer options
as well as those mentioned in the
[install](../production/install.html#installer-options) documentation:
* `--postgresql-version`: Sets the version of PostgreSQL that will be
- `--postgresql-version`: Sets the version of PostgreSQL that will be
installed. We currently support PostgreSQL 10, 11, 12, and 13.
* `--postgresql-missing-dictionaries`: Set
- `--postgresql-missing-dictionaries`: Set
`postgresql.missing_dictionaries` ([docs][doc-settings]) in the
Zulip settings, which omits some configuration needed for full-text
indexing. This should be used with [cloud managed databases like
RDS](#using-zulip-with-amazon-rds-as-the-database). This option
conflicts with `--no-overwrite-settings`.
* `--no-init-db`: This option instructs the installer to not do any
- `--no-init-db`: This option instructs the installer to not do any
database initialization. This should be used when you already have a
Zulip database.
* `--no-overwrite-settings`: This option preserves existing
- `--no-overwrite-settings`: This option preserves existing
`/etc/zulip` configuration files.
## Running Zulip's service dependencies on different machines
@@ -86,7 +86,7 @@ configuration to be completely modular.
For example, to install a Zulip Redis server on a machine, you can run
the following after unpacking a Zulip production release tarball:
```
```bash
env PUPPET_CLASSES=zulip::profile::redis ./scripts/setup/install
```
@@ -119,7 +119,7 @@ Follow the [standard instructions](../production/install.md), with one
change. When running the installer, pass the `--no-init-db`
flag, e.g.:
```
```bash
sudo -s # If not already root
./zulip-server-*/scripts/setup/install --certbot \
--email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
@@ -130,7 +130,7 @@ The script also installs and starts PostgreSQL on the server by
default. We don't need it, so run the following command to
stop and disable the local PostgreSQL server.
```
```bash
sudo service postgresql stop
sudo update-rc.d postgresql disable
```
@@ -142,9 +142,9 @@ This complication will be removed in a future version.
Access an administrative `psql` shell on your PostgreSQL database, and
run the commands in `scripts/setup/create-db.sql` to:
* Create a database called `zulip`.
* Create a user called `zulip`.
* Now log in with the `zulip` user to create a schema called
- Create a database called `zulip`.
- Create a user called `zulip`.
- Now log in with the `zulip` user to create a schema called
`zulip` in the `zulip` database. You might have to grant `create`
privileges first for the `zulip` user to do this.
@@ -159,21 +159,21 @@ In `/etc/zulip/settings.py` on your Zulip server, configure the
following settings with details for how to connect to your PostgreSQL
server. Your database provider should provide these details.
* `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
* `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server.
* `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server.
- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server.
- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server.
If you're using password authentication, you should specify the
password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
follows:
```
```ini
postgres_password = abcd1234
```
Now complete the installation by running the following commands.
```
```bash
# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
@@ -191,7 +191,7 @@ configure that as follows:
with `/home/zulip/deployments/current/scripts/restart-server`.
1. Add the following block to `/etc/zulip/zulip.conf`:
```
```ini
[application_server]
nginx_listen_port = 12345
```
@@ -219,7 +219,8 @@ To use Smokescreen:
1. Add `, zulip::profile::smokescreen` to the list of `puppet_classes`
in `/etc/zulip/zulip.conf`. A typical value after this change is:
```
```ini
puppet_classes = zulip::profile::standalone, zulip::profile::smokescreen
```
@@ -231,12 +232,17 @@ To use Smokescreen:
1. Add the following block to `/etc/zulip/zulip.conf`, substituting in
your proxy's hostname/IP and port:
```
```ini
[http_proxy]
host = 127.0.0.1
port = 4750
```
1. If you intend to also make the Smokescreen install available to
other hosts, set `listen_address` in the same block. Note that you
must control access to the Smokescreen port if you do this, as
failing to do so opens a public HTTP proxy!
1. As root, run
`/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
will compile and install Smokescreen, reconfigure services to use
@@ -275,7 +281,7 @@ HTTP as follows:
1. Add the following block to `/etc/zulip/zulip.conf`:
```
```ini
[application_server]
http_only = true
```
@@ -291,15 +297,16 @@ instead of HTTPS.
### nginx configuration
For `nginx` configuration, there's two things you need to set up:
* The root `nginx.conf` file. We recommend using
- The root `nginx.conf` file. We recommend using
`/etc/nginx/nginx.conf` from your Zulip server for our recommended
settings. E.g. if you don't set `client_max_body_size`, it won't be
possible to upload large files to your Zulip server.
* The `nginx` site-specific configuration (in
- The `nginx` site-specific configuration (in
`/etc/nginx/sites-available`) for the Zulip app. The following
example is a good starting point:
```
```nginx
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
@@ -323,9 +330,9 @@ Don't forget to update `server_name`, `ssl_certificate`,
`ssl_certificate_key` and `proxy_pass` with the appropriate values for
your installation.
[nginx-proxy-longpolling-config]: https://github.com/zulip/zulip/blob/master/puppet/zulip/files/nginx/zulip-include-common/proxy_longpolling
[standalone.pp]: https://github.com/zulip/zulip/blob/master/puppet/zulip/manifests/profile/standalone.pp
[zulipchat-puppet]: https://github.com/zulip/zulip/tree/master/puppet/zulip_ops/manifests
[nginx-proxy-longpolling-config]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-common/proxy_longpolling
[standalone.pp]: https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/profile/standalone.pp
[zulipchat-puppet]: https://github.com/zulip/zulip/tree/main/puppet/zulip_ops/manifests
### Apache2 configuration
@@ -336,23 +343,23 @@ make the following changes in two configuration files.
1. Follow the instructions for [Configure Zulip to allow HTTP](#configuring-zulip-to-allow-http).
2. Add the following to `/etc/zulip/settings.py`:
```
```python
EXTERNAL_HOST = 'zulip.example.com'
ALLOWED_HOSTS = ['zulip.example.com', '127.0.0.1']
USE_X_FORWARDED_HOST = True
```
3. Restart your Zulip server with `/home/zulip/deployments/current/scripts/restart-server`.
4. Create an Apache2 virtual host configuration file, similar to the
following. Place it the appropriate path for your Apache2
installation and enable it (E.g. if you use Debian or Ubuntu, then
place it in `/etc/apache2/sites-available/zulip.example.com.conf`
and then run `a2ensite zulip.example.com && systemctl reload
apache2`):
and then run
`a2ensite zulip.example.com && systemctl reload apache2`):
```
```apache
<VirtualHost *:80>
ServerName zulip.example.com
RewriteEngine On
@@ -393,7 +400,7 @@ make the following changes in two configuration files.
If you want to use HAProxy with Zulip, this `backend` config is a good
place to start.
```
```text
backend zulip
mode http
balance leastconn
@@ -425,9 +432,9 @@ does this.
The key configuration options are, for the `/json/events` and
`/api/1/events` endpoints:
* `proxy_read_timeout 1200;`. It's critical that this be
- `proxy_read_timeout 1200;`. It's critical that this be
significantly above 60s, but the precise value isn't important.
* `proxy_buffering off`. If you don't do this, your `nginx` proxy may
- `proxy_buffering off`. If you don't do this, your `nginx` proxy may
return occasional 502 errors to clients using Zulip's events API.
3. The other tricky failure mode we've seen with `nginx` reverse
@@ -439,7 +446,6 @@ unexpectedly (e.g. pointing to a DNS name that you haven't configured
with multiple IPs for your Zulip machine; sometimes this happens with
IPv6 configuration).
## System and deployment configuration
The file `/etc/zulip/zulip.conf` is used to configure properties of
@@ -456,6 +462,7 @@ The most common is **`zulip::profile::standalone`**, used for a
stand-alone single-host deployment.
[Components](../overview/architecture-overview.html#components) of
that include:
- **`zulip::profile::app_frontend`**
- **`zulip::profile::memcached`**
- **`zulip::profile::postgresql`**
@@ -472,8 +479,6 @@ Set to the string `enabled` if enabling the [multi-language PGroonga
search
extension](../subsystems/full-text-search.html#multi-language-full-text-search).
### `[deployment]`
#### `deploy_options`
@@ -497,8 +502,6 @@ for servers that are upgraded frequently by core Zulip developers.
Default repository URL used when [upgrading from a Git
repository](../production/upgrade-or-modify.html#upgrading-from-a-git-repository).
### `[application_server]`
#### `http_only`
@@ -548,8 +551,6 @@ Override the default uwsgi backlog of 128 connections.
Override the default `uwsgi` (Django) process count of 6 on hosts with
more than 3.5GiB of RAM, 4 on hosts with less.
### `[certbot]`
#### `auto_renew`
@@ -558,8 +559,6 @@ If set to the string `yes`, [Certbot will attempt to automatically
renew its certificate](../production/ssl-certificates.html#certbot-recommended). Do
no set by hand; use `scripts/setup/setup-certbot` to configure this.
### `[postfix]`
#### `mailname`
@@ -610,16 +609,12 @@ connections.
The version of PostgreSQL that is in use. Do not set by hand; use the
[PostgreSQL upgrade tool](../production/upgrade-or-modify.html#upgrading-postgresql).
### `[rabbitmq]`
#### `nodename`
The name used to identify the local RabbitMQ server; do not modify.
### `[memcached]`
#### `memory`
@@ -627,8 +622,6 @@ The name used to identify the local RabbitMQ server; do not modify.
Override the number of megabytes of memory that memcached should be
configured to consume; defaults to 1/8th of the total server memory.
### `[loadbalancer]`
#### `ips`
@@ -636,8 +629,6 @@ configured to consume; defaults to 1/8th of the total server memory.
Comma-separated list of IP addresses or netmasks of external
load balancers whose `X-Forwarded-For` should be respected.
### `[http_proxy]`
#### `host`
@@ -648,3 +639,8 @@ proxy](#using-an-outgoing-http-proxy).
#### `port`
The TCP port of the HTTP `CONNECT` proxy on the host specified above.
#### `listen_address`
The IP address that Smokescreen should bind to and listen on.
Defaults to `127.0.0.1`.

View File

@@ -4,10 +4,10 @@ Zulip's incoming email gateway integration makes it possible to send
messages into Zulip by sending an email. It's highly recommended
because it enables:
* When users reply to one of Zulip's message notification emails
- When users reply to one of Zulip's message notification emails
from their email client, the reply can go directly
into Zulip.
* Integrating third-party services that can send email notifications
- Integrating third-party services that can send email notifications
into Zulip. See the [integration
documentation](https://zulip.com/integrations/doc/email) for
details.
@@ -29,14 +29,14 @@ polling option is convenient for testing/developing this feature
because it doesn't require a public IP address or setting up MX
records in DNS.
```eval_rst
.. note::
:::{note}
Incoming emails are rate-limited, with the following limits:
* 50 emails per minute.
* 120 emails per 5 minutes.
* 600 emails per hour.
```
- 50 emails per minute.
- 120 emails per 5 minutes.
- 600 emails per hour.
:::
## Local delivery setup
@@ -57,7 +57,7 @@ using an [HTTP reverse proxy][reverse-proxy]).
configuring email for `emaildomain.example.com` to be processed by
`hostname.example.com`. You can check your work using this command:
```
```console
$ dig +short emaildomain.example.com -t MX
1 hostname.example.com
```
@@ -66,7 +66,8 @@ using an [HTTP reverse proxy][reverse-proxy]).
1. Add `, zulip::postfix_localmail` to `puppet_classes` in
`/etc/zulip/zulip.conf`. A typical value after this change is:
```
```ini
puppet_classes = zulip::profile::standalone, zulip::postfix_localmail
```
@@ -74,7 +75,7 @@ using an [HTTP reverse proxy][reverse-proxy]).
`emaildomain.example.com`, add a section to `/etc/zulip/zulip.conf`
on your Zulip server like this:
```
```ini
[postfix]
mailname = emaildomain.example.com
```
@@ -112,12 +113,13 @@ Congratulations! The integration should be fully operational.
([Here's how it works with Gmail](https://support.google.com/mail/answer/7126229?hl=en))
1. Configure IMAP access in the appropriate Zulip settings:
* Login and server connection details in `/etc/zulip/settings.py`
- Login and server connection details in `/etc/zulip/settings.py`
in the email gateway integration section (`EMAIL_GATEWAY_LOGIN` and others).
* Password in `/etc/zulip/zulip-secrets.conf` as `email_gateway_password`.
- Password in `/etc/zulip/zulip-secrets.conf` as `email_gateway_password`.
1. Install a cron job to poll the inbox every minute for new messages:
```
```bash
cd /home/zulip/deployments/current/
sudo cp puppet/zulip/files/cron.d/email-mirror /etc/cron.d/
```

View File

@@ -61,14 +61,14 @@ services) have free options. Once you've signed up, you'll want to
find the service's provided "SMTP credentials", and configure Zulip as
follows:
* The hostname like `EMAIL_HOST = 'smtp.mailgun.org'` in `/etc/zulip/settings.py`
* The username like `EMAIL_HOST_USER = 'username@example.com'` in
- The hostname like `EMAIL_HOST = 'smtp.mailgun.org'` in `/etc/zulip/settings.py`
- The username like `EMAIL_HOST_USER = 'username@example.com'` in
`/etc/zulip/settings.py`.
* The TLS setting as `EMAIL_USE_TLS = True` in
- The TLS setting as `EMAIL_USE_TLS = True` in
`/etc/zulip/settings.py`, for most providers
* The port as `EMAIL_PORT = 587` in `/etc/zulip/settings.py`, for most
- The port as `EMAIL_PORT = 587` in `/etc/zulip/settings.py`, for most
providers
* The password like `email_password = abcd1234` in `/etc/zulip/zulip-secrets.conf`.
- The password like `email_password = abcd1234` in `/etc/zulip/zulip-secrets.conf`.
### Using system email
@@ -78,7 +78,7 @@ configuration on the system that forwards email sent locally into your
corporate email system), you will likely need to use something like
these setting values:
```
```python
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25
EMAIL_USE_TLS = False
@@ -100,16 +100,17 @@ We don't recommend using an inbox product like Gmail for outgoing
email, because Gmail's anti-spam measures make this annoying. But if
you want to use a Gmail account to send outgoing email anyway, here's
how to make it work:
* Create a totally new Gmail account for your Zulip server; you don't
- Create a totally new Gmail account for your Zulip server; you don't
want Zulip's automated emails to come from your personal email address.
* If you're using 2-factor authentication on the Gmail account, you'll
- If you're using 2-factor authentication on the Gmail account, you'll
need to use an
[app-specific password](https://support.google.com/accounts/answer/185833).
* If you're not using 2-factor authentication, read this Google
- If you're not using 2-factor authentication, read this Google
support answer and configure that account as
["less secure"](https://support.google.com/accounts/answer/6010255);
Gmail doesn't allow servers to send outgoing email by default.
* Note also that the rate limits for Gmail are also quite low
- Note also that the rate limits for Gmail are also quite low
(e.g. 100 / day), so it's easy to get rate-limited if your server
has significant traffic. For more active servers, we recommend
moving to a free account on a transactional email service.
@@ -122,7 +123,7 @@ can log them to a file instead.
To do so, add these lines to `/etc/zulip/settings.py`:
```
```python
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH = '/var/log/zulip/emails'
```
@@ -137,7 +138,7 @@ later set up a real SMTP provider!
You can quickly test your outgoing email configuration using:
```
```bash
su zulip -c '/home/zulip/deployments/current/manage.py send_test_email user@example.com'
```
@@ -148,11 +149,11 @@ default From address for your Zulip server, and one sent by the
If it doesn't work, check these common failure causes:
* Your hosting provider may block outgoing SMTP traffic in its default
- Your hosting provider may block outgoing SMTP traffic in its default
firewall rules. Check whether the port `EMAIL_PORT` is blocked in
your hosting provider's firewall.
* Your SMTP server's permissions might not allow the email account
- Your SMTP server's permissions might not allow the email account
you're using to send email from the `noreply` email addresses used
by Zulip when sending confirmation emails.
@@ -170,11 +171,11 @@ If it doesn't work, check these common failure causes:
the security issue with helpdesk software that
`ADD_TOKENS_TO_NOREPLY_ADDRESS` helps protect against.
* Make sure you set the password in `/etc/zulip/zulip-secrets.conf`.
- Make sure you set the password in `/etc/zulip/zulip-secrets.conf`.
* Check the username and password for typos.
- Check the username and password for typos.
* Be sure to restart your Zulip server after editing either
- Be sure to restart your Zulip server after editing either
`settings.py` or `zulip-secrets.conf`, using
`/home/zulip/deployments/current/scripts/restart-server` .
Note that the `manage.py` command above will read the latest
@@ -186,24 +187,24 @@ If it doesn't work, check these common failure causes:
Here are a few final notes on what to look at when debugging why you
aren't receiving emails from Zulip:
* Most transactional email services have an "outgoing email" log where
- Most transactional email services have an "outgoing email" log where
you can inspect the emails that reached the service, whether an
email was flagged as spam, etc.
* Starting with Zulip 1.7, Zulip logs an entry in
- Starting with Zulip 1.7, Zulip logs an entry in
`/var/log/zulip/send_email.log` whenever it attempts to send an
email. The log entry includes whether the request succeeded or failed.
* If attempting to send an email throws an exception, a traceback
- If attempting to send an email throws an exception, a traceback
should be in `/var/log/zulip/errors.log`, along with any other
exceptions Zulip encounters.
* If your SMTP provider uses SSL on port 465 (and not TLS on port
- If your SMTP provider uses SSL on port 465 (and not TLS on port
587), you need to set `EMAIL_PORT = 465` as well as replacing
`EMAIL_USE_TLS = True` with `EMAIL_USE_SSL = True`; otherwise, Zulip
will try to use the TLS protocol on port 465, which won't work.
* Zulip's email sending configuration is based on the standard Django
- Zulip's email sending configuration is based on the standard Django
[SMTP backend](https://docs.djangoproject.com/en/2.0/topics/email/#smtp-backend)
configuration. So if you're having trouble getting your email
provider working, you may want to search for documentation related

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -13,12 +13,13 @@ you'd like to watch the downtime phase of the upgrade closely, you
can run them manually before starting the upgrade:
1. Log in to your Zulip server as the `zulip` user (or as `root` and
then run `su zulip` to drop privileges), and `cd
/home/zulip/deployments/current`
then run `su zulip` to drop privileges), and
`cd /home/zulip/deployments/current`
2. Run `./manage.py dbshell`. This will open a shell connected to the
PostgreSQL database.
3. In the PostgreSQL shell, run the following commands:
```postgresql
CREATE INDEX CONCURRENTLY
zerver_usermessage_is_private_message_id
ON zerver_usermessage (user_profile_id, message_id)
@@ -28,9 +29,11 @@ can run them manually before starting the upgrade:
zerver_usermessage_active_mobile_push_notification_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 4096) != 0;
```
(These first migrations are the only new ones in Zulip 1.9).
```postgresql
CREATE INDEX CONCURRENTLY
zerver_usermessage_mentioned_message_id
ON zerver_usermessage (user_profile_id, message_id)
@@ -55,6 +58,7 @@ can run them manually before starting the upgrade:
zerver_usermessage_unread_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 1) = 0;
```
These will take some time to run, during which the server will
continue to serve user traffic as usual with no disruption. Once they

View File

@@ -5,16 +5,16 @@ move data from one Zulip server to another, do backups, compliance
work, or migrate from your own servers to the hosted Zulip Cloud
service (or back):
* The [Backup](#backups) tool is designed for exact restoration of a
- The [Backup](#backups) tool is designed for exact restoration of a
Zulip server's state, for disaster recovery, testing with production
data, or hardware migration. This tool has a few limitations:
* Backups must be restored on a server running the same Zulip
- Backups must be restored on a server running the same Zulip
version (most precisely, one where `manage.py showmigrations` has
the same output).
* Backups must be restored on a server running the same PostgreSQL
- Backups must be restored on a server running the same PostgreSQL
version.
* Backups aren't useful for migrating organizations between
- Backups aren't useful for migrating organizations between
self-hosting and Zulip Cloud (which may require renumbering all
the users/messages/etc.).
@@ -24,7 +24,7 @@ service (or back):
document [backup details](#backup-details) for users managing
backups manually.
* The logical [Data export](#data-export) tool is designed for
- The logical [Data export](#data-export) tool is designed for
migrating data between Zulip Cloud and other Zulip servers, as well
as various auditing purposes. The logical export tool produces a
`.tar.gz` archive with most of the Zulip database data encoded in
@@ -41,13 +41,13 @@ service (or back):
tool isn't applicable, including situations where an easily
machine-parsable export format is desired.
* Zulip also has an [HTML archive
- Zulip also has an [HTML archive
tool](https://github.com/zulip/zulip-archive), which is primarily
intended for public archives, but can also be useful to
inexpensively preserve public stream conversations when
decommissioning a Zulip organization.
* It's possible to set up [PostgreSQL streaming
- It's possible to set up [PostgreSQL streaming
replication](#postgresql-streaming-replication) and the [S3 file
upload
backend](../production/upload-backends.html#s3-backend-configuration)
@@ -57,7 +57,7 @@ service (or back):
The Zulip server has a built-in backup tool:
```
```bash
# As the zulip user
/home/zulip/deployments/current/manage.py backup
# Or as root
@@ -65,6 +65,7 @@ su zulip -c '/home/zulip/deployments/current/manage.py backup'
```
The backup tool provides the following options:
- `--output=/tmp/backup.tar.gz`: Filename to write the backup tarball
to (default: write to a file in `/tmp`). On success, the
console output will show the path to the output tarball.
@@ -84,7 +85,7 @@ First, [install a new Zulip server through Step 3][install-server]
with the same version of both the base OS and Zulip from your previous
installation. Then, run as root:
```
```bash
/home/zulip/deployments/current/scripts/setup/restore-backup /path/to/backup
```
@@ -113,7 +114,7 @@ tarball: `postgres-version`, `os-version`, and `zulip-version`. The
following command may be useful for viewing these files without
extracting the entire archive.
```
```bash
tar -Oaxf /path/to/archive/zulip-backup-rest.tar.gz zulip-backup/zulip-version
```
@@ -128,11 +129,11 @@ server, including the database, settings, secrets from
The following data is not included in these backup archives,
and you may want to back up separately:
* The server access/error logs from `/var/log/zulip`. The Zulip
- The server access/error logs from `/var/log/zulip`. The Zulip
server only appends to logs, and they can be very large compared to
the rest of the data for a Zulip server.
* Files uploaded with the Zulip
- Files uploaded with the Zulip
[S3 file upload backend](../production/upload-backends.md). We
don't include these for two reasons. First, the uploaded file data
in S3 can easily be many times larger than the rest of the backup,
@@ -140,7 +141,7 @@ and you may want to back up separately:
exceed its disk capacity. Additionally, S3 is a reliable persistent
storage system with its own high-quality tools for doing backups.
* SSL certificates. These are not included because they are
- SSL certificates. These are not included because they are
particularly security-sensitive and are either trivially replaced
(if generated via Certbot) or provided by the system administrator.
@@ -159,7 +160,7 @@ backup strategy), and also serves as documentation for what is
included in the backups generated by Zulip's standard tools. The
data includes:
* The PostgreSQL database. You can back this up with any standard
- The PostgreSQL database. You can back this up with any standard
database export or backup tool. Zulip has built-in support for taking
daily incremental backups using
[wal-g](https://github.com/wal-g/wal-g); these backups are stored for
@@ -167,7 +168,7 @@ daily incremental backups using
storing the backups, edit `/etc/zulip/zulip-secrets.conf` on the
PostgreSQL server to add:
```
```ini
s3_backups_key = # aws public key
s3_backups_secret_key = # aws secret key
s3_backups_bucket = # name of S3 backup
@@ -179,12 +180,12 @@ PostgreSQL server to add:
the Nagios plugin installed into
`/usr/lib/nagios/plugins/zulip_postgresql_backups/check_postgresql_backup`.
* Any user-uploaded files. If you're using S3 as storage for file
- Any user-uploaded files. If you're using S3 as storage for file
uploads, this is backed up in S3. But if you have instead set
`LOCAL_UPLOADS_DIR`, any files uploaded by users (including avatars)
will be stored in that directory and you'll want to back it up.
* Your Zulip configuration including secrets from `/etc/zulip/`.
- Your Zulip configuration including secrets from `/etc/zulip/`.
E.g. if you lose the value of `secret_key`, all users will need to
log in again when you set up a replacement server since you won't be
able to verify their cookies. If you lose `avatar_salt`, any
@@ -198,27 +199,27 @@ email), etc.
To restore from a manual backup, the process is basically the reverse of the above:
* Install new server as normal by downloading a Zulip release tarball
- Install new server as normal by downloading a Zulip release tarball
and then using `scripts/setup/install`. You don't need
to run the `initialize-database` second stage which puts default
data into the database.
* Unpack to `/etc/zulip` the `settings.py` and `zulip-secrets.conf` files
- Unpack to `/etc/zulip` the `settings.py` and `zulip-secrets.conf` files
from your backups.
* If you ran `initialize-database` anyway above, you'll want to run
- If you ran `initialize-database` anyway above, you'll want to run
`scripts/setup/postgresql-init-db` to drop the initial database first.
* Restore your database from the backup.
- Restore your database from the backup.
* Reconfigure rabbitmq to use the password from `secrets.conf`
- Reconfigure rabbitmq to use the password from `secrets.conf`
by running, as root, `scripts/setup/configure-rabbitmq`.
* If you're using local file uploads, restore those files to the path
- If you're using local file uploads, restore those files to the path
specified by `settings.LOCAL_UPLOADS_DIR` and (if appropriate) any
logs.
* Start the server using `scripts/restart-server`.
- Start the server using `scripts/restart-server`.
This restoration process can also be used to migrate a Zulip
installation from one server to another.
@@ -233,8 +234,8 @@ that they are up to date using the Nagios plugin at:
Zulip has database configuration for using PostgreSQL streaming
replication. You can see the configuration in these files:
* `puppet/zulip_ops/manifests/profile/postgresql.pp`
* `puppet/zulip_ops/files/postgresql/*`
- `puppet/zulip_ops/manifests/profile/postgresql.pp`
- `puppet/zulip_ops/files/postgresql/*`
We use this configuration for Zulip Cloud, and it works well in
production, but it's not fully generic. Contributions to make it a
@@ -280,7 +281,7 @@ of the lines for the appropriate option.
Log in to a shell on your Zulip server as the `zulip` user. Run the
following commands:
```
```bash
cd /home/zulip/deployments/current
# ./scripts/stop-server
# export DEACTIVATE_FLAG="--deactivate" # Deactivates the organization
@@ -300,22 +301,23 @@ archive of all the organization's uploaded files.
1. [Install a new Zulip server](../production/install.md),
**skipping Step 3** (you'll create your Zulip organization via the data
import tool instead).
* Ensure that the Zulip server you're importing into is running the same
- Ensure that the Zulip server you're importing into is running the same
version of Zulip as the server you're exporting from.
* For exports from Zulip Cloud (zulip.com), you need to [upgrade to
master][upgrade-zulip-from-git], since we run run master on
- For exports from Zulip Cloud (zulip.com), you need to [upgrade to
`main`][upgrade-zulip-from-git], since we run run `main` on
Zulip Cloud:
```
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git master
```bash
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git main
```
It is not sufficient to be on the latest stable release, as
zulip.com runs pre-release versions of Zulip that are often
several months of development ahead of the latest release.
* Note that if your server has limited free RAM, you'll want to
- Note that if your server has limited free RAM, you'll want to
shut down the Zulip server with `./scripts/stop-server` while
you run the import, since our minimal system requirements do not
budget extra RAM for running the data import tool.
@@ -326,10 +328,10 @@ new server to reuse the server-level configuration and secret keys
from your old server. There are a few important details to understand
about doing so:
* Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is
- Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is
safe and recommended. Care is required when copying secrets from
`/etc/zulip/zulip-secrets.conf` (details below).
* If you copy `zulip_org_id` and `zulip_org_key` (the credentials
- If you copy `zulip_org_id` and `zulip_org_key` (the credentials
for the [mobile push notifications
service](../production/mobile-push-notifications.md)), you should
be very careful to make sure the no users had their IDs
@@ -345,15 +347,15 @@ about doing so:
simply to not copy these settings and re-register your server for
mobile push notifications if any users had their IDs renumbered
during the logical export/import process.
* If you copy the `rabbitmq_password` secret from
- If you copy the `rabbitmq_password` secret from
`zulip-secrets.conf`, you'll need to run
`scripts/setup/configure-rabbitmq` to update your local RabbitMQ
installation to use the password in your Zulip secrets file.
* You will likely want to copy `camo_key` (required to avoid
- You will likely want to copy `camo_key` (required to avoid
breaking certain links) and any settings you added related to
authentication and email delivery so that those work on your new
server.
* Copying `avatar_salt` is not recommended, due to similar issues
- Copying `avatar_salt` is not recommended, due to similar issues
to the mobile push notifications service. Zulip will
automatically rewrite avatars at URLs appropriate for the new
user IDs, and using the same avatar salt (and same server URL)
@@ -364,7 +366,7 @@ about doing so:
following commands, replacing the filename with the path to your data
export tarball:
```
```bash
cd ~
tar -xf /path/to/export/file/zulip-export-zcmpxfm6.tar.gz
cd /home/zulip/deployments/current
@@ -386,7 +388,7 @@ custom subdomain, e.g. if you already have an existing organization on the
root domain. Replace the last three lines above with the following, after replacing
`<subdomain>` with the desired subdomain.
```
```bash
./manage.py import <subdomain> ~/zulip-export-zcmpxfm6
./manage.py reactivate_realm -r <subdomain> # Reactivates the organization
```
@@ -403,12 +405,13 @@ You can use the `./manage.py send_password_reset_email` command to
send password reset emails to your users. We
recommend starting with sending one to yourself for testing:
```
```bash
./manage.py send_password_reset_email -u username@example.com
```
and then once you're ready, you can email them to everyone using e.g.
```
```bash
./manage.py send_password_reset_email -r '' --all-users
```
@@ -421,12 +424,12 @@ delete the test import data from your Zulip server before doing a
final import. You can **permanently delete** all data from a Zulip
organization using the following procedure:
* Start a [Zulip management shell](../production/management-commands.html#manage-py-shell)
* In the management shell, run the following commands, replacing `""`
- Start a [Zulip management shell](../production/management-commands.html#manage-py-shell)
- In the management shell, run the following commands, replacing `""`
with the subdomain if [you are hosting the organization on a
subdomain](../production/multiple-organizations.md):
```
```python
realm = Realm.objects.get(string_id="")
realm.delete()
```
@@ -434,7 +437,8 @@ realm.delete()
The output contains details on the objects deleted from the database.
Now, exit the management shell and run this to clear Zulip's cache:
```
```bash
/home/zulip/deployments/current/scripts/setup/flush-memcached
```
@@ -444,7 +448,7 @@ can additionally delete all file uploads, avatars, and custom emoji on
a Zulip server (across **all organizations**) with the following
command:
```
```bash
rm -rf /home/zulip/uploads/*/*
```
@@ -454,7 +458,7 @@ in the management shell before deleting the organization from the
database (this will be `2` for the first organization created on a
Zulip server, shown in the example below), e.g.:
```
```bash
rm -rf /home/zulip/uploads/*/2/
```

View File

@@ -41,7 +41,5 @@ Zulip server. Your users can now use the integration as described in
[the help center article][help-center-giphy]. (A browser reload may
be required).
[help-center-giphy]: https://zulip.com/help/animated-gifs-from-giphy
[giphy-dashboard]: https://developers.giphy.com/dashboard/

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -11,7 +11,7 @@ configuration files in /etc, so we recommend against installing it on
a server running other nginx or django apps.
But if you do, here are some things you can do that may make it
possible to retain your existing site. However, this is *NOT*
possible to retain your existing site. However, this is _NOT_
recommended, and you may break your server. Make sure you have backups
and a provisioning script ready to go to wipe and restore your
existing services if (when) your server goes down.
@@ -19,24 +19,31 @@ existing services if (when) your server goes down.
These instructions are only for experts. If you're not an experienced
Linux sysadmin, you will have a much better experience if you get a
dedicated VM to install Zulip on instead (or [use
zulip.com](https://zulip.com).
zulip.com](https://zulip.com)).
### Nginx
Copy your existing nginx configuration to a backup and then merge the
one created by Zulip into it:
```shell
```bash
sudo cp /etc/nginx/nginx.conf /etc/nginx.conf.before-zulip-install
sudo wget -O /etc/nginx/nginx.conf.zulip \
https://raw.githubusercontent.com/zulip/zulip/master/puppet/zulip/files/nginx/nginx.conf
https://raw.githubusercontent.com/zulip/zulip/main/puppet/zulip/templates/nginx.conf.template.erb
sudo meld /etc/nginx/nginx.conf /etc/nginx/nginx.conf.zulip # be sure to merge to the right
```
Since the file in Zulip is an [ERB Puppet
template](https://puppet.com/docs/puppet/7/lang_template_erb.html),
you will also need to replace any `<%= ... %>` sections with
appropriate content. For instance `<%= @ca_crt %>` should be replaced
with `/etc/ssl/certs/ca-certificates.crt` on Debian and Ubuntu
installs.
After the Zulip installation completes, then you can overwrite (or
merge) your new nginx.conf with the installed one:
```shell
```console
$ sudo meld /etc/nginx/nginx.conf.zulip /etc/nginx/nginx.conf # be sure to merge to the right
$ sudo service nginx restart
```
@@ -51,13 +58,13 @@ If you have a Puppet server running on your server, you will get an
error message about not being able to connect to the client during the
install process:
```shell
```console
puppet-agent[29873]: Could not request certificate: Failed to open TCP connection to puppet:8140
```
So you'll need to shut down any Puppet servers.
```shell
```console
$ sudo service puppet-agent stop
$ sudo service puppet stop
```

View File

@@ -14,19 +14,19 @@ you can create a test organization at <https://zulip.com/new>.
## Step 1: Download the latest release
Download and unpack [the latest built server
tarball](https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz)
with the following commands:
Download and unpack [the latest server
release](https://download.zulip.com/server/zulip-server-latest.tar.gz)
(**Zulip Server {{ LATEST_RELEASE_VERSION }}**) with the following commands:
```
```bash
cd $(mktemp -d)
wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
wget https://download.zulip.com/server/zulip-server-latest.tar.gz
tar -xf zulip-server-latest.tar.gz
```
* If you'd like to verify the download, we
[publish the sha256sums of our release tarballs](https://www.zulip.org/dist/releases/SHA256SUMS.txt).
* You can also
- If you'd like to verify the download, we
[publish the sha256sums of our release tarballs](https://download.zulip.com/server/SHA256SUMS.txt).
- You can also
[install a pre-release version of Zulip](../production/deployment.html#installing-zulip-from-git)
using code from our [repository on GitHub](https://github.com/zulip/zulip/).
@@ -35,7 +35,7 @@ using code from our [repository on GitHub](https://github.com/zulip/zulip/).
To set up Zulip with the most common configuration, you can run the
installer as follows:
```
```bash
sudo -s # If not already root
./zulip-server-*/scripts/setup/install --certbot \
--email=YOUR_EMAIL --hostname=YOUR_HOSTNAME
@@ -48,21 +48,21 @@ If the script gives an error, consult [Troubleshooting](#troubleshooting) below.
#### Installer options
* `--email=you@example.com`: The email address of the person or team
- `--email=you@example.com`: The email address of the person or team
who should get support and error emails from this Zulip server.
This becomes `ZULIP_ADMINISTRATOR` ([docs][doc-settings]) in the
Zulip settings.
* `--hostname=zulip.example.com`: The user-accessible domain name for
- `--hostname=zulip.example.com`: The user-accessible domain name for
this Zulip server, i.e., what users will type in their web browser.
This becomes `EXTERNAL_HOST` ([docs][doc-settings]) in the Zulip
settings.
* `--self-signed-cert`: With this option, the Zulip installer
- `--self-signed-cert`: With this option, the Zulip installer
generates a self-signed SSL certificate for the server. This isn't
suitable for production use, but may be convenient for testing.
* `--certbot`: With this option, the Zulip installer automatically
- `--certbot`: With this option, the Zulip installer automatically
obtains an SSL certificate for the server [using Certbot][doc-certbot].
If you'd prefer to acquire an SSL certificate yourself in any other
way, it's easy to [provide it to Zulip][doc-ssl-manual].
@@ -90,54 +90,56 @@ your organization, and your own user account as an administrator.
Then, log in!
The link is a secure one-time-use link. If you need another
later, you can generate a new one by running `manage.py
generate_realm_creation_link` on the server. See also our doc on
running [multiple organizations on the same server](multiple-organizations.md)
if that's what you're planning to do.
later, you can generate a new one by running
`manage.py generate_realm_creation_link` on the server. See also our
doc on running [multiple organizations on the same
server](multiple-organizations.md) if that's what you're planning to
do.
## Step 4: Configure and use
To really see Zulip in action, you'll need to get the people you work
together with using it with you.
* [Set up outgoing email](email.md) so Zulip can confirm new users'
- [Set up outgoing email](email.md) so Zulip can confirm new users'
email addresses and send notifications.
* Learn how to [get your organization started][realm-admin-docs] using
- Learn how to [get your organization started][realm-admin-docs] using
Zulip at its best.
Learning more:
* Subscribe to the [Zulip announcements email
- Subscribe to the [Zulip announcements email
list](https://groups.google.com/forum/#!forum/zulip-announce) for
server administrators. This extremely low-traffic list is for
important announcements, including [new
releases](../overview/release-lifecycle.md) and security issues. You
can also use the [RSS
feed](https://groups.google.com/forum/#!aboutgroup/zulip-announce).
* Follow [Zulip on Twitter](https://twitter.com/zulip).
* Learn how to [configure your Zulip server settings](settings.md).
* Learn about [Backups, export and import](../production/export-and-import.md)
- Follow [Zulip on Twitter](https://twitter.com/zulip).
- Learn how to [configure your Zulip server settings](settings.md).
- Learn about [Backups, export and import](../production/export-and-import.md)
and [upgrading](../production/upgrade-or-modify.md) a production Zulip
server.
[realm-admin-docs]: https://zulip.com/help/getting-your-organization-started-with-zulip
```eval_rst
.. _installer-details:
```
(installer-details)=
## Details: What the installer does
The install script does several things:
* Creates the `zulip` user, which the various Zulip servers will run as.
* Creates `/home/zulip/deployments/`, which the Zulip code for this
- Creates the `zulip` user, which the various Zulip servers will run as.
- Creates `/home/zulip/deployments/`, which the Zulip code for this
deployment (and future deployments when you upgrade) goes into. At the
very end of the install process, the script moves the Zulip code tree
it's running from (which you unpacked from a tarball above) to a
directory there, and makes `/home/zulip/deployments/current` as a
symbolic link to it.
* Installs Zulip's various dependencies.
* Configures the various third-party services Zulip uses, including
- Installs Zulip's various dependencies.
- Configures the various third-party services Zulip uses, including
PostgreSQL, RabbitMQ, Memcached and Redis.
* Initializes Zulip's database.
- Initializes Zulip's database.
If you'd like to deploy Zulip with these services on different
machines, check out our [deployment options documentation](deployment.md).

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -8,6 +8,7 @@ This was once a long page covering a bunch of topics; those topics
have since all moved to dedicated pages:
### Monitoring
Moved to [Troubleshooting](../production/troubleshooting.html#monitoring).
### Securing your Zulip server

View File

@@ -10,7 +10,7 @@ framework][django-management].
Start by logging in as the `zulip` user on the Zulip server. Then run
them as follows:
```
```bash
cd /home/zulip/deployments/current
# Start by reading the help
@@ -39,7 +39,7 @@ string ID (usually the subdomain).
You can see all the organizations on your Zulip server using
`./manage.py list_realms`.
```
```console
zulip@zulip:~$ /home/zulip/deployments/current/manage.py list_realms
id string_id name
-- --------- ----
@@ -56,7 +56,7 @@ Unless you are
your single Zulip organization on the root domain will have the empty
string (`''`) as its `string_id`. So you can run e.g.:
```
```console
zulip@zulip:~$ /home/zulip/deployments/current/manage.py show_admins -r ''
```
@@ -73,7 +73,7 @@ You can get an IPython shell with full access to code within the Zulip
project using `manage.py shell`, e.g., you can do the following to
change a user's email address:
```
```console
$ cd /home/zulip/deployments/current/
$ ./manage.py shell
In [1]: user_profile = get_user_profile_by_email("email@example.com")
@@ -104,34 +104,37 @@ access other functions, you'll need to import them yourself.
There are dozens of useful management commands under
`zerver/management/commands/`. We detail a few here:
* `./manage.py help`: Lists all available management commands.
* `./manage.py dbshell`: If you're more comfortable with raw SQL than
- `./manage.py help`: Lists all available management commands.
- `./manage.py dbshell`: If you're more comfortable with raw SQL than
Python, this will open a PostgreSQL SQL shell connected to the Zulip
server's database. Beware of changing data; editing data directly
with SQL will often not behave correctly because PostgreSQL doesn't
know to flush Zulip's caches or notify browsers of changes.
* `./manage.py send_custom_email`: Can be used to send an email to a set
of users. The `--help` documents how to run it from a `manage.py
shell` for use with more complex programmatically computed sets of
users.
* `./manage.py send_password_reset_email`: Sends password reset email(s)
- `./manage.py send_custom_email`: Can be used to send an email to a set
of users. The `--help` documents how to run it from a
`manage.py shell` for use with more complex programmatically
computed sets of users.
- `./manage.py send_password_reset_email`: Sends password reset email(s)
to one or more users.
* `./manage.py change_realm_subdomain`: Change subdomain of a realm.
* `./manage.py change_user_email`: Change a user's email address.
* `./manage.py change_user_role`: Can change are user's role
- `./manage.py change_realm_subdomain`: Change subdomain of a realm.
- `./manage.py change_user_email`: Change a user's email address.
- `./manage.py change_user_role`: Can change are user's role
(easier done [via the
UI](https://zulip.com/help/change-a-users-role)) or give bots the
`can_forge_sender` permission, which is needed for certain special API features.
* `./manage.py export_single_user`: does a limited version of the [main
- `./manage.py export_single_user`: does a limited version of the [main
export tools](../production/export-and-import.md) containing just
the messages accessible by a single user.
* `./manage.py reactivate_realm`: Reactivates a realm.
* `./manage.py deactivate_user`: Deactivates a user. This can be done
- `./manage.py reactivate_realm`: Reactivates a realm.
- `./manage.py deactivate_user`: Deactivates a user. This can be done
more easily in Zulip's organization administrator UI.
* `./manage.py delete_user`: Completely delete a user from the database.
- `./manage.py delete_user`: Completely delete a user from the database.
For most purposes, deactivating users is preferred, since that does not
alter message history for other users.
See the `./manage.py delete_user --help` documentation for details.
- `./manage.py clear_auth_rate_limit_history`: If a user failed authenticaton
attempts too many times and further attempts are disallowed by the rate limiter,
this can be used to reset the limit.
All of our management commands have internal documentation available
via `manage.py command_name --help`.
@@ -141,13 +144,13 @@ via `manage.py command_name --help`.
Zulip supports several mechanisms for running custom code on a
self-hosted Zulip server:
* Using an existing [integration][integrations] or writing your own
- Using an existing [integration][integrations] or writing your own
[webhook integration][webhook-integrations] or [bot][writing-bots].
* Writing a program using the [Zulip API][zulip-api].
* [Modifying the Zulip server][modifying-zulip].
* Using the interactive [management shell](#manage-py-shell),
- Writing a program using the [Zulip API][zulip-api].
- [Modifying the Zulip server][modifying-zulip].
- Using the interactive [management shell](#manage-py-shell),
documented above, for one-time work or prototyping.
* Writing a custom management command, detailed here.
- Writing a custom management command, detailed here.
Custom management commands are Python 3 programs that run inside
Zulip's context, so that they can access its libraries, database, and

View File

@@ -11,8 +11,6 @@ will need to register your Zulip server with the Zulip mobile push
notification service. This service will forward push notifications
generated by your server to the Zulip mobile app automatically.
## How to sign up
Starting with Zulip 1.6 for both Android and iOS, Zulip servers
@@ -25,16 +23,18 @@ first.
You can enable this for your Zulip server as follows:
1. Uncomment the `PUSH_NOTIFICATION_BOUNCER_URL =
'https://push.zulipchat.com'` line in your `/etc/zulip/settings.py`
file (i.e. remove the `#` at the start of the line), and
[restart your Zulip server](../production/settings.html#making-changes).
If you installed your Zulip server with a version older than 1.6,
you'll need to add the line (it won't be there to uncomment).
1. Uncomment the
`PUSH_NOTIFICATION_BOUNCER_URL = 'https://push.zulipchat.com'` line
in your `/etc/zulip/settings.py` file (i.e. remove the `#` at the
start of the line), and [restart your Zulip
server](../production/settings.html#making-changes). If you
installed your Zulip server with a version older than 1.6, you'll
need to add the line (it won't be there to uncomment).
1. If you're running Zulip 1.8.1 or newer, you can run the
registration command:
```
```bash
# As root:
su zulip -c '/home/zulip/deployments/current/manage.py register_server'
# Or as the zulip user, you can skip the `su zulip -c`:
@@ -43,6 +43,7 @@ You can enable this for your Zulip server as follows:
# docker-zulip users can run this inside the container with `docker exec`:
docker exec -it -u zulip <container_name> /home/zulip/deployments/current/manage.py register_server
```
This command will print the registration data it would send to the
mobile push notifications service, ask you to accept the terms of
service, and if you accept, register your server. If you have trouble,
@@ -57,15 +58,15 @@ Congratulations! You've successfully set up the service.
If you'd like to verify that everything is working, you can do the
following. Please follow the instructions carefully:
* [Configure mobile push notifications to always be sent][mobile-notifications-always]
- [Configure mobile push notifications to always be sent][mobile-notifications-always]
(normally they're only sent if you're idle, which isn't ideal for
this sort of testing).
* On an Android device, download and log in to the
- On an Android device, download and log in to the
[Zulip Android app](https://play.google.com/store/apps/details?id=com.zulipmobile).
If you were already logged in before configuring the server, you'll
need to log out first, since the app only registers for push
notifications on login.
* Hit the home button, so Zulip is running in the background, and then
- Hit the home button, so Zulip is running in the background, and then
have **another user** send you a **private message** (By default,
Zulip only sends push notifications for private messages sent by other
users and messages mentioning you). A push notification should appear
@@ -82,9 +83,9 @@ in the installer). You can update your server's registration data by
running `manage.py register_server` again.
If you'd like to rotate your server's API key for this service
(`zulip_org_key`), you need to use `manage.py register_server
--rotate-key` option; it will automatically generate a new
`zulip_org_key` and store that new key in
(`zulip_org_key`), you need to use
`manage.py register_server --rotate-key` option; it will automatically
generate a new `zulip_org_key` and store that new key in
`/etc/zulip/zulip-secrets.conf`.
## Why this is necessary
@@ -112,27 +113,28 @@ push notifications, you agree to those terms.
We've designed this push notification bouncer service with security
and privacy in mind:
* A central design goal of the the Push Notification Service is to
- A central design goal of the the Push Notification Service is to
avoid any message content being stored or logged by the service,
even in error cases.
* The Push Notification Service only stores the necessary metadata for
- The Push Notification Service only stores the necessary metadata for
delivering the notifications to the appropriate devices, and nothing
else:
* The APNS/FCM tokens needed to securely send mobile push
- The APNS/FCM tokens needed to securely send mobile push
notifications to iOS and Android devices, one per device
registered to be notified by your Zulip server.
* User ID numbers generated by your Zulip server, needed to route
- User ID numbers generated by your Zulip server, needed to route
a given notification to the appropriate set of mobile devices.
These user ID numbers are are opaque to the Push Notification
Service and Kandra Labs.
* The Push Notification Service receives (but does not store) the
- The Push Notification Service receives (but does not store) the
contents of individual mobile push notifications:
* The numeric message ID generated by your Zulip server.
* Metadata on the message's sender (name and avatar URL).
* Metadata on the message's recipient (stream name + ID, topic,
- The numeric message ID generated by your Zulip server.
- Metadata on the message's sender (name and avatar URL).
- Metadata on the message's recipient (stream name + ID, topic,
private message recipients, etc.).
* A timestamp.
* The message's content.
- A timestamp.
- The message's content.
There's a `PUSH_NOTIFICATION_REDACT_CONTENT` setting available to
disable any message content being sent via the push notification
@@ -145,14 +147,15 @@ and privacy in mind:
which would eliminate that usability tradeoff and additionally allow
us to not have any access to the other details mentioned in this
section.
* All of the network requests (both from Zulip servers to the Push
- All of the network requests (both from Zulip servers to the Push
Notification Service and from the Push Notification Service to the
relevant Google and Apple services) are encrypted over the wire with
SSL/TLS.
* The code for the push notification forwarding service is 100% open
- The code for the push notification forwarding service is 100% open
source and available as part of the
[Zulip server project on GitHub](https://github.com/zulip/zulip).
* The push notification forwarding servers are professionally managed
- The push notification forwarding servers are professionally managed
by a small team of security expert engineers.
If you have any questions about the security model, contact
@@ -216,17 +219,18 @@ the app stores yourself.
If you've done that work, the Zulip server configuration for sending
push notifications through the new app is quite straightforward:
* Create a
- Create a
[FCM push notifications](https://firebase.google.com/docs/cloud-messaging)
key in the Google Developer console and set `android_gcm_api_key` in
`/etc/zulip/zulip-secrets.conf` to that key.
* Register for a
- Register for a
[mobile push notification certificate][apple-docs]
from Apple's developer console. Set `APNS_SANDBOX=False` and
`APNS_CERT_FILE` to be the path of your APNS certificate file in
`/etc/zulip/settings.py`.
* Set the `APNS_TOPIC` and `ZULIP_IOS_APP_ID` settings to the ID for
- Set the `APNS_TOPIC` and `ZULIP_IOS_APP_ID` settings to the ID for
your app (for the official Zulip apps, they are both `org.zulip.Zulip`).
* Restart the Zulip server.
- Restart the Zulip server.
[apple-docs]: https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -28,27 +28,27 @@ server at the same time.
When you want to create a new organization, you need to do a few
things:
* If you're using Zulip older than 1.7, you'll need to set
- If you're using Zulip older than 1.7, you'll need to set
`REALMS_HAVE_SUBDOMAINS=True` in your `/etc/zulip/settings.py`
file. That setting is the default in 1.7 and later.
* Make sure you have SSL certificates for all of the subdomains you're
- Make sure you have SSL certificates for all of the subdomains you're
going to use. If you're using
[our Let's Encrypt instructions](ssl-certificates.md), it's easy to
just specify multiple subdomains in your certificate request.
* If necessary, modify your `nginx` configuration to use your new
- If necessary, modify your `nginx` configuration to use your new
certificates.
* Use `./manage.py generate_realm_creation_link` again to create your
- Use `./manage.py generate_realm_creation_link` again to create your
new organization. Review
[the install instructions](install.md) if you need a
refresher on how this works.
* If you're planning on using GitHub auth or another social
- If you're planning on using GitHub auth or another social
authentication method, review
[the notes on `SOCIAL_AUTH_SUBDOMAIN` below](#authentication).
For servers hosting a large number of organizations, like
[zulip.com](https://zulip.com), one can set `ROOT_DOMAIN_LANDING_PAGE
= True` in `/etc/zulip/settings.py` so that the homepage for the
server is a copy of the Zulip homepage.
[zulip.com](https://zulip.com), one can set
`ROOT_DOMAIN_LANDING_PAGE = True` in `/etc/zulip/settings.py` so that
the homepage for the server is a copy of the Zulip homepage.
### SSL certificates
@@ -71,7 +71,7 @@ If you'd like to use hostnames that are not subdomains of each other,
you can set the `REALM_HOSTS` setting in `/etc/zulip/settings.py` to a
Python dictionary, like this:
```
```python
REALM_HOSTS = {
'mysubdomain': 'hostname.example.com',
}

View File

@@ -1,4 +1,4 @@
```eval_rst
```{eval-rst}
:orphan:
```
@@ -14,7 +14,7 @@ detail how we set the default threshold (`PASSWORD_MIN_GUESSES`) we use.
First, read the doc section there. (It's short.)
Then, the CACM article ["Passwords and the Evolution of Imperfect
Authentication"][BHOS15] is comprehensive, educational, and readable,
Authentication"][bhos15] is comprehensive, educational, and readable,
and is especially recommended.
The CACM article is convincing that password requirements should be
@@ -36,7 +36,7 @@ overestimation (allowing a weak password) sharply degrades at 100k
guesses, while underestimation (rejecting a strong password) jumps up
just after 10k guesses, and grows steadily thereafter.
Moreover, the [Yahoo study][Bon12] shows that resistance to even 1M
Moreover, the [Yahoo study][bon12] shows that resistance to even 1M
guesses is more than nearly half of users accomplish with a freely
chosen password, and 100k is too much for about 20%. (See Figure 6.)
It doesn't make sense for a Zulip server to try to educate or push so
@@ -54,6 +54,6 @@ zxcvbn rarely underestimates the strength of a password too severely,
and only about 10% of users do worse than this without prompting.
[zxcvbn]: https://github.com/dropbox/zxcvbn
[BHOS15]: https://www.cl.cam.ac.uk/~fms27/papers/2015-BonneauHerOorSta-passwords.pdf
[bhos15]: https://www.cl.cam.ac.uk/~fms27/papers/2015-BonneauHerOorSta-passwords.pdf
[zxcvbn-paper]: https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_wheeler.pdf
[Bon12]: https://ieeexplore.ieee.org/document/6234435
[bon12]: https://ieeexplore.ieee.org/document/6234435

View File

@@ -1,5 +1,4 @@
PostgreSQL database details
=========================
# PostgreSQL database details
Starting with Zulip 3.0, Zulip supports a range of PostgreSQL
versions. PostgreSQL 13 is the current default for new installations;
@@ -32,34 +31,34 @@ called "zulip" in your database server. You can configure these
options in `/etc/zulip/settings.py` (the below descriptions are from the
PostgreSQL documentation):
* `REMOTE_POSTGRES_HOST`: Name or IP address of the remote host
* `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server,
- `REMOTE_POSTGRES_HOST`: Name or IP address of the remote host
- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server,
different options you can use are:
* disable: I don't care about security, and I don't want to pay the
- disable: I don't care about security, and I don't want to pay the
overhead of encryption.
* allow: I don't care about security, but I will pay the overhead of
- allow: I don't care about security, but I will pay the overhead of
encryption if the server insists on it.
* prefer: I don't care about encryption, but I wish to pay the
- prefer: I don't care about encryption, but I wish to pay the
overhead of encryption if the server supports it.
* require: I want my data to be encrypted, and I accept the
- require: I want my data to be encrypted, and I accept the
overhead. I trust that the network will make sure I always connect
to the server I want.
* verify-ca: I want my data encrypted, and I accept the overhead. I
- verify-ca: I want my data encrypted, and I accept the overhead. I
want to be sure that I connect to a server that I trust.
* verify-full: I want my data encrypted, and I accept the
- verify-full: I want my data encrypted, and I accept the
overhead. I want to be sure that I connect to a server I trust,
and that it's the one I specify.
Then you should specify the password of the user zulip for the
database in /etc/zulip/zulip-secrets.conf:
```
```ini
postgres_password = xxxx
```
Finally, you can stop your database on the Zulip server via:
```
```bash
sudo service postgresql stop
sudo update-rc.d postgresql disable
```
@@ -76,7 +75,7 @@ can give you some tips.
When debugging PostgreSQL issues, in addition to the standard `pg_top`
tool, often it can be useful to use this query:
```
```postgresql
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY procpid;
```
@@ -84,20 +83,21 @@ which shows the currently running backends and their activity. This is
similar to the pg_top output, with the added advantage of showing the
complete query, which can be valuable in debugging.
To stop a runaway query, you can run `SELECT pg_cancel_backend(pid
int)` or `SELECT pg_terminate_backend(pid int)` as the 'postgres'
user. The former cancels the backend's current query and the latter
terminates the backend process. They are implemented by sending SIGINT
and SIGTERM to the processes, respectively. We recommend against
sending a PostgreSQL process SIGKILL. Doing so will cause the database
to kill all current connections, roll back any pending transactions,
and enter recovery mode.
To stop a runaway query, you can run
`SELECT pg_cancel_backend(pid int)` or
`SELECT pg_terminate_backend(pid int)` as the 'postgres' user. The
former cancels the backend's current query and the latter terminates
the backend process. They are implemented by sending SIGINT and
SIGTERM to the processes, respectively. We recommend against sending
a PostgreSQL process SIGKILL. Doing so will cause the database to kill
all current connections, roll back any pending transactions, and enter
recovery mode.
#### Stopping the Zulip PostgreSQL database
To start or stop PostgreSQL manually, use the pg_ctlcluster command:
```
```bash
pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
```
@@ -107,15 +107,15 @@ prohibitively long. If you use the --force option with stop,
pg_ctlcluster will try to use the "fast" mode for shutting
down. "Fast" mode is described by the manpage thusly:
With the --force option the "fast" mode is used which rolls back all
active transactions, disconnects clients immediately and thus shuts
down cleanly. If that does not work, shutdown is attempted again in
"immediate" mode, which can leave the cluster in an inconsistent state
and thus will lead to a recovery run at the next start. If this still
does not help, the postmaster process is killed. Exits with 0 on
success, with 2 if the server is not running, and with 1 on other
failure conditions. This mode should only be used when the machine is
about to be shut down.
> With the --force option the "fast" mode is used which rolls back all
> active transactions, disconnects clients immediately and thus shuts
> down cleanly. If that does not work, shutdown is attempted again in
> "immediate" mode, which can leave the cluster in an inconsistent state
> and thus will lead to a recovery run at the next start. If this still
> does not help, the postmaster process is killed. Exits with 0 on
> success, with 2 if the server is not running, and with 1 on other
> failure conditions. This mode should only be used when the machine is
> about to be shut down.
Many database parameters can be adjusted while the database is
running. Just modify /etc/postgresql/9.1/main/postgresql.conf and
@@ -128,7 +128,7 @@ database failed to start. It may tell you to check the logs, but you
won't find any information there. pg_ctlcluster runs the following
command underneath when it actually goes to start PostgreSQL:
```
```bash
/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o \
'-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
```
@@ -139,7 +139,6 @@ stop PostgreSQL and restart it using pg_ctlcluster after you've debugged
with this approach, since it does bypass some of the work that
pg_ctlcluster does.
#### PostgreSQL vacuuming alerts
The `autovac_freeze` PostgreSQL alert from `check_postgres` is

View File

@@ -1,15 +1,17 @@
# Requirements and scalability
To run a Zulip server, you will need:
* A dedicated machine or VM
* A supported OS:
* Ubuntu 20.04 Focal
* Ubuntu 18.04 Bionic
* Debian 10 Buster
* At least 2GB RAM, and 10GB disk space
* If you expect 100+ users: 4GB RAM, and 2 CPUs
* A hostname in DNS
* Credentials for sending email
- A dedicated machine or VM
- A supported OS:
- Ubuntu 20.04 Focal
- Ubuntu 18.04 Bionic
- Debian 11 Bullseye
- Debian 10 Buster
- At least 2GB RAM, and 10GB disk space
- If you expect 100+ users: 4GB RAM, and 2 CPUs
- A hostname in DNS
- Credentials for sending email
For details on each of these requirements, see below.
@@ -28,29 +30,28 @@ can't support you, but
#### Operating system
Ubuntu 20.04 Focal, 18.04 Bionic, and Debian 10 Buster are supported
for running Zulip in production. 64-bit is recommended. We recommend
installing on the newest supported OS release you're comfortable with,
to save a bit of future work [upgrading the operating
Ubuntu 20.04 Focal, 18.04 Bionic, Debian 11 Bullseye, and Debian 10 Buster
are supported for running Zulip in production. 64-bit is recommended.
We recommend installing on the newest supported OS release you're
comfortable with, to save a bit of future work [upgrading the operating
system][upgrade-os].
If you're using Ubuntu, the
[Ubuntu universe repository][ubuntu-repositories] must be
[enabled][enable-universe], which is usually just:
```
```bash
sudo add-apt-repository universe
sudo apt update
```
[upgrade-os]: ../production/upgrade-or-modify.html#upgrading-the-operating-system
[ubuntu-repositories]:
https://help.ubuntu.com/community/Repositories/Ubuntu
[ubuntu-repositories]: https://help.ubuntu.com/community/Repositories/Ubuntu
[enable-universe]: https://help.ubuntu.com/community/Repositories/CommandLine#Adding_the_Universe_and_Multiverse_Repositories
#### Hardware specifications
* CPU and memory: For installations with 100+ users you'll need a
- CPU and memory: For installations with 100+ users you'll need a
minimum of **2 CPUs** and **4GB RAM**. For installations with fewer
users, 1 CPU and 2GB RAM is sufficient. We strongly recommend against
installing with less than 2GB of RAM, as you will likely experience
@@ -58,7 +59,7 @@ https://help.ubuntu.com/community/Repositories/Ubuntu
using highly CPU-limited servers like the AWS `t2` style instances
for organizations with hundreds of users (active or no).
* Disk space: You'll need at least 10GB of free disk space for a
- Disk space: You'll need at least 10GB of free disk space for a
server with dozens of users. We recommend using an SSD and avoiding
cloud storage backends that limit the IOPS per second, since the
disk is primarily used for the Zulip database.
@@ -68,38 +69,38 @@ on hardware requirements for larger organizations.
#### Network and security specifications
* Incoming HTTPS access (usually port 443, though this is
- Incoming HTTPS access (usually port 443, though this is
[configurable](../production/deployment.html#using-an-alternate-port))
from the networks where your users are (usually, the public
Internet).
* Incoming port 80 access (optional). Zulip only serves content over
- Incoming port 80 access (optional). Zulip only serves content over
HTTPS, and will redirect HTTP requests to HTTPS.
* Incoming port 25 if you plan to enable Zulip's [incoming email
- Incoming port 25 if you plan to enable Zulip's [incoming email
integration](../production/email-gateway.md).
* Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so
- Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so
that Zulip can properly manage image and website previews and mobile
push notifications. Outgoing Internet access is not required if you
[disable those
features](https://zulip.com/help/allow-image-link-previews).
* Outgoing SMTP access (usually port 587) to your [SMTP
- Outgoing SMTP access (usually port 587) to your [SMTP
server](../production/email.md) so that Zulip can send emails.
* A domain name (e.g. `zulip.example.com`) that your users will use to
- A domain name (e.g. `zulip.example.com`) that your users will use to
access the Zulip server. In order to generate valid SSL
certificates [with Certbot][doc-certbot], and to enable other
services such as Google authentication, public DNS name is simpler,
but Zulip can be configured to use a non-public domain or even an IP
address as its external hostname (though we don't recommend that
configuration).
* Zulip supports [running behind a reverse proxy][reverse-proxy].
* Zulip servers running inside a private network should configure the
- Zulip supports [running behind a reverse proxy][reverse-proxy].
- Zulip servers running inside a private network should configure the
[Smokescreen integration][smokescreen-proxy] to protect against
[SSRF attacks][SSRF], where users could make the Zulip server make
[SSRF attacks][ssrf], where users could make the Zulip server make
requests to private resources.
[SSRF]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[smokescreen-proxy]: ../production/deployment.html#using-an-outgoing-http-proxy
[reverse-proxy]: ../production/deployment.html#putting-the-zulip-application-behind-a-reverse-proxy
[email-mirror-code]: https://github.com/zulip/zulip/blob/master/zerver/management/commands/email_mirror.py
[email-mirror-code]: https://github.com/zulip/zulip/blob/main/zerver/management/commands/email_mirror.py
## Credentials needed
@@ -123,7 +124,7 @@ certificate documentation](ssl-certificates.md).
#### Outgoing email
* Outgoing email (SMTP) credentials that Zulip can use to send
- Outgoing email (SMTP) credentials that Zulip can use to send
outgoing emails to users (e.g. email address confirmation emails
during the signup process, message notification emails, password
reset, etc.). If you don't have an existing outgoing SMTP solution,
@@ -140,10 +141,11 @@ Zulip in production](../production/install.md).
This section details some basic guidelines for running a Zulip server
for larger organizations (especially >1000 users or 500+ daily active
users). Zulip's resource needs depend mainly on 3 parameters:
* daily active users (e.g. number of employees if everyone's an
- daily active users (e.g. number of employees if everyone's an
employee)
* total user accounts (can be much larger)
* message volume.
- total user accounts (can be much larger)
- message volume.
In the following, we discuss a configuration with at most two types of
servers: application servers (running Django, Tornado, RabbitMQ,
@@ -155,20 +157,21 @@ most use cases, there's little scalability benefit to doing so. See
[deployment options](../production/deployment.md) for details on
installing Zulip with a dedicated database server.
* **Dedicated database**. For installations with hundreds of daily
- **Dedicated database**. For installations with hundreds of daily
active users, we recommend using a [remote PostgreSQL
database](postgresql.md), but it's not required.
* **RAM:** We recommended more RAM for larger installations:
* With 25+ daily active users, 4GB of RAM.
* With 100+ daily active users, 8GB of RAM.
* With 400+ daily active users, 16GB of RAM for the Zulip
application server, plus 16GB for the database.
* With 2000+ daily active users 32GB of RAM, plus 32GB for the
database.
* Roughly linear scaling beyond that.
- **RAM:** We recommended more RAM for larger installations:
* **CPU:** The Zulip application server's CPU usage is heavily
- With 25+ daily active users, 4GB of RAM.
- With 100+ daily active users, 8GB of RAM.
- With 400+ daily active users, 16GB of RAM for the Zulip
application server, plus 16GB for the database.
- With 2000+ daily active users 32GB of RAM, plus 32GB for the
database.
- Roughly linear scaling beyond that.
- **CPU:** The Zulip application server's CPU usage is heavily
optimized due to extensive work on optimizing the performance of
requests for latency reasons. Because most servers with sufficient
RAM have sufficient CPU resources, CPU requirements are rarely an
@@ -177,34 +180,34 @@ installing Zulip with a dedicated database server.
database-optimized (usually low CPU, high memory) instance for the
database.
* **Disk for application server:** We recommend using [the S3 file
- **Disk for application server:** We recommend using [the S3 file
uploads backend][s3-uploads] to store uploaded files at scale. With
the S3 backend configuration, we recommend 50GB of disk for the OS,
Zulip software, logs and scratch/free space. Disk needs when
storing uploads locally
* **Disk for database:** SSD disk is highly recommended. For
- **Disk for database:** SSD disk is highly recommended. For
installations where most messages have <100 recipients, 10GB per 1M
messages of history is sufficient plus 1GB per 1000 users is
sufficient. If most messages are to public streams with 10K+ users
subscribed (like on chat.zulip.org), add 20GB per (1000 user
accounts) per (1M messages to public streams).
* **Example:** When the
- **Example:** When the
[chat.zulip.org](../contributing/chat-zulip-org.md) community server
had 12K user accounts (~300 daily actives) and 800K messages of
history (400K to public streams), it was a default configuration
single-server installation with 16GB of RAM, 4 cores (essentially
always idle), and its database was using about 100GB of disk.
* **Disaster recovery:** One can easily run a hot spare application
- **Disaster recovery:** One can easily run a hot spare application
server and a hot spare database (using [PostgreSQL streaming
replication][streaming-replication]). Make sure the hot spare
application server has copies of `/etc/zulip` and you're either
syncing `LOCAL_UPLOADS_DIR` or using the [S3 file uploads
backend][s3-uploads].
* **Sharding:** Zulip releases do not fully support dividing Tornado
- **Sharding:** Zulip releases do not fully support dividing Tornado
traffic for a single Zulip realm/organization between multiple
application servers, which is why we recommend a hot spare over
load-balancing. We don't have an easily deployed configuration for
@@ -212,7 +215,7 @@ installing Zulip with a dedicated database server.
can't currently offer this model outside of enterprise support
contracts.
* Zulip 2.0 and later supports running multiple Tornado servers
- Zulip 2.0 and later supports running multiple Tornado servers
sharded by realm/organization, which is how we scale Zulip Cloud.
[Contact us][contact-support] for help implementing the sharding policy.

View File

@@ -11,7 +11,7 @@ announcement).
## Secure your Zulip server like your email server
* It's reasonable to think about security for a Zulip server like you
- It's reasonable to think about security for a Zulip server like you
do security for a team email server -- only trusted individuals
within an organization should have shell access to the server.
@@ -27,15 +27,15 @@ announcement).
## Encryption and authentication
* Traffic between clients (web, desktop and mobile) and the Zulip
- Traffic between clients (web, desktop and mobile) and the Zulip
server is encrypted using HTTPS. By default, all Zulip services
talk to each other either via a localhost connection or using an
encrypted SSL connection.
* Zulip requires CSRF tokens in all interactions with the web API to
- Zulip requires CSRF tokens in all interactions with the web API to
prevent CSRF attacks.
* The preferred way to log in to Zulip is using an SSO solution like
- The preferred way to log in to Zulip is using an SSO solution like
Google auth, LDAP, or similar, but Zulip also supports password
authentication. See
[the authentication methods documentation](../production/authentication-methods.md)
@@ -51,11 +51,11 @@ are rejected, and strong passwords encouraged. The minimum password
strength allowed is controlled by two settings in
`/etc/zulip/settings.py`:
* `PASSWORD_MIN_LENGTH`: The minimum acceptable length, in characters.
- `PASSWORD_MIN_LENGTH`: The minimum acceptable length, in characters.
Shorter passwords are rejected even if they pass the `zxcvbn` test
controlled by `PASSWORD_MIN_GUESSES`.
* `PASSWORD_MIN_GUESSES`: The minimum acceptable strength of the
- `PASSWORD_MIN_GUESSES`: The minimum acceptable strength of the
password, in terms of the estimated number of passwords an attacker
is likely to guess before trying this one. If the user attempts to
set a password that `zxcvbn` estimates to be guessable in less than
@@ -70,10 +70,10 @@ strength allowed is controlled by two settings in
Estimating the guessability of a password is a complex problem and
impossible to efficiently do perfectly. For background or when
considering an alternate value for this setting, the article
["Passwords and the Evolution of Imperfect Authentication"][BHOS15]
["Passwords and the Evolution of Imperfect Authentication"][bhos15]
is recommended. The [2016 zxcvbn paper][zxcvbn-paper] adds useful
information about the performance of zxcvbn, and [a large 2012 study
of Yahoo users][Bon12] is informative about the strength of the
of Yahoo users][bon12] is informative about the strength of the
passwords users choose.
<!---
@@ -86,59 +86,60 @@ strength allowed is controlled by two settings in
-->
[zxcvbn]: https://github.com/dropbox/zxcvbn
[BHOS15]: http://www.cl.cam.ac.uk/~fms27/papers/2015-BonneauHerOorSta-passwords.pdf
[bhos15]: http://www.cl.cam.ac.uk/~fms27/papers/2015-BonneauHerOorSta-passwords.pdf
[zxcvbn-paper]: https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_wheeler.pdf
[Bon12]: http://ieeexplore.ieee.org/document/6234435/
[bon12]: http://ieeexplore.ieee.org/document/6234435/
## Messages and history
* Zulip message content is rendered using a specialized Markdown
- Zulip message content is rendered using a specialized Markdown
parser which escapes content to protect against cross-site scripting
attacks.
* Zulip supports both public streams and private streams.
* Any non-guest user can join any public stream in the organization,
- Zulip supports both public streams and private streams.
- Any non-guest user can join any public stream in the organization,
and can view the complete message history of any public stream
without joining the stream. Guests can only access streams that
another user adds them to.
* Organization owners and administrators can see and modify most
- Organization owners and administrators can see and modify most
aspects of a private stream, including the membership and
estimated traffic. Owners and administrators generally cannot see
messages sent to private streams or do things that would
indirectly give them access to those messages, like adding members
or changing the stream privacy settings.
* Non-admins cannot easily see which private streams exist, or interact
- Non-admins cannot easily see which private streams exist, or interact
with them in any way until they are added. Given a stream name, they can
figure out whether a stream with that name exists, but cannot see any
other details about the stream.
* See [Stream permissions](https://zulip.com/help/stream-permissions) for more details.
- See [Stream permissions](https://zulip.com/help/stream-permissions) for more details.
* Zulip supports editing the content and topics of messages that have
- Zulip supports editing the content and topics of messages that have
already been sent. As a general philosophy, our policies provide
hard limits on the ways in which message content can be changed or
undone. In contrast, our policies around message topics favor
usefulness (e.g. for conversational organization) over faithfulness
to the original. In all configurations:
* Message content can only ever be modified by the original author.
- Message content can only ever be modified by the original author.
* Any message visible to an organization owner or administrator can
- Any message visible to an organization owner or administrator can
be deleted at any time by that administrator.
* See
- See
[Configuring message editing and deletion](https://zulip.com/help/configure-message-editing-and-deletion)
for more details.
## Users and bots
* There are several types of users in a Zulip organization: organization
- There are several types of users in a Zulip organization: organization
owners, organization administrators, members (normal users), guests,
and bots.
* Owners and administrators have the ability to deactivate and
- Owners and administrators have the ability to deactivate and
reactivate other human and bot users, archive streams, add/remove
administrator privileges, as well as change configuration for the
organization.
@@ -148,21 +149,21 @@ strength allowed is controlled by two settings in
streams to which the administrator is not subscribed. There are two
exceptions:
* Organization owners may get access to private messages via some types of
- Organization owners may get access to private messages via some types of
[data export](https://zulip.com/help/export-your-organization).
* Administrators can change the ownership of a bot. If a bot is subscribed
- Administrators can change the ownership of a bot. If a bot is subscribed
to a private stream, then an administrator can indirectly get access to
stream messages by taking control of the bot, though the access will be
limited to what the bot can do. (E.g. incoming webhook bots cannot read
messages.)
* Every Zulip user has an API key, available on the settings page.
- Every Zulip user has an API key, available on the settings page.
This API key can be used to do essentially everything the user can
do; for that reason, users should keep their API key safe. Users
can rotate their own API key if it is accidentally compromised.
* To properly remove a user's access to a Zulip team, it does not
- To properly remove a user's access to a Zulip team, it does not
suffice to change their password or deactivate their account in a
SSO system, since neither of those prevents authenticating with the
user's API key or those of bots the user has created. Instead, you
@@ -170,23 +171,23 @@ strength allowed is controlled by two settings in
[deactivate the user's account](https://zulip.com/help/deactivate-or-reactivate-a-user)
via Zulip's "Organization settings" interface.
* The Zulip mobile apps authenticate to the server by sending the
- The Zulip mobile apps authenticate to the server by sending the
user's password and retrieving the user's API key; the apps then use
the API key to authenticate all future interactions with the site.
Thus, if a user's phone is lost, in addition to changing passwords,
you should rotate the user's Zulip API key.
* Guest users are like Members, but they do not have automatic access
- Guest users are like Members, but they do not have automatic access
to public streams.
* Zulip supports several kinds of bots with different capabilities.
- Zulip supports several kinds of bots with different capabilities.
* Incoming webhook bots can only send messages into Zulip.
* Outgoing webhook bots and Generic bots can essentially do anything a
- Incoming webhook bots can only send messages into Zulip.
- Outgoing webhook bots and Generic bots can essentially do anything a
non-administrator user can, with a few exceptions (e.g. a bot cannot
log in to the web application, register for mobile push
notifications, or create other bots).
* Bots with the `can_forge_sender` permission can send messages that appear to have been sent by
- Bots with the `can_forge_sender` permission can send messages that appear to have been sent by
another user. They also have the ability to see the names of all
streams, including private streams. This is important for implementing
integrations like the Jabber, IRC, and Zephyr mirrors.
@@ -197,7 +198,7 @@ strength allowed is controlled by two settings in
## User-uploaded content and user-generated requests
* Zulip supports user-uploaded files. Ideally they should be hosted
- Zulip supports user-uploaded files. Ideally they should be hosted
from a separate domain from the main Zulip server to protect against
various same-domain attacks (e.g. zulip-user-content.example.com).
@@ -235,34 +236,34 @@ strength allowed is controlled by two settings in
browser is logged into a Zulip account that has received the
uploaded file in question).
* Zulip supports using the Camo image proxy to proxy content like
- Zulip supports using the Camo image proxy to proxy content like
inline image previews, that can be inserted into the Zulip message feed by
other users. This ensures that clients do not make requests to external
servers to fetch images, improving privacy.
* By default, Zulip will provide image previews inline in the body of
- By default, Zulip will provide image previews inline in the body of
messages when a message contains a link to an image. You can
control this using the `INLINE_IMAGE_PREVIEW` setting.
* Zulip may make outgoing HTTP connections to other servers in a
- Zulip may make outgoing HTTP connections to other servers in a
number of cases:
* Outgoing webhook bots (creation of which can be restricted)
* Inline image previews in messages (enabled by default, but can be disabled)
* Inline webpage previews and embeds (must be configured to be enabled)
* Twitter message previews (must be configured to be enabled)
* BigBlueButton and Zoom API requests (must be configured to be enabled)
* Mobile push notifications (must be configured to be enabled)
- Outgoing webhook bots (creation of which can be restricted)
- Inline image previews in messages (enabled by default, but can be disabled)
- Inline webpage previews and embeds (must be configured to be enabled)
- Twitter message previews (must be configured to be enabled)
- BigBlueButton and Zoom API requests (must be configured to be enabled)
- Mobile push notifications (must be configured to be enabled)
* Notably, these first 3 features give end users (limited) control to cause
- Notably, these first 3 features give end users (limited) control to cause
the Zulip server to make HTTP requests on their behalf. As a result,
Zulip supports routing all outgoing outgoing HTTP requests [through
Smokescreen][smokescreen-setup] to ensure that Zulip cannot be
used to execute [SSRF attacks][SSRF] against other systems on an
used to execute [SSRF attacks][ssrf] against other systems on an
internal corporate network. The default Smokescreen configuration
denies access to all non-public IP addresses, including 127.0.0.1.
[SSRF]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[smokescreen-setup]: ../production/deployment.html#using-an-outgoing-http-proxy
## Final notes and security response

View File

@@ -15,7 +15,8 @@ This page discusses additional configuration that a system
administrator can do. To change any of the following settings, edit
the `/etc/zulip/settings.py` file on your Zulip server, and then
restart the server with the following command:
```
```bash
su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
```
@@ -28,7 +29,7 @@ comment documentation for new configuration settings after upgrading
to each new major release.
[update-settings-docs]: ../production/upgrade-or-modify.html#updating-settings-py-inline-documentation
[settings-py-template]: https://github.com/zulip/zulip/blob/master/zproject/prod_settings_template.py
[settings-py-template]: https://github.com/zulip/zulip/blob/main/zproject/prod_settings_template.py
Since Zulip's settings file is a Python script, there are a number of
other things that one can configure that are not documented; ask on
@@ -95,11 +96,12 @@ configuration along with your other Zulip server configuration.
### Miscellaneous server settings
Some popular settings in `/etc/zulip/settings.py` include:
* The Twitter integration, which provides pretty inline previews of
- The Twitter integration, which provides pretty inline previews of
tweets.
* The [email gateway](../production/email-gateway.md), which lets
- The [email gateway](../production/email-gateway.md), which lets
users send emails into Zulip.
* The [Video call integrations](../production/video-calls.md).
- The [Video call integrations](../production/video-calls.md).
## Zulip announcement list

View File

@@ -11,8 +11,9 @@ chore (nor expense) that it used to be.
If you already have an SSL certificate, just install (or symlink) its
files into place at the following paths:
* `/etc/ssl/private/zulip.key` for the private key
* `/etc/ssl/certs/zulip.combined-chain.crt` for the certificate.
- `/etc/ssl/private/zulip.key` for the private key
- `/etc/ssl/certs/zulip.combined-chain.crt` for the certificate.
Your certificate file should contain not only your own certificate but
its **full chain, including any intermediate certificates** used by
@@ -32,15 +33,16 @@ browsers ignore errors that others don't.
Two good tests include:
* If your server is accessible from the public Internet, use the [SSL
- If your server is accessible from the public Internet, use the [SSL
Labs tester][ssllabs-tester]. Be sure to check for "Chain issues";
if any, your certificate file is missing intermediate certificates.
* Alternatively, run a command like `curl -SsI https://zulip.example.com`
- Alternatively, run a command like `curl -SsI https://zulip.example.com`
(using your server's URL) from a machine that can reach your server.
Make sure that on the same machine, `curl -SsI
https://incomplete-chain.badssl.com` gives an error; `curl` on some
machines, including Macs, will accept incomplete chains.
Make sure that on the same machine,
`curl -SsI https://incomplete-chain.badssl.com` gives an error;
`curl` on some machines, including Macs, will accept incomplete
chains.
[ssllabs-tester]: https://www.ssllabs.com/ssltest/analyze.html
@@ -54,11 +56,12 @@ SSL certificates from Let's Encrypt and renew them automatically.
We recommend most Zulip servers use Certbot. You'll want something
else if:
* you have an existing workflow for managing SSL certificates
- you have an existing workflow for managing SSL certificates
that you prefer;
* you need wildcard certificates (support from Let's Encrypt released
- you need wildcard certificates (support from Let's Encrypt released
in [March 2018][letsencrypt-wildcard]); or
* your Zulip server is not on the public Internet. (In this case you
- your Zulip server is not on the public Internet. (In this case you
can [still use Certbot][certbot-manual-mode], but it's less
convenient; and you'll want to ignore Zulip's automation.)
@@ -84,10 +87,12 @@ one as described in the section below after installing Zulip.
To enable the Certbot automation on an already-installed Zulip
server, run the following commands:
```
```bash
sudo -s # If not already root
/home/zulip/deployments/current/scripts/setup/setup-certbot --email=EMAIL HOSTNAME [HOSTNAME2...]
```
where HOSTNAME is the domain name users see in their browser when
using the server (e.g., `zulip.example.com`), and EMAIL is a contact
address for the server admins. Additional hostnames can also be
@@ -108,7 +113,6 @@ checks if any certificates are due for renewal, and if they are (so
approximately once every 60 days), repeats the process of request,
prove, get a fresh certificate.
## Self-signed certificate
If you aren't able to use Certbot, you can generate a self-signed SSL
@@ -125,22 +129,24 @@ just pass the `--self-signed-cert` flag when
To generate a self-signed certificate for an already-installed Zulip
server, run the following commands:
```
```bash
sudo -s # If not already root
/home/zulip/deployments/current/scripts/setup/generate-self-signed-cert HOSTNAME
```
where HOSTNAME is the domain name (or IP address) to use on the
generated certificate.
After replacing the certificates, you need to reload `nginx` by
running the following as `root`:
```
```bash
service nginx reload
```
[desktop-certs]: https://zulip.com/help/custom-certificates
## Troubleshooting
### The Android app can't connect to the server
@@ -148,7 +154,6 @@ service nginx reload
This is most often caused by an incomplete certificate chain. See
discussion in the [Manual install](#manual-install) section above.
### The iOS app can't connect to the server
This can be caused by a server set up to support only TLS 1.1 or
@@ -156,7 +161,7 @@ older (including TLS 1.0, SSL 3, or SSL 2.)
TLS 1.2 has been a standard for over 10 years, and all modern web
server software supports it. Starting in early 2020, all major
browsers [will *require* TLS 1.2 or later][tls12-required-news], and
browsers [will _require_ TLS 1.2 or later][tls12-required-news], and
will refuse to connect over TLS 1.1 or older. And on iOS, Apple [has
since iOS 9][apple-ats] required TLS 1.2 for all connections made by
apps, unless the app specifically opts into lower security.
@@ -174,7 +179,6 @@ directive][nginx-doc-protocols] in your configuration.
[nginx-doc-protocols]: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols
### The Android app connects to the server on some devices but not others
An issue on Android 7.0 ([report][android7.0-tls-issue],
@@ -202,10 +206,10 @@ details.
Two signs for diagnosing this issue in contrast to some other root
cause:
* This issue affects only Android 7.0; it's fixed in Android 7.1.1 and
- This issue affects only Android 7.0; it's fixed in Android 7.1.1 and
later.
* If your server is reachable from the public Internet, use the [SSL
- If your server is reachable from the public Internet, use the [SSL
Labs tester][ssllabs-tester]. Under "Cipher Suites" you may see
lines beginning with `TLS_ECDHE`, for cipher suites which use
elliptic-curve cryptography. These lines will have further text

View File

@@ -37,13 +37,14 @@ and restart various services.
### Checking status with `supervisorctl status`
You can check if the Zulip application is running using:
```
```bash
supervisorctl status
```
When everything is running as expected, you will see something like this:
```
```console
process-fts-updates RUNNING pid 2194, uptime 1:13:11
zulip-django RUNNING pid 2192, uptime 1:13:11
zulip-tornado RUNNING pid 2193, uptime 1:13:11
@@ -75,7 +76,7 @@ After you change configuration in `/etc/zulip/settings.py` or fix a
misconfiguration, you will often want to restart the Zulip application.
You can restart Zulip using:
```
```bash
supervisorctl restart all
```
@@ -83,7 +84,7 @@ supervisorctl restart all
Similarly, you can stop Zulip using:
```
```bash
supervisorctl stop all
```
@@ -96,39 +97,42 @@ The Zulip application uses several major open source services to store
and cache data, queue messages, and otherwise support the Zulip
application:
* PostgreSQL
* RabbitMQ
* Nginx
* Redis
* memcached
- PostgreSQL
- RabbitMQ
- Nginx
- Redis
- memcached
If one of these services is not installed or functioning correctly,
Zulip will not work. Below we detail some common configuration
problems and how to resolve them:
* If your browser reports no webserver is running, that is likely
- If your browser reports no webserver is running, that is likely
because nginx is not configured properly and thus failed to start.
nginx will fail to start if you configured SSL incorrectly or did
not provide SSL certificates. To fix this, configure them properly
and then run:
```
```bash
service nginx restart
```
* If your host is being port scanned by unauthorized users, you may see
- If your host is being port scanned by unauthorized users, you may see
messages in `/var/log/zulip/server.log` like
```
```text
2017-02-22 14:11:33,537 ERROR Invalid HTTP_HOST header: '10.2.3.4'. You may need to add u'10.2.3.4' to ALLOWED_HOSTS.
```
Django uses the hostnames configured in `ALLOWED_HOSTS` to identify
legitimate requests and block others. When an incoming request does
not have the correct HTTP Host header, Django rejects it and logs the
attempt. For more on this issue, see the [Django release notes on Host header
poisoning](https://www.djangoproject.com/weblog/2013/feb/19/security/#s-issue-host-header-poisoning)
* An AMQPConnectionError traceback or error running rabbitmqctl
- An AMQPConnectionError traceback or error running rabbitmqctl
usually means that RabbitMQ is not running; to fix this, try:
```
```bash
service rabbitmq-server restart
```
If RabbitMQ fails to start, the problem is often that you are using
@@ -137,16 +141,14 @@ problems and how to resolve them:
### Restrict unattended upgrades
```eval_rst
.. important::
We recommend that you `disable or limit Ubuntu's unattended-upgrades
to skip some server packages
<https://linoxide.com/ubuntu-how-to/enable-disable-unattended-upgrades-ubuntu-16-04/>`;
if you disable them, do not forget to regularly install apt upgrades
manually. With unattended upgrades enabled but not limited, the
moment a new PostgreSQL release is published, your Zulip server will
have its PostgreSQL server upgraded (and thus restarted).
```
:::{important}
We recommend that you disable or limit Ubuntu's unattended-upgrades
to skip some server packages. With unattended upgrades enabled but
not limited, the moment a new PostgreSQL release is published, your
Zulip server will have its PostgreSQL server upgraded (and thus
restarted). If you do disable unattended-upgrades, do not forget to
regularly install apt upgrades manually!
:::
Restarting one of the system services that Zulip uses (PostgreSQL,
memcached, Redis, or Rabbitmq) will drop the connections that
@@ -177,7 +179,7 @@ You can ensure that the `unattended-upgrades` package never upgrades
PostgreSQL, memcached, Redis, or RabbitMQ, by configuring in
`/etc/apt/apt.conf.d/50unattended-upgrades`:
```
```text
// Python regular expressions, matching packages to exclude from upgrading
Unattended-Upgrade::Package-Blacklist {
"libc\d+";
@@ -203,14 +205,14 @@ reporting/investigating any that you do see.
Beyond that, the most important monitoring for a Zulip server is
standard stuff:
* Basic host health monitoring for issues running out of disk space,
- Basic host health monitoring for issues running out of disk space,
especially for the database and where uploads are stored.
* Service uptime and standard monitoring for the [services Zulip
- Service uptime and standard monitoring for the [services Zulip
depends on](#troubleshooting-services). Most monitoring software
has standard plugins for Nginx, PostgreSQL, Redis, RabbitMQ,
and memcached, and those will work well with Zulip.
* `supervisorctl status` showing all services `RUNNING`.
* Checking for processes being OOM killed.
- `supervisorctl status` showing all services `RUNNING`.
- Checking for processes being OOM killed.
Beyond that, Zulip ships a few application-specific end-to-end health
checks. The Nagios plugins `check_send_receive_time`,
@@ -232,31 +234,31 @@ useful Nagios plugins included with Zulip and what they check:
Application server and queue worker monitoring:
* `check_send_receive_time`: Sends a test message through the system
- `check_send_receive_time`: Sends a test message through the system
between two bot users to check that end-to-end message sending
works. An effective end-to-end check for Zulip's Django and Tornado
systems being healthy.
* `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective
- `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective
checks for Zulip's RabbitMQ-based queuing systems being healthy.
* `check_worker_memory`: Monitors for memory leaks in queue workers.
* `check_email_deliverer_backlog` and `check_email_deliverer_process`:
- `check_worker_memory`: Monitors for memory leaks in queue workers.
- `check_email_deliverer_backlog` and `check_email_deliverer_process`:
Monitors for whether scheduled outgoing emails (e.g. invitation
reminders) are being sent properly.
Database monitoring:
* `check_fts_update_log`: Checks whether full-text search updates are
- `check_fts_update_log`: Checks whether full-text search updates are
being processed properly or getting backlogged.
* `check_postgres`: General checks for database health.
* `check_postgresql_backup`: Checks status of PostgreSQL backups.
* `check_postgresql_replication_lag`: Checks whether PostgreSQL streaming
- `check_postgres`: General checks for database health.
- `check_postgresql_backup`: Checks status of PostgreSQL backups.
- `check_postgresql_replication_lag`: Checks whether PostgreSQL streaming
replication is up to date.
Standard server monitoring:
* `check_website_response.sh`: Basic HTTP check.
* `check_debian_packages`: Checks whether the system is behind on `apt
upgrade`.
- `check_website_response.sh`: Basic HTTP check.
- `check_debian_packages`: Checks whether the system is behind on
`apt upgrade`.
If you're using these plugins, bug reports and pull requests to make
it easier to monitor Zulip and maintain it in production are

View File

@@ -10,7 +10,7 @@ This page explains how to upgrade, patch, or modify Zulip, including:
- [Upgrading the operating system](#upgrading-the-operating-system)
- [Upgrading PostgreSQL](#upgrading-postgresql)
- [Modifying Zulip](#modifying-zulip)
- [Applying changes from master](#applying-changes-from-master)
- [Applying changes from `main`](#applying-changes-from-main)
## Upgrading to a release
@@ -24,11 +24,12 @@ to a new Zulip release:
for all releases newer than what is currently installed.
1. Download the appropriate release tarball from
<https://www.zulip.org/dist/releases/> You can download the latest
release with:
<https://download.zulip.com/server/>. You can get the latest
release (**Zulip Server {{ LATEST_RELEASE_VERSION }}**) with the
following command:
```
wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
```bash
wget https://download.zulip.com/server/zulip-server-latest.tar.gz
```
You also have the option of upgrading Zulip [to a version in a Git
@@ -39,18 +40,19 @@ to a new Zulip release:
1. Log in to your Zulip and run as root:
```
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
```bash
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-latest.tar.gz
```
The upgrade process will:
* Run `apt-get upgrade`
* Install new versions of Zulip's dependencies (mainly Python packages).
* (`upgrade-zulip-from-git` only) Build Zulip's frontend assets using `webpack`.
* Shut down the Zulip service
* Run a `puppet apply`
* Run any database migrations
* Bring the Zulip service back up on the new version.
- Run `apt-get upgrade`
- Install new versions of Zulip's dependencies (mainly Python packages).
- (`upgrade-zulip-from-git` only) Build Zulip's frontend assets using `webpack`.
- Shut down the Zulip service
- Run a `puppet apply`
- Run any database migrations
- Bring the Zulip service back up on the new version.
Upgrading will result in brief downtime for the service, which should
be under 30 seconds unless there is an expensive database migration
@@ -68,15 +70,15 @@ run into any issues or need to roll back the upgrade.
Zulip supports upgrading a production installation to any commit in a
Git repository, which is great for [running pre-release changes from
master](#applying-changes-from-master) or [maintaining a
`main`](#applying-changes-from-main) or [maintaining a
fork](#making-changes). The process is simple:
```
```bash
# Upgrade to an official release
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 1.8.1
# Upgrade to a branch (or other Git ref)
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 2.1.x
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git master
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git main
```
Zulip will automatically fetch the relevant Git commit and upgrade to
@@ -87,15 +89,15 @@ containing the changes planned for the next minor release
(E.g. 2.1.5); we support these stable release branches as though they
were a published release.
The `master` branch contains changes planned for the next major
The `main` branch contains changes planned for the next major
release (E.g. 3.0); see our documentation on [running
master](#upgrading-to-master) before upgrading to it.
`main`](#upgrading-to-main) before upgrading to it.
By default, this uses the main upstream Zulip server repository, but
you can configure any other Git repository by adding a section like
this to `/etc/zulip/zulip.conf`:
```
```ini
[deployment]
git_repo_url = https://github.com/zulip/zulip.git
```
@@ -123,7 +125,7 @@ suggest using that updated template to update
do not have a recent [complete backup][backups]), and make a copy
of the current template:
```
```bash
cp -a /etc/zulip/settings.py ~/zulip-settings-backup.py
cp -a /home/zulip/deployments/current/zproject/prod_settings_template.py /etc/zulip/settings-new.py
```
@@ -137,7 +139,7 @@ suggest using that updated template to update
the template that your `/etc/zulip/settings.py` was installed
using, and the differences that your file has from that:
```
```bash
/home/zulip/deployments/current/scripts/setup/compare-settings-to-template
```
@@ -149,7 +151,7 @@ suggest using that updated template to update
the server to pick up the new file; this should be a no-op, but it
is much better to discover immediately if it is not:
```
```bash
cp -a /etc/zulip/settings-new.py /etc/zulip/settings.py
su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
```
@@ -165,10 +167,10 @@ guide](../production/troubleshooting.md).
The upgrade scripts are idempotent, so there's no harm in trying again
after resolving an issue. The most common causes of errors are:
* Networking issues (e.g. your Zulip server doesn't have reliable
- Networking issues (e.g. your Zulip server doesn't have reliable
Internet access or needs a proxy set up). Fix the networking issue
and try again.
* Especially when using `upgrade-zulip-from-git`, systems with the
- Especially when using `upgrade-zulip-from-git`, systems with the
minimal RAM for running Zulip can run into out-of-memory issues
during the upgrade process (generally `tools/webpack` is the step
that fails). You can get past this by shutting down the Zulip
@@ -176,9 +178,10 @@ after resolving an issue. The most common causes of errors are:
the upgrade process.
Useful logs are available in a few places:
* The Zulip upgrade scripts log all output to
- The Zulip upgrade scripts log all output to
`/var/log/zulip/upgrade.log`.
* The Zulip server logs all Internal Server Errors to
- The Zulip server logs all Internal Server Errors to
`/var/log/zulip/errors.log`.
If you need help and don't have a support contract, you can visit
@@ -212,12 +215,11 @@ the version corresponding to the `restart-server` path you call.
## Preserving local changes to service configuration files
```eval_rst
.. warning::
:::{warning}
If you have modified service configuration files installed by
Zulip (e.g. the nginx configuration), the Zulip upgrade process will
overwrite your configuration when it does the ``puppet apply``.
```
overwrite your configuration when it does the `puppet apply`.
:::
You can test whether this will happen assuming no upstream changes to
the configuration using `scripts/zulip-puppet-apply` (without the
@@ -261,7 +263,7 @@ instructions for other supported platforms.
2. As the Zulip user, stop the Zulip server and run the following
to back up the system:
```
```bash
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
```
@@ -271,7 +273,7 @@ instructions for other supported platforms.
`do-release-upgrade` and following the prompts until it completes
successfully:
```
```bash
sudo -i # Or otherwise get a root shell
do-release-upgrade -d
```
@@ -288,17 +290,26 @@ instructions for other supported platforms.
4. As root, upgrade the database to the latest version of PostgreSQL:
```
```bash
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
```
5. Finally, we need to reinstall the current version of Zulip, which
5. Ubuntu 20.04 has a different version of the low-level glibc
library, which affects how PostgreSQL orders text data (known as
"collations"); this corrupts database indexes that rely on
collations. Regenerate the affected indexes by running:
```bash
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
```
6. Finally, we need to reinstall the current version of Zulip, which
among other things will recompile Zulip's Python module
dependencies for your new version of Python and rewrite Zulip's
full-text search indexes to work with the upgraded dictionary
packages:
```
```bash
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
@@ -321,7 +332,7 @@ instructions for other supported platforms.
4. As root, upgrade the database installation and OS configuration to
match the new OS version:
```
```bash
touch /usr/share/postgresql/10/pgroonga_setup.sql.applied
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
pg_dropcluster 10 main --stop
@@ -337,7 +348,7 @@ instructions for other supported platforms.
among other things will recompile Zulip's Python module
dependencies for your new version of Python:
```
```bash
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
@@ -352,7 +363,7 @@ instructions for other supported platforms.
7. As root, finish by verifying the contents of the full-text indexes:
```
```bash
/home/zulip/deployments/current/manage.py audit_fts_indexes
```
@@ -369,7 +380,7 @@ instructions for other supported platforms.
4. As root, upgrade the database installation and OS configuration to
match the new OS version:
```
```bash
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
pg_dropcluster 9.5 main --stop
@@ -385,7 +396,7 @@ instructions for other supported platforms.
among other things will recompile Zulip's Python module
dependencies for your new version of Python:
```
```bash
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
@@ -399,13 +410,80 @@ instructions for other supported platforms.
Bionic](#upgrading-from-ubuntu-16-04-xenial-to-18-04-bionic), so
that you are running a supported operating system.
### Upgrading from Debian Buster to Debian Bullseye
1. Upgrade your server to the latest Zulip `4.x` release.
2. As the Zulip user, stop the Zulip server and run the following
to back up the system:
```bash
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
```
3. Follow [Debian's instructions to upgrade the OS][bullseye-upgrade].
[bullseye-upgrade]: https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.html
When prompted for you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
Nginx, and memcached, the best choice is `N` to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
4. As root, run the following steps to regenerate configurations
for services used by Zulip:
```bash
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
```
5. Reinstall the current version of Zulip, which among other things
will recompile Zulip's Python module dependencies for your new
version of Python:
```bash
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
```
This will finish by restarting your Zulip server; you should now
be able to navigate to its URL and confirm everything is working
correctly.
6. Debian Bullseye has a different version of the low-level glibc
library, which affects how PostgreSQL orders text data (known as
"collations"); this corrupts database indexes that rely on
collations. Regenerate the affected indexes by running:
```bash
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
```
7. As root, finish by verifying the contents of the full-text indexes:
```bash
/home/zulip/deployments/current/manage.py audit_fts_indexes
```
8. As an additional step, you can also [upgrade the postgresql version](#upgrading-postgresql).
### Upgrading from Debian Stretch to Debian Buster
1. Upgrade your server to the latest Zulip `2.1.x` release. You can
only upgrade to Zulip 3.0 and newer after completing this process,
since newer releases don't support Ubuntu Debian Stretch.
2. Same as for Bionic to Focal.
2. As the Zulip user, stop the Zulip server and run the following
to back up the system:
```bash
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
```
3. Follow [Debian's instructions to upgrade the OS][debian-upgrade-os].
@@ -420,7 +498,7 @@ instructions for other supported platforms.
4. As root, upgrade the database installation and OS configuration to
match the new OS version:
```
```bash
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
pg_dropcluster 11 main --stop
@@ -436,7 +514,7 @@ instructions for other supported platforms.
among other things will recompile Zulip's Python module
dependencies for your new version of Python:
```
```bash
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
@@ -449,9 +527,18 @@ instructions for other supported platforms.
6. [Upgrade to the latest Zulip release](#upgrading-to-a-release), now
that your server is running a supported operating system.
7. As root, finish by verifying the contents of the full-text indexes:
7. Debian Buster has a different version of the low-level glibc
library, which affects how PostgreSQL orders text data (known as
"collations"); this corrupts database indexes that rely on
collations. Regenerate the affected indexes by running:
```bash
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
```
8. As root, finish by verifying the contents of the full-text indexes:
```bash
/home/zulip/deployments/current/manage.py audit_fts_indexes
```
@@ -467,23 +554,34 @@ To upgrade the version of PostgreSQL on the Zulip server:
1. Upgrade your server to the latest Zulip release (at least 3.0).
2. Stop the server and take a backup:
1. Stop the server, as the `zulip` user:
```bash
# On Zulip before 4.0, use `supervisor stop all` instead
/home/zulip/deployments/current/scripts/stop-server
```
supervisorctl stop all
1. Take a backup, in case of any problems:
```bash
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/postgresql-upgrade.backup.tar.gz
```
3. As root, run the database upgrade tool:
1. As root, run the database upgrade tool:
```
```bash
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
```
`upgrade-postgresql` will have finished by restarting your Zulip server;
you should now be able to navigate to its URL and confirm everything
is working correctly.
1. As the `zulip` user, start the server again:
```bash
# On Zulip before 4.0, use `restart-server` instead of `start-server` instead
/home/zulip/deployments/current/scripts/start-server
```
You should now be able to navigate to the Zulip server's URL and
confirm everything is working correctly.
## Modifying Zulip
@@ -495,15 +593,15 @@ If you do modify Zulip and then report an issue you see in your
modified version of Zulip, please be responsible about communicating
that fact:
* Ideally, you'd reproduce the issue in an unmodified version (e.g. on
- Ideally, you'd reproduce the issue in an unmodified version (e.g. on
[chat.zulip.org](../contributing/chat-zulip-org.md) or
[zulip.com](https://zulip.com)).
* Where that is difficult or you think it's very unlikely your changes
- Where that is difficult or you think it's very unlikely your changes
are related to the issue, just mention your changes in the issue report.
If you're looking to modify Zulip by applying changes developed by the
Zulip core team and merged into master, skip to [this
section](#applying-changes-from-master).
Zulip core team and merged into `main`, skip to [this
section](#applying-changes-from-main).
## Making changes
@@ -512,43 +610,43 @@ One way to modify Zulip is to just edit files under
can work OK for testing small changes to Python code or shell scripts.
But we don't recommend this approach for maintaining changes because:
* You cannot modify JavaScript, CSS, or other frontend files this way,
- You cannot modify JavaScript, CSS, or other frontend files this way,
because we don't include them in editable form in our production
release tarballs (doing so would make our release tarballs much
larger without any runtime benefit).
* You will need to redo your changes after you next upgrade your Zulip
- You will need to redo your changes after you next upgrade your Zulip
server (or they will be lost).
* You need to remember to restart the server or your changes won't
- You need to remember to restart the server or your changes won't
have effect.
* Your changes aren't tracked, so mistakes can be hard to debug.
- Your changes aren't tracked, so mistakes can be hard to debug.
Instead, we recommend the following GitHub-based workflow (see [our
Git guide][git-guide] if you need a primer):
* Decide where you're going to edit Zulip's code. We recommend [using
- Decide where you're going to edit Zulip's code. We recommend [using
the Zulip development environment](../development/overview.md) on
a desktop or laptop as it will make it extremely convenient for you
to test your changes without deploying them in production. But if
your changes are small or you're OK with risking downtime, you don't
strictly need it; you just need an environment with Git installed.
* **Important**. Determine what Zulip version you're running on your
- **Important**. Determine what Zulip version you're running on your
server. You can check by inspecting `ZULIP_VERSION` in
`/home/zulip/deployments/current/version.py` (we'll use `2.0.4`
below). If you apply your changes to the wrong version of Zulip,
it's likely to fail and potentially cause downtime.
* [Fork and clone][fork-clone] the [zulip/zulip][] repository on
- [Fork and clone][fork-clone] the [zulip/zulip][] repository on
[GitHub](https://github.com).
* Create a branch (named `acme-branch` below) containing your changes:
- Create a branch (named `acme-branch` below) containing your changes:
```
```bash
cd zulip
git checkout -b acme-branch 2.0.4
```
* Use your favorite code editor to modify Zulip.
* Commit your changes and push them to GitHub:
- Use your favorite code editor to modify Zulip.
- Commit your changes and push them to GitHub:
```
```bash
git commit -a
# Use `git diff` to verify your changes are what you expect
@@ -558,7 +656,7 @@ git diff 2.0.4 acme-branch
git push origin +acme-branch
```
* Log in to your Zulip server and configure and use
- Log in to your Zulip server and configure and use
[upgrade-zulip-from-git][] to install the changes; remember to
configure `git_repo_url` to point to your fork on GitHub and run it as
`upgrade-zulip-from-git acme-branch`.
@@ -574,9 +672,9 @@ Eventually, you'll want to upgrade to a new Zulip release. If your
changes were integrated into that Zulip release or are otherwise no
longer needed, you can just [upgrade as
usual](#upgrading-to-a-release). If you [upgraded to
master](#upgrading-to-master); review that section again; new
`main`](#upgrading-to-main); review that section again; new
maintenance releases are likely "older" than your current installation
and you might need to upgrade to the master again rather than to the
and you might need to upgrade to `main` again rather than to the
new maintenance release.
Otherwise, you'll need to update your branch by rebasing your changes
@@ -584,7 +682,7 @@ Otherwise, you'll need to update your branch by rebasing your changes
repository). The example below assumes you have a branch off of 2.0.4
and want to upgrade to 2.1.0.
```
```bash
cd zulip
git fetch --tags upstream
git checkout acme-branch
@@ -605,17 +703,17 @@ branch, as before.
If you are using [docker-zulip][], there are two things that are
different from the above:
* Because of how container images work, editing files directly is even
- Because of how container images work, editing files directly is even
more precarious, because Docker is designed for working with
container images and may lose your changes.
* Instead of running `upgrade-zulip-from-git`, you will need to use
- Instead of running `upgrade-zulip-from-git`, you will need to use
the [docker upgrade workflow][docker-zulip-upgrade] to build a
container image based on your modified version of Zulip.
[docker-zulip]: https://github.com/zulip/docker-zulip
[docker-zulip-upgrade]: https://github.com/zulip/docker-zulip#upgrading-from-a-git-repository
## Applying changes from master
## Applying changes from `main`
If you are experiencing an issue that has already been fixed by the
Zulip development community, and you'd like to get the fix now, you
@@ -627,7 +725,7 @@ fixes on your local Zulip server without waiting for an official release.
Many bugs have small/simple fixes. In this case, you can use the Git
workflow [described above](#making-changes), using:
```
```bash
git fetch upstream
git cherry-pick abcd1234
```
@@ -637,7 +735,7 @@ of the change you'd like).
In general, we can't provide unpaid support for issues caused by
cherry-picking arbitrary commits if the issues don't also affect
master or an official release.
`main` or an official release.
The exception to this rule is when we ask or encourage a user to apply
a change to their production system to help verify the fix resolves
@@ -651,21 +749,21 @@ addition to scheduling that change for Zulip's next bug fix release,
we support changes in stable release branches as though they were
released.
### Upgrading to master
### Upgrading to `main`
Many Zulip servers (including chat.zulip.org and zulip.com) upgrade to
master on a regular basis to get the latest features. Before doing
`main` on a regular basis to get the latest features. Before doing
so, it's important to understand how to happily run a server based on
master.
`main`.
For background, it's backporting arbitrary patches from master to an
For background, it's backporting arbitrary patches from `main` to an
older version requires some care. Common issues include:
* Changes containing database migrations (new files under
- Changes containing database migrations (new files under
`*/migrations/`), which includes most new features. We
don't support applying database migrations out of order.
* Changes that are stacked on top of other changes to the same system.
* Essentially any patch with hundreds of lines of changes will have
- Changes that are stacked on top of other changes to the same system.
- Essentially any patch with hundreds of lines of changes will have
merge conflicts and require extra work to apply.
While it's possible to backport these sorts of changes, you're
@@ -673,46 +771,46 @@ unlikely to succeed without help from the core team via a support
contract.
If you need an unreleased feature, the best path is usually to
upgrade to Zulip master using [upgrade-zulip-from-git][]. Before
upgrading to master, make sure you understand:
upgrade to Zulip `main` using [upgrade-zulip-from-git][]. Before
upgrading to `main`, make sure you understand:
* In Zulip's version numbering scheme, `master` will always be "newer"
- In Zulip's version numbering scheme, `main` will always be "newer"
than the latest maintenance release (E.g. `3.1` or `2.1.6`) and
"older" than the next major release (E.g. `3.0` or `4.0`).
* The `master` branch is under very active development; dozens of new
changes are integrated into it on most days. The `master` branch
- The `main` branch is under very active development; dozens of new
changes are integrated into it on most days. The `main` branch
can have thousands of changes not present in the latest release (all
of which will be included in our next major release). On average
`master` usually has fewer total bugs than the latest release
`main` usually has fewer total bugs than the latest release
(because we fix hundreds of bugs in every major release) but it
might have some bugs that are more severe than we would consider
acceptable for a release.
* We deploy `master` to chat.zulip.org and zulip.com on a regular
- We deploy `main` to chat.zulip.org and zulip.com on a regular
basis (often daily), so it's very important to the project that it
be stable. Most regressions will be minor UX issues or be fixed
quickly, because we need them to be fixed for Zulip Cloud.
* The development community is very interested in helping debug issues
that arise when upgrading from the latest release to master, since
- The development community is very interested in helping debug issues
that arise when upgrading from the latest release to `main`, since
they provide us an opportunity to fix that category of issue before
our next major release. (Much more so than we are in helping folks
debug other custom changes). That said, we cannot make any
guarantees about how quickly we'll resolve an issue to folks without
a formal support contract.
* We do not support downgrading from `master` to earlier versions, so
- We do not support downgrading from `main` to earlier versions, so
if downtime for your Zulip server is unacceptable, make sure you
have a current
[backup](../production/export-and-import.html#backups) in case the
upgrade fails.
* Our changelog contains [draft release
- Our changelog contains [draft release
notes](../overview/changelog.md) available listing major changes
since the last release. The **Upgrade notes** section will always
be current, even if some new features aren't documented.
* Whenever we push a security or maintenance release, the changes in
that release will always be merged to master; so you can get the
security fixes by upgrading to master.
* You can always upgrade from master to the next major release when it
- Whenever we push a security or maintenance release, the changes in
that release will always be merged to `main`; so you can get the
security fixes by upgrading to `main`.
- You can always upgrade from `main` to the next major release when it
comes out, using either [upgrade-zulip-from-git][] or the release
tarball. So there's no risk of upgrading to `master` resulting in
tarball. So there's no risk of upgrading to `main` resulting in
a system that's not upgradeable back to a normal release.
## Contributing patches

View File

@@ -50,7 +50,7 @@ as world-readable, whereas the "uploaded files" one is not.
With Zulip 1.9.0 and newer, you can do this automatically with the
following commands run as root:
```
```bash
crudini --set /etc/zulip/zulip.conf application_server no_serve_uploads true
/home/zulip/deployments/current/scripts/zulip-puppet-apply
```
@@ -59,9 +59,9 @@ as world-readable, whereas the "uploaded files" one is not.
With older Zulip, you need to edit
`/etc/nginx/sites-available/zulip-enterprise` to comment out the
`nginx` configuration block for `/user_avatars` and the `include
/etc/nginx/zulip-include/uploads.route` line and then reload the
`nginx` service (`service nginx reload`).
`nginx` configuration block for `/user_avatars` and the
`include /etc/nginx/zulip-include/uploads.route` line and then
reload the `nginx` service (`service nginx reload`).
1. Finally, restart the Zulip server so that your settings changes
take effect
@@ -83,7 +83,7 @@ each of the two buckets, you'll want to
[add an S3 bucket policy](https://awspolicygen.s3.amazonaws.com/policygen.html)
entry that looks something like this:
```
```json
{
"Version": "2012-10-17",
"Id": "Policy1468991802321",
@@ -117,7 +117,7 @@ entry that looks something like this:
The avatars bucket is intended to be world-readable, so you'll also
need a block like this:
```
```json
{
"Sid": "Stmt1468991795389",
"Effect": "Allow",
@@ -127,7 +127,6 @@ need a block like this:
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
}
```
The file-uploads bucket should not be world-readable. See the

View File

@@ -15,18 +15,18 @@ installation, you'll need to register a custom Zoom app as follows:
1. Create an app with the **OAuth** type.
* Choose an app name such as "ExampleCorp Zulip".
* Select **User-managed app**.
* Disable the option to publish the app on the Marketplace.
* Click **Create**.
- Choose an app name such as "ExampleCorp Zulip".
- Select **User-managed app**.
- Disable the option to publish the app on the Marketplace.
- Click **Create**.
1. Inside of the Zoom app management page:
* On the **App Credentials** tab, set both the **Redirect URL for
- On the **App Credentials** tab, set both the **Redirect URL for
OAuth** and the **Whitelist URL** to
`https://zulip.example.com/calls/zoom/complete` (replacing
`zulip.example.com` by your main Zulip hostname).
* On the **Scopes** tab, add the `meeting:write` scope.
- On the **Scopes** tab, add the `meeting:write` scope.
You can then configure your Zulip server to use that Zoom app as
follows:

View File

@@ -29,7 +29,7 @@ There are three main components:
The next several sections will dive into the details of these components.
## The *Count database tables
## The \*Count database tables
The Zulip analytics system is built around collecting time series data in a
set of database tables. Each of these tables has the following fields:
@@ -76,7 +76,7 @@ by the system and with what data.
## The FillState table
The default Zulip production configuration runs a cron job once an hour that
updates the *Count tables for each of the CountStats in the COUNT_STATS
updates the \*Count tables for each of the CountStats in the COUNT_STATS
dictionary. The FillState table simply keeps track of the last end_time that
we successfully updated each stat. It also enables the analytics system to
recover from errors (by retrying) and to monitor that the cron job is
@@ -103,7 +103,7 @@ There are a few important principles that we use to make the system
efficient:
- Not repeating work to keep things up to date (via FillState)
- Storing data in the *Count tables to avoid our endpoints hitting the core
- Storing data in the \*Count tables to avoid our endpoints hitting the core
Message/UserMessage tables is key, because some queries could take minutes
to calculate. This allows any expensive operations to run offline, and
then the endpoints to server data to users can be fast.
@@ -119,7 +119,7 @@ efficient:
each user, we just query for each user, and then add up the numbers for
the users to get the totals for the realm.
- Not storing rows when the value is 0. An hourly user stat would otherwise
collect 24 * 365 * roughly .5MB per db row = 4GB of data per user per
collect 24 \* 365 \* roughly .5MB per db row = 4GB of data per user per
year, most of whose values are 0. A related note is to be cautious about
adding queries that are typically non-0 instead of being typically 0.
@@ -137,11 +137,11 @@ system:
- Tests for the backend views code logic for extracting data from the
database and serving it to clients.
For manual backend testing, it sometimes can be valuable to use `./manage.py
dbshell` to inspect the tables manually to check that things look right; but
usually anything you feel the need to check manually, you should add some
sort of assertion for to the backend analytics tests, to make sure it stays
that way as we refactor.
For manual backend testing, it sometimes can be valuable to use
`./manage.py dbshell` to inspect the tables manually to check that
things look right; but usually anything you feel the need to check
manually, you should add some sort of assertion for to the backend
analytics tests, to make sure it stays that way as we refactor.
## LoggingCountStats

View File

@@ -4,8 +4,9 @@ Zulip uses a third party (Stripe) for billing, so working on the billing
system requires a little bit of setup.
To set up the development environment to work on the billing code:
* Create a Stripe account
* Go to <https://dashboard.stripe.com/account/apikeys>, and add the
- Create a Stripe account
- Go to <https://dashboard.stripe.com/account/apikeys>, and add the
publishable key and secret key as `stripe_publishable_key` and
`stripe_secret_key` to `zproject/dev-secrets.conf`.
@@ -15,13 +16,14 @@ Nearly all the billing-relevant code lives in `corporate/`.
Stripe makes pretty regular updates to their API. The process for upgrading
our code is:
* Go to <https://dashboard.stripe.com/developers> in your Stripe account.
* Upgrade the API version.
* Run `tools/test-backend --generate-stripe-fixtures`
* Fix any failing tests, and manually look through `git diff` to understand
- Go to <https://dashboard.stripe.com/developers> in your Stripe account.
- Upgrade the API version.
- Run `tools/test-backend --generate-stripe-fixtures`
- Fix any failing tests, and manually look through `git diff` to understand
the changes.
* If there are no material changes, commit the diff, and open a PR.
* Ask Rishi or Tim to go to <https://dashboard.stripe.com/developers> in the
- If there are no material changes, commit the diff, and open a PR.
- Ask Rishi or Tim to go to <https://dashboard.stripe.com/developers> in the
zulipchat Stripe account, and upgrade the API version there.
We currently aren't set up to do version upgrades where there are breaking

View File

@@ -73,7 +73,8 @@ def get_user(email: str, realm: Realm) -> UserProfile:
```
This decorator implements a pretty classic caching paradigm:
* The `user_profile_cache_key` function defines a unique map from a
- The `user_profile_cache_key` function defines a unique map from a
canonical form of its arguments to a string. These strings are
namespaced (the `user_profile:` part) so that they won't overlap
with other caches, and encode the arguments so that two uses of this
@@ -82,13 +83,13 @@ This decorator implements a pretty classic caching paradigm:
is important to ensure we don't send special characters to
memcached). And we have two versions, depending whether the caller
has access to a `Realm` or just a `realm_id`.
* When `get_user` is called, `cache_with_key` will compute the key,
- When `get_user` is called, `cache_with_key` will compute the key,
and do a Django `cache_get` query for the key (which goes to
memcached). If the key is in the cache, it just returns the value.
Otherwise, it fetches the value from the database (using the actual
code in the body of `get_user`), and then stores the value back to
that memcached key before returning the result to the caller.
* Cache entries expire after the timeout; in this case, a week.
- Cache entries expire after the timeout; in this case, a week.
Though in frequently deployed environments like chat.zulip.org,
often cache entries will stop being used long before that, because
`KEY_PREFIX` is rotated every time we deploy to production; see
@@ -104,7 +105,7 @@ that if an item is in the cache, the body of `get_user` (above) is
never called. This means some things that might seem like clever code
reuse are actually a really bad idea. For example:
* Don't add a `get_active_user` function that uses the same cache key
- Don't add a `get_active_user` function that uses the same cache key
function as `get_user` (but with a different query that filters our
deactivated users). If one called `get_active_user` to access a
deactivated user, the right thing would happen, but if you called
@@ -132,7 +133,7 @@ you configure some code to run every time Django does something (for
There's a handful of lines in `zerver/models.py` like these that
configure this:
```
```python
post_save.connect(flush_realm, sender=Realm)
post_save.connect(flush_user_profile, sender=UserProfile)
```
@@ -237,11 +238,12 @@ objects to minimize data transfer between Django and memcached).
We generally try to avoid in-process backend caching in Zulip's Django
codebase, because every Zulip production installation involves
multiple servers. We do have a few, however:
* `per_request_display_recipient_cache`: A cache flushed at the start
- `per_request_display_recipient_cache`: A cache flushed at the start
of every request; this simplifies correctly implementing our goal of
not repeatedly fetching the "display recipient" (e.g. stream name)
for each message in the `GET /messages` codebase.
* Caches of various data, like the `SourceMap` object, that are
- Caches of various data, like the `SourceMap` object, that are
expensive to construct, not needed for most requests, and don't
change once a Zulip server has been deployed in production.
@@ -260,5 +262,4 @@ cached by clients is changed. Clients are responsible for handling
the events, updating their state, and rerendering any UI components
that might display the modified state.
[post-save-signals]:
https://docs.djangoproject.com/en/2.0/ref/signals/#post-save
[post-save-signals]: https://docs.djangoproject.com/en/2.0/ref/signals/#post-save

View File

@@ -6,15 +6,15 @@ this document, we discuss the various classes of dependencies that
Zulip has, and how we manage them. Zulip's dependency management has
some really nice properties:
* **Fast provisioning**. When switching to a different commit in the
- **Fast provisioning**. When switching to a different commit in the
Zulip project with the same dependencies, it takes under 5 seconds
to re-provision a working Zulip development environment after
switching. If there are new dependencies, one only needs to wait to
download the new ones, not all the pre-existing dependencies.
* **Consistent provisioning**. Every time a Zulip development or
- **Consistent provisioning**. Every time a Zulip development or
production environment is provisioned/installed, it should end up
using the exactly correct versions of all major dependencies.
* **Low maintenance burden**. To the extent possible, we want to
- **Low maintenance burden**. To the extent possible, we want to
avoid manual work and keeping track of things that could be
automated. This makes it easy to keep running the latest versions
of our various dependencies.
@@ -103,10 +103,11 @@ on specific versions of these packages wherever possible.
The exact lists of `apt` packages needed by Zulip are maintained in a
few places:
* For production, in our Puppet configuration, `puppet/zulip/`, using
- For production, in our Puppet configuration, `puppet/zulip/`, using
the `Package` and `SafePackage` directives.
* For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`.
* The packages needed to build a Zulip virtualenv, in
- For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`.
- The packages needed to build a Zulip virtualenv, in
`VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are
separate from the rest because (1) we may need to install a
virtualenv before running the more complex scripts that, in turn,
@@ -124,7 +125,7 @@ about how Zulip makes this system work well for us that are worth
highlighting. The system is largely managed by the code in
`scripts/lib/setup_venv.py`
* **Using `pip` to manage dependencies**. This is standard in the
- **Using `pip` to manage dependencies**. This is standard in the
Python ecosystem, and means we only need to record a list of
versions in a `requirements.txt` file to declare what we're using.
Since we have a few different installation targets, we maintain
@@ -134,7 +135,7 @@ highlighting. The system is largely managed by the code in
majority of packages common to prod and development, etc.). We use
`pip install --no-deps` to ensure we only install the packages we
explicitly declare as dependencies.
* **virtualenv with pinned versions**. For a large application like
- **virtualenv with pinned versions**. For a large application like
Zulip, it is important to ensure that we're always using consistent,
predictable versions of all of our Python dependencies. To ensure
this, we install our dependencies in a [virtualenv][] that contains
@@ -145,7 +146,7 @@ highlighting. The system is largely managed by the code in
effect is that it's easy to debug problems caused by dependency
upgrades, since we're always doing those upgrades with an explicit
commit updating the `requirements/` directory.
* **Pinning versions of indirect dependencies**. We "pin" or "lock"
- **Pinning versions of indirect dependencies**. We "pin" or "lock"
the versions of our indirect dependencies files with
`tools/update-locked-requirements` (powered by `pip-compile`). What
this means is that we have some "source" requirements files, like
@@ -153,16 +154,16 @@ highlighting. The system is largely managed by the code in
depends on directly. Those packages have their own recursive
dependencies. When adding or removing a dependency from Zulip, one
simply edits the appropriate "source" requirements files, and then
runs `tools/update-locked-requirements`. That tool will use `pip
compile` to generate the locked requirements files like `prod.txt`,
`dev.txt` etc files that explicitly declare versions of all of
Zulip's recursive dependencies. For indirect dependencies (i.e.
dependencies not explicitly declared in the source requirements files),
it provides helpful comments explaining which direct dependency (or
dependencies) needed that indirect dependency. The process for
using this system is documented in more detail in
runs `tools/update-locked-requirements`. That tool will use
`pip-compile` to generate the locked requirements files like
`prod.txt`, `dev.txt` etc files that explicitly declare versions of
all of Zulip's recursive dependencies. For indirect dependencies
(i.e. dependencies not explicitly declared in the source
requirements files), it provides helpful comments explaining which
direct dependency (or dependencies) needed that indirect dependency.
The process for using this system is documented in more detail in
`requirements/README.md`.
* **Caching of virtualenvs and packages**. To make updating the
- **Caching of virtualenvs and packages**. To make updating the
dependencies of a Zulip installation efficient, we maintain a cache
of virtualenvs named by the hash of the relevant `requirements.txt`
file (`scripts/lib/hash_reqs.py`). These caches live under
@@ -175,21 +176,21 @@ highlighting. The system is largely managed by the code in
needed, making small version upgrades extremely efficient. And
finally, we use `pip`'s built-in caching to ensure that a specific
version of a specific package is only downloaded once.
* **Garbage-collecting caches**. We have a tool,
- **Garbage-collecting caches**. We have a tool,
`scripts/lib/clean_venv_cache.py`, which will clean old cached
virtualenvs that are no longer in use. In production, the algorithm
preserves recent virtualenvs as well as those in use by any current
production deployment directory under `/home/zulip/deployments/`.
This helps ensure that a Zulip installation doesn't leak large
amounts of disk over time.
* **Scripts**. Often, we want a script running in production to use
- **Scripts**. Often, we want a script running in production to use
the Zulip virtualenv. To make that work without a lot of duplicated
code, we have a helpful function,
`scripts.lib.setup_path.setup_path`, which on import will put the
currently running Python script into the Zulip virtualenv. This is
called by `./manage.py` to ensure that our Django code always uses
the correct virtualenv as well.
* **Mypy type checker**. Because we're using mypy in a strict mode,
- **Mypy type checker**. Because we're using mypy in a strict mode,
when you add use of a new Python dependency, you usually need to
either adds stubs to the `stubs/` directory for the library, or edit
`mypy.ini` in the root of the Zulip project to configure
@@ -202,7 +203,7 @@ See the [README][requirements-readme] file in `requirements/` directory
to learn how to upgrade a single Python package.
[mypy-docs]: ../testing/mypy.md
[requirements-readme]: https://github.com/zulip/zulip/blob/master/requirements/README.md#requirements
[requirements-readme]: https://github.com/zulip/zulip/blob/main/requirements/README.md#requirements
[stack-overflow]: https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
[caching]: https://help.github.com/en/articles/caching-your-github-password-in-git
@@ -212,19 +213,19 @@ We use the same set of strategies described for Python dependencies
for most of our JavaScript dependencies, so we won't repeat the
reasoning here.
* In a fashion very analogous to the Python codebase,
- In a fashion very analogous to the Python codebase,
`scripts/lib/node_cache.py` manages cached `node_modules`
directories in `/srv/zulip-npm-cache`. Each is named by its hash,
computed by the `generate_sha1sum_node_modules` function.
`scripts/lib/clean_node_cache.py` handles garbage-collection.
* We use [yarn][], a `pip`-like tool for JavaScript, to download most
- We use [yarn][], a `pip`-like tool for JavaScript, to download most
JavaScript dependencies. Yarn talks to standard the [npm][]
repository. We use the standard `package.json` file to declare our
direct dependencies, with sections for development and
production. Yarn takes care of pinning the versions of indirect
dependencies in the `yarn.lock` file; `yarn install` updates the
`yarn.lock` files.
* `tools/update-prod-static`. This process is discussed in detail in
- `tools/update-prod-static`. This process is discussed in detail in
the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline)
article, but we don't use the `node_modules` directories directly in
production. Instead, static assets are compiled using our static
@@ -233,7 +234,7 @@ reasoning here.
directory in a Zulip production release tarball, which is a good
thing, because doing so would more than double the size of a Zulip
release tarball.
* **Checked-in packages**. In contrast with Python, we have a few
- **Checked-in packages**. In contrast with Python, we have a few
JavaScript dependencies that we have copied into the main Zulip
repository under `static/third`, often with patches. These date
from an era before `npm` existed. It is a project goal to eliminate
@@ -248,11 +249,11 @@ its version) and `scripts/lib/third/install-yarn.sh` (the standard
installer for `yarn`, modified to support installing to a path that is
not the current user's home directory).
* `nvm` has its own system for installing each version of `node` at
- `nvm` has its own system for installing each version of `node` at
its own path, which we use, though we install a `/usr/local/bin/node`
wrapper to access the desired version conveniently and efficiently
(`nvm` has a lot of startup overhead).
* `install-yarn.sh` is configured to install `yarn` at
- `install-yarn.sh` is configured to install `yarn` at
`/srv/zulip-yarn`. We don't do anything special to try to manage
multiple versions of `yarn`.
@@ -305,16 +306,16 @@ our JavaScript Markdown processor has access to the supported list.
When making changes to Zulip's provisioning process or dependencies,
usually one needs to think about making changes in 3 places:
* `tools/lib/provision.py`. This is the main provisioning script,
- `tools/lib/provision.py`. This is the main provisioning script,
used by most developers to maintain their development environment.
* `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation"
- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation"
documentation. Strategically, we'd like to move the support for more
versions of Linux from here into `tools/lib/provision.py`.
* Production. Our tools for compiling/generating static assets need
- Production. Our tools for compiling/generating static assets need
to be called from `tools/update-prod-static`, which is called by
`tools/build-release-tarball` (for doing Zulip releases) as well as
`tools/upgrade-zulip-from-git` (for deploying a Zulip server off of
master).
`main`).
[virtualenv]: https://virtualenv.pypa.io/en/stable/
[virtualenv-clone]: https://github.com/edwardgeorge/virtualenv-clone/

View File

@@ -3,30 +3,30 @@
This article documents notes on the process for upgrading Zulip to
new major versions of Django. Here are the steps:
* Carefully read the Django upstream changelog, and `git grep` to
- Carefully read the Django upstream changelog, and `git grep` to
check if we're using anything deprecated or significantly modified
and put them in an issue (and then starting working through them).
Also, note any new features we might want to use after the upgrade,
and open an issue listing them;
[example](https://github.com/zulip/zulip/issues/2564).
* Start submitting PRs to do any deprecation-type migrations that work
- Start submitting PRs to do any deprecation-type migrations that work
on both the old and new version of Django. The goal here is to have
the actual cutover commit be as small as possible, and to test as
much of the changes for the migration as we can independently from
the big cutover.
* Check the version support of the third-party Django packages we use
- Check the version support of the third-party Django packages we use
(`git grep django requirements/` to see a list), upgrade any as
needed and file bugs upstream for any that lack support. Look into
fixing said bugs.
* Look at the pieces of Django code that we've copied and then
- Look at the pieces of Django code that we've copied and then
adapted, and confirm whether Django has any updates to the modified
code we should apply. Partial list:
* `CursorDebugWrapper`, which we have a modified version of in
- `CursorDebugWrapper`, which we have a modified version of in
`zerver/lib/db.py`. See
[the issue for contributing this upstream](https://github.com/zulip/zulip/issues/974)
* `PasswordResetForm` and any other forms we import from
- `PasswordResetForm` and any other forms we import from
`django.contrib.auth.forms` in `zerver/forms.py` (which has all of
our Django forms).
* Our AsyncDjangoHandler class has some code copied from the core
- Our AsyncDjangoHandler class has some code copied from the core
Django handlers code; look at whether that code was changed in
Django upstream.

View File

@@ -11,32 +11,33 @@ our instructions for
On to the documentation. Zulip's email system is fairly straightforward,
with only a few things you need to know to get started.
* All email templates are in `templates/zerver/emails/`. Each email has three
- All email templates are in `templates/zerver/emails/`. Each email has three
template files: `<template_prefix>.subject.txt`, `<template_prefix>.txt`, and
`<template_prefix>.source.html`. Email templates, along with all other templates
in the `templates/` directory, are Jinja2 templates.
* Most of the CSS and HTML layout for emails is in `email_base.html`. Note
- Most of the CSS and HTML layout for emails is in `email_base.html`. Note
that email has to ship with all of its CSS and HTML, so nothing in
`static/` is useful for an email. If you're adding new CSS or HTML for an
email, there's a decent chance it should go in `email_base.html`.
* All email is eventually sent by `zerver.lib.send_email.send_email`. There
- All email is eventually sent by `zerver.lib.send_email.send_email`. There
are several other functions in `zerver.lib.send_email`, but all of them
eventually call the `send_email` function. The most interesting one is
`send_future_email`. The `ScheduledEmail` entries are eventually processed
by a supervisor job that runs `zerver/management/commands/deliver_scheduled_emails.py`.
* Always use `user_profile.delivery_email`, not `user_profile.email`,
- Always use `user_profile.delivery_email`, not `user_profile.email`,
when passing data into the `send_email` library. The
`user_profile.email` field may not always be valid.
* A good way to find a bunch of example email pathways is to `git grep` for
- A good way to find a bunch of example email pathways is to `git grep` for
`zerver/emails` in the `zerver/` directory.
One slightly complicated decision you may have to make when adding an email
is figuring out how to schedule it. There are 3 ways to schedule email.
* Send it immediately, in the current Django process, e.g. by calling
- Send it immediately, in the current Django process, e.g. by calling
`send_email` directly. An example of this is the `confirm_registration`
email.
* Add it to a queue. An example is the `invitation` email.
* Send it (approximately) at a specified time in the future, using
- Add it to a queue. An example is the `invitation` email.
- Send it (approximately) at a specified time in the future, using
`send_future_email`. An example is the `followup_day2` email.
Email takes about a quarter second per email to process and send. Generally
@@ -53,10 +54,10 @@ we've set the email backend (aka what happens when you call the email
`.send()` method in Django) in the development environment to be our
custom backend, `EmailLogBackEnd`. It does the following:
* Logs any sent emails to `var/log/email_content.log`. This log is
- Logs any sent emails to `var/log/email_content.log`. This log is
displayed by the `/emails` endpoint
(e.g. http://zulip.zulipdev.com:9991/emails).
* Print a friendly message on console advertising `/emails` to make
- Print a friendly message on console advertising `/emails` to make
this nice and discoverable.
### Testing in a real email client
@@ -81,16 +82,16 @@ Once you have the login credentials of the SMTP provider, since there
is not `/etc/zulip/settings.py` in development, configure it using the
following keys in `zproject/dev-secrets.conf`
* `email_host` - SMTP hostname.
* `email_port` - SMTP port.
* `email_host_user` - Username of the SMTP user
* `email_password` - Password of the SMTP user.
* `email_use_tls` - Set to `true` for most providers. Else, don't set any value.
- `email_host` - SMTP hostname.
- `email_port` - SMTP port.
- `email_host_user` - Username of the SMTP user
- `email_password` - Password of the SMTP user.
- `email_use_tls` - Set to `true` for most providers. Else, don't set any value.
Here is an example of how `zproject/dev-secrets.conf` might look if
you are using Gmail.
```
```ini
email_host = smtp.gmail.com
email_port = 587
email_host_user = username@gmail.com
@@ -103,18 +104,18 @@ email_password = gmail_password
### Notes
* After changing any HTML email or `email_base.html`, you need to run
- After changing any HTML email or `email_base.html`, you need to run
`scripts/setup/inline_email_css.py` for the changes to be reflected
in the development environment. The script generates files like
`templates/zerver/emails/compiled/<template_prefix>.html`.
* Images won't be displayed in a real email client unless you change
- Images won't be displayed in a real email client unless you change
the `base_image_uri` used for emails to a public URL such as
`https://chat.zulip.org/static/images/emails` (image links to
`localhost:9991` aren't allowed by modern email providers). See
`zproject/email_backends.py` for more details.
* While running the backend test suite, we use
- While running the backend test suite, we use
`django.core.mail.backends.locmem.EmailBackend` as the email
backend. The `locmem` backend stores messages in a special attribute
of the django.core.mail module, "outbox". The outbox attribute is

View File

@@ -6,10 +6,10 @@ document discusses a number of these issues.
Currently, Zulip supports these four display formats for emoji:
* Google modern
* Google classic
* Twitter
* Plain text
- Google modern
- Google classic
- Twitter
- Plain text
## Emoji codes
@@ -68,15 +68,16 @@ emoji tooling). See [our dependencies document](../subsystems/dependencies.md)
for more details on this strategy.
The emoji tree generated by this process contains several import elements:
* `emoji_codes.json`: A set of mappings used by the Zulip frontend to
- `emoji_codes.json`: A set of mappings used by the Zulip frontend to
understand what Unicode emoji exist and what their shortnames are,
used for autocomplete, emoji pickers, etc. This has been
deduplicated using the logic in
`tools/setup/emoji/emoji_setup_utils.py` to generally only have
`:angry:` and not also `:angry_face:`, since having both is ugly and
pointless for purposes like autocomplete and emoji pickers.
* `images/emoji/unicode/*.png`: A farm of emoji
* `images/emoji/*.png`: A farm of symlinks from emoji names to the
- `images/emoji/unicode/*.png`: A farm of emoji
- `images/emoji/*.png`: A farm of symlinks from emoji names to the
`images/emoji/unicode/` tree. This is used to serve individual emoji
images, as well as for the
[backend Markdown processor](../subsystems/markdown.md) to know which emoji
@@ -84,7 +85,7 @@ The emoji tree generated by this process contains several import elements:
tree, we currently include all of the emoji in `emoji-map.json`;
this means that if you send `:angry_face:`, it won't autocomplete,
but will still work (but not in previews).
* Some CSS and PNGs for the emoji spritesheets, used in Zulip for
- Some CSS and PNGs for the emoji spritesheets, used in Zulip for
emoji pickers where we would otherwise need to download over 1000 of
individual emoji images (which would cause a browser performance
problem). We have multiple spritesheets: one for each emoji
@@ -102,36 +103,36 @@ The following set of considerations is not comprehensive, but has a few
principles that were applied to the current set of names. We use (strong),
(medium), and (weak) denote how strong a consideration it is.
* Even with over 1000 symbols, emoji feels surprisingly sparse as a language,
- Even with over 1000 symbols, emoji feels surprisingly sparse as a language,
and more often than not, if you search for something, you don't find an
appropriate emoji for it. So a primary goal for our set of names is to
maximize the number of situations in which the user finds an emoji that
feels appropriate. (strong)
* Conversely, we remove generic words that will gum up the typeahead. So
- Conversely, we remove generic words that will gum up the typeahead. So
`:outbox:` instead of `:outbox_tray:`. Each word should count. (medium)
* We aim for the set of names to be as widely culturally applicable as
- We aim for the set of names to be as widely culturally applicable as
possible, even if the glyphs are not. So `:statue:` instead of
`:new_york:` for the statue of liberty, and `:tower:` instead of
`:tokyo_tower:`. (strong)
* We remove unnecessary gender descriptions. So `:ok_signal:` instead of
- We remove unnecessary gender descriptions. So `:ok_signal:` instead of
`:ok_woman:`. (strong)
* We don't add names that could be inappropriate in school or work
- We don't add names that could be inappropriate in school or work
environments, even if the use is common on the internet. For example, we
have not added `:butt:` for `:peach:`, or `:cheers:` for
`:beers:`. (strong)
* Names should be compatible with the four emoji sets we support, but don't
- Names should be compatible with the four emoji sets we support, but don't
have to be compatible with any other emoji set. (medium)
* We try not to use a creative canonical_name for emoji that are likely to
- We try not to use a creative canonical_name for emoji that are likely to
be familiar to a large subset of users. This largely applies to certain
faces. (medium)
* The set of names should be compatible with the iamcal, gemoji, and Unicode
- The set of names should be compatible with the iamcal, gemoji, and Unicode
names. Compatible here means that if there is an emoji name a user knows
from one of those sets, and the user searches for the key word of that
name, they will get an emoji in our set. It is okay if this emoji has a
@@ -142,26 +143,26 @@ Much of the work of picking names went into the first bullet above: making
the emoji language less sparse. Some tricks and heuristics that were used
for that:
* There are many near duplicates, like `:dog:` and `:dog_face:`, or
- There are many near duplicates, like `:dog:` and `:dog_face:`, or
`:mailbox:`, `:mailbox_with_mail:`, and `:mailbox_with_no_mail:`. In these
cases we repurpose the duplicates to be as useful as we can, like `:dog:`
and `:puppy:`, and `:mailbox:`, `:unread_mail:`, `:inbox_zero:` for the
ones above. There isn't a ton of flexibility, since we can't change the
glyphs. But in most cases we have been able to come up with something.
* Many emoji have commonly understood meanings among people that use emoji a
- Many emoji have commonly understood meanings among people that use emoji a
lot, and there are websites and articles that document some of these
meanings. A commonly understood meaning can be a great thing to add as an
alternate name, since often it is a sign that the meaning is addressing a
real gap in the emoji system.
* Many emoji names are unnecessarily specific in iamcal/etc, like
- Many emoji names are unnecessarily specific in iamcal/etc, like
`:flower_playing_cards:`, `:izakaya_lantern:`, or `:amphora:`. Renaming
them to `:playing_cards:`, `:lantern:`, and `:vase:` makes them more
widely usable. In such cases we often keep the specific name as an
alternate.
* If there are natural things someone might type, like `:happy:`, we try to
- If there are natural things someone might type, like `:happy:`, we try to
find an emoji to match. This extends to things that someone might not
think to type, but as soon as someone in the organization discovers it it
could get wide use, like `:working_on_it:`. Good future work would be to
@@ -171,7 +172,7 @@ for that:
Other notes
* Occasionally there are near duplicates where we don't have ideas for
- Occasionally there are near duplicates where we don't have ideas for
useful names for the second one. In that case we sometimes remove the
emoji rather than have two nearly identical glyphs in the emoji picker and
typeahead. For instance, we kept `:spiral_notepad:` and dropped
@@ -179,7 +180,7 @@ Other notes
of glyphs look very different, we'll find two names that allow them both
to stay.
* We removed many of the moons and clocks, to make the typeahead experience
- We removed many of the moons and clocks, to make the typeahead experience
better when searching for something that catches all the moons or all the
clocks. We kept all the squares and diamonds and other shapes, even though
they have the same problem, since they are commonly used to make emoji art

View File

@@ -34,11 +34,11 @@ little notification that the operation succeeded).
Architecturally, there are a few things needed to make a successful
real-time sync system work:
* **Generation**. Generating events when changes happen to data, and
- **Generation**. Generating events when changes happen to data, and
determining which users should receive each event.
* **Delivery**. Efficiently delivering those events to interested
- **Delivery**. Efficiently delivering those events to interested
clients, ideally in an exactly-once fashion.
* **UI updates**. Updating the UI in the client once it has received
- **UI updates**. Updating the UI in the client once it has received
events from the server.
Reactive JavaScript libraries like React and Vue can help simplify the
@@ -64,10 +64,10 @@ to be consumed by the delivery system.
Usually, this list of users is one of 3 things:
* A single user (e.g. for user-level settings changes).
* Everyone in the realm (e.g. for organization-level settings changes,
- A single user (e.g. for user-level settings changes).
- Everyone in the realm (e.g. for organization-level settings changes,
like new realm emoji).
* Everyone who would receive a given message (for messages, emoji
- Everyone who would receive a given message (for messages, emoji
reactions, message editing, etc.); i.e. the subscribers to a stream
or the people on a private message thread.
@@ -138,7 +138,7 @@ soon as it attempted to send them to the client; if that specific HTTP
response didn't reach the client due to a network TCP failure, then
those events could be lost).
[api-bindings-code]: https://github.com/zulip/python-zulip-api/blob/master/zulip/zulip/__init__.py
[api-bindings-code]: https://github.com/zulip/python-zulip-api/blob/main/zulip/zulip/__init__.py
The queue servers are a very high-traffic system, processing at a
minimum one request for every message delivered to every Zulip client.
@@ -175,10 +175,10 @@ anyway).
When a client starts up, it usually wants to get 2 things from the
server:
* The "current state" of various pieces of data, e.g. the current
- The "current state" of various pieces of data, e.g. the current
settings, set of users in the organization (for typeahead), stream,
messages, etc. (aka the "initial state").
* A subscription to receive updates to those data when they are
- A subscription to receive updates to those data when they are
changed by a client (aka an event queue).
Ideally, one would get those two things atomically, i.e. if some other
@@ -203,15 +203,15 @@ subroutines. Here's how it works when you make a `register` API
request; the logic is in `zerver/views/events_register.py` and
`zerver/lib/events.py`. The request is directly handled by Django:
* Django makes an HTTP request to Tornado, requesting that a new event
- Django makes an HTTP request to Tornado, requesting that a new event
queue be created, and records its queue ID.
* Django does all the various database/cache/etc. queries to fetch the
- Django does all the various database/cache/etc. queries to fetch the
data, non-atomically, from the various data sources (see
the `fetch_initial_state_data` function).
* Django makes a second HTTP request to Tornado, requesting any events
- Django makes a second HTTP request to Tornado, requesting any events
that had been added to the Tornado event queue since it
was created.
* Finally, Django "applies" the events (see the `apply_events`
- Finally, Django "applies" the events (see the `apply_events`
function) to the initial state that it fetched. E.g. for a name
change event, it finds the user data in the `realm_user` data
structure, and updates it to have the new name.
@@ -233,19 +233,21 @@ ready to write a test in `test_events.py`.
The actual code for a `test_events` test can be quite concise:
```python
def test_default_streams_events(self) -> None:
stream = get_stream("Scotland", self.user_profile.realm)
events = self.verify_action(lambda: do_add_default_stream(stream))
check_default_streams("events[0]", events[0])
# (some details omitted)
```
The real trick is debugging these tests.
The test example above has three things going on:
* Set up some data (`get_stream`)
* Call `verify_action` with an action function (`do_add_default_stream`)
* Use a schema checker to validate data (`check_default_streams`)
- Set up some data (`get_stream`)
- Call `verify_action` with an action function (`do_add_default_stream`)
- Use a schema checker to validate data (`check_default_streams`)
#### verify_action
@@ -263,15 +265,15 @@ action and then fetching a fresh copy of the state.
In particular, `verify_action` does the following:
* Call `fetch_initial_state_data` to get the current state.
* Call the action function (e.g. `do_add_default_stream`).
* Capture the events generated by the action function.
* Check the events generated are documented in the [OpenAPI
- Call `fetch_initial_state_data` to get the current state.
- Call the action function (e.g. `do_add_default_stream`).
- Capture the events generated by the action function.
- Check the events generated are documented in the [OpenAPI
schema](../documentation/api.md) defined in
`zerver/openapi/zulip.yaml`.
* Call `apply_events(state, events)`, to get the resulting "hybrid state".
* Call `fetch_initial_state_data` again to get the "normal state".
* Compare the two results.
- Call `apply_events(state, events)`, to get the resulting "hybrid state".
- Call `fetch_initial_state_data` again to get the "normal state".
- Compare the two results.
In the event that you wrote the `apply_events` logic correctly the
first time, then the two states will be identical, and the
@@ -292,20 +294,22 @@ only has one required parameter, which is the action function. We
typically express the action function as a lambda, so that we
can pass in arguments:
```python
events = self.verify_action(lambda: do_add_default_stream(stream))
```
There are some notable optional parameters for `verify_action`:
* `state_change_expected` must be set to `False` if your action
- `state_change_expected` must be set to `False` if your action
doesn't actually require state changes for some reason; otherwise,
`verify_action` will complain that your test doesn't really
exercise any `apply_events` logic. Typing notifications (which
are ephemereal) are a common place where we use this.
* `num_events` will tell `verify_action` how many events the
- `num_events` will tell `verify_action` how many events the
`hamlet` user will receive after the action (the default is 1).
* parameters such as `client_gravatar` and `slim_presence` get
- parameters such as `client_gravatar` and `slim_presence` get
passed along to `fetch_initial_state_data` (and it's important
to test both boolean values of these parameters for relevant
actions).
@@ -327,9 +331,11 @@ The second is higher-detail check inside `test_events` that this
specific test generated the expected series of events. Let's look at
the last line of our example test snippet:
```python
# ...
events = self.verify_action(lambda: do_add_default_stream(stream))
check_default_streams("events[0]", events[0])
```
We have discussed `verify_action` in some detail, and you will
note that it returns the actual events generated by the action
@@ -346,6 +352,7 @@ If you are creating a new event format, then you will have to
write your own schema checker in `event_schema.py`. Here is
the example relevant to our example:
```python
default_streams_event = event_dict_type(
required_keys=[
("type", Equals("default_streams")),
@@ -353,6 +360,7 @@ the example relevant to our example:
]
)
check_default_streams = make_checker(default_streams_event)
```
Note that `basic_stream_fields` is not shown in these docs. The
best way to understand how to write schema checkers is to read
@@ -399,4 +407,4 @@ correctly, clients are responsible for discarding events related to
messages that the client has not yet fetched.
Additionally, see
[the master documentation on sending messages](../subsystems/sending-messages.md)
[the main documentation on sending messages](../subsystems/sending-messages.md)

View File

@@ -41,33 +41,44 @@ All steps in this section should be run as the `root` user; on most installs, th
1. Alter the deployment setting:
```bash
crudini --set /etc/zulip/zulip.conf machine pgroonga enabled
```
1. Update the deployment to respect that new setting:
```bash
/home/zulip/deployments/current/scripts/zulip-puppet-apply
```
1. Edit `/etc/zulip/settings.py`, to add:
```python
USING_PGROONGA = True
```
1. Apply the PGroonga migrations:
```bash
su zulip -c '/home/zulip/deployments/current/manage.py migrate pgroonga'
```
Note that the migration may take a long time, and users will be
unable to send new messages until the migration finishes.
1. Once the migrations are complete, restart Zulip:
```bash
su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
```
### Disabling PGroonga
1. Remove the PGroonga migration:
```bash
su zulip -c '/home/zulip/deployments/current/manage.py migrate pgroonga zero'
```
If you intend to re-enable PGroonga later, you can skip this step,
at the cost of your Message table being slightly larger than it would
@@ -75,12 +86,18 @@ All steps in this section should be run as the `root` user; on most installs, th
1. Edit `/etc/zulip/settings.py`, editing the line containing `USING_PGROONGA` to read:
```python
USING_PGROONGA = False
```
1. Restart Zulip:
```bash
su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
```
1. Finally, remove the deployment setting:
```bash
crudini --del /etc/zulip/zulip.conf machine pgroonga
```

View File

@@ -7,12 +7,12 @@ be used to deep-link into the application and allow the browser's
"back" functionality to let the user navigate between parts of the UI.
Some examples are:
* `/#settings/your-bots`: Bots section of the settings overlay.
* `/#streams`: Streams overlay, where the user manages streams
- `/#settings/your-bots`: Bots section of the settings overlay.
- `/#streams`: Streams overlay, where the user manages streams
(subscription etc.)
* `/#streams/11/announce`: Streams overlay with stream ID 11 (called
- `/#streams/11/announce`: Streams overlay with stream ID 11 (called
"announce") selected.
* `/#narrow/stream/42-android/topic/fun`: Message feed showing stream
- `/#narrow/stream/42-android/topic/fun`: Message feed showing stream
"android" and topic "fun". (The `42` represents the id of the
stream.
@@ -22,27 +22,27 @@ code), which is unfortunately one of our thorniest modules. Part of
the reason that it's thorny is that it needs to support a lot of
different flows:
* The user clicking on an in-app link, which in turn opens an overlay.
- The user clicking on an in-app link, which in turn opens an overlay.
For example the streams overlay opens when the user clicks the small
cog symbol on the left sidebar, which is in fact a link to
`/#streams`. This makes it easy to have simple links around the app
without custom click handlers for each one.
* The user uses the "back" button in their browser (basically
equivalent to the previous one, as a *link* out of the browser history
- The user uses the "back" button in their browser (basically
equivalent to the previous one, as a _link_ out of the browser history
will be visited).
* The user clicking some in-app click handler (e.g. "Stream settings"
- The user clicking some in-app click handler (e.g. "Stream settings"
for an individual stream), that potentially does
several UI-manipulating things including e.g. loading the streams
overlay, and needs to update the hash without re-triggering the open
animation (etc.).
* Within an overlay like the streams overlay, the user clicks to
- Within an overlay like the streams overlay, the user clicks to
another part of the overlay, which should update the hash but not
re-trigger loading the overlay (which would result in a confusing
animation experience).
* The user is in a part of the webapp, and reloads their browser window.
- The user is in a part of the webapp, and reloads their browser window.
Ideally the reloaded browser window should return them to their
original state.
* A server-initiated browser reload (done after a new version is
- A server-initiated browser reload (done after a new version is
deployed, or when a user comes back after being idle for a while,
see [notes below][self-server-reloads]), where we try to preserve
extra state (e.g. content of compose box, scroll position within a
@@ -56,21 +56,21 @@ that it's easy to accidentally break something.
The main external API lives in `static/js/browser_history.js`:
* `browser_history.update` is used to update the browser
- `browser_history.update` is used to update the browser
history, and it should be called when the app code is taking care
of updating the UI directly
* `browser_history.go_to_location` is used when you want the `hashchange`
- `browser_history.go_to_location` is used when you want the `hashchange`
module to actually dispatch building the next page
Internally you have these functions:
* `hashchange.hashchanged` is the function used to handle the hash,
- `hashchange.hashchanged` is the function used to handle the hash,
whether it's changed by the browser (e.g. by clicking on a link to
a hash or using the back button) or triggered internally.
* `hashchange.do_hashchange_normal` handles most cases, like loading the main
- `hashchange.do_hashchange_normal` handles most cases, like loading the main
page (but maybe with a specific URL if you are narrowed to a
stream or topic or PMs, etc.).
* `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have
- `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have
some minor complexity related to remembering the page from
which the overlay was launched, as well as optimizing in-page
transitions (i.e. don't close/re-open the overlay if you can
@@ -81,7 +81,7 @@ Internally you have these functions:
There are a few circumstances when the Zulip browser window needs to
reload itself:
* If the browser has been offline for more than 10 minutes, the
- If the browser has been offline for more than 10 minutes, the
browser's [event queue][events-system] will have been
garbage-collected by the server, meaning the browser can no longer
get real-time updates altogether. In this case, the browser
@@ -89,7 +89,7 @@ reload itself:
unsuspend callback (based on some clever time logic) that ensures we
check immediately when a client unsuspends; grep for `watchdog` to
see the code.
* If a new version of the server has been deployed, we want to reload
- If a new version of the server has been deployed, we want to reload
the browser so that it will start running the latest code. However,
we don't want server deploys to be disruptive. So, the backend
preserves user-side event queues (etc.) and just pushes a special
@@ -106,10 +106,10 @@ reload itself:
Here are some key functions in the reload system:
* `reload.preserve_state` is called when a server-initiated browser
- `reload.preserve_state` is called when a server-initiated browser
reload happens, and encodes a bunch of data like the current scroll
position into the hash.
* `reload.initialize` handles restoring the preserved state after a
- `reload.initialize` handles restoring the preserved state after a
reload where the hash starts with `/#reload`.
## All reloads

View File

@@ -17,7 +17,7 @@ In `zerver/lib/hotspots.py`, add your content to the `ALL_HOTSPOTS` dictionary.
Each key-value pair in `ALL_HOTSPOTS` associates the name of the hotspot with the
content displayed to the user.
```
```python
ALL_HOTSPOTS = {
...
'new_hotspot_name': {
@@ -32,8 +32,8 @@ ALL_HOTSPOTS = {
The target element and visual orientation of each hotspot is specified in
`HOTSPOT_LOCATIONS` of `static/js/hotspots.js`.
The `icon_offset` property specifies where the pulsing icon is placed *relative to
the width and height of the target element*.
The `icon_offset` property specifies where the pulsing icon is placed _relative to
the width and height of the target element_.
By default, `popovers.compute_placement` is used to responsively
determine whether a popover is best displayed above (TOP), below (BOTTOM),
@@ -53,6 +53,7 @@ in multiple copies of hotspots appearing; you can clear that by
reloading the browser.
Here are some visual characteristics to confirm:
- popover content is readable
- icons reposition themselves on resize
- icons are hidden and shown along with their associated elements
@@ -67,8 +68,9 @@ a target element on a sidebar or overlay, the icon's z-index may need to
be increased to 101, 102, or 103.
This adjustment can be made at the bottom of `static/styles/hotspots.css`:
```
\#hotspot_new_hotspot_name_icon {
```css
#hotspot_new_hotspot_name_icon {
z-index: 103;
}
```

View File

@@ -62,12 +62,12 @@ browsers to make sure things look the same.
### Behavior
* Templates are automatically recompiled in development when the file
- Templates are automatically recompiled in development when the file
is saved; a refresh of the page should be enough to display the latest
version. You might need to do a hard refresh, as some browsers cache
webpages.
* Variables can be used in templates. The variables available to the
- Variables can be used in templates. The variables available to the
template are called the **context**. Passing the context to the HTML
template sets the values of those variables to the value they were
given in the context. The sections below contain specifics on how the
@@ -84,24 +84,24 @@ found [here][jconditionals].
The context for Jinja2 templates is assembled from a few places:
* `zulip_default_context` in `zerver/context_processors.py`. This is
- `zulip_default_context` in `zerver/context_processors.py`. This is
the default context available to all Jinja2 templates.
* As an argument in the `render` call in the relevant function that
- As an argument in the `render` call in the relevant function that
renders the template. For example, if you want to find the context
passed to `index.html`, you can do:
```
```console
$ git grep zerver/app/index.html '*.py'
zerver/views/home.py: response = render(request, 'zerver/app/index.html',
```
The next line in the code being the context definition.
* `zproject/urls.py` for some fairly static pages that are rendered
- `zproject/urls.py` for some fairly static pages that are rendered
using `TemplateView`, for example:
```
```python
path('config-error/google', TemplateView.as_view(
template_name='zerver/config_error.html',),
{'google_error': True},),
@@ -203,14 +203,14 @@ For your asset to be included in a development/production bundle, it
needs to be accessible from one of the entry points defined either in
`tools/webpack.assets.json` or `tools/webpack.dev-assets.json`.
* If you plan to only use the file within the app proper, and not on the login
- If you plan to only use the file within the app proper, and not on the login
page or other standalone pages, put it in the `app` bundle by importing it
in `static/js/bundles/app.js`.
* If it needs to be available both in the app and all
- If it needs to be available both in the app and all
logged-out/portico pages, import it to
`static/js/bundles/common.js` which itself is imported to the
`app` and `common` bundles.
* If it's just used on a single standalone page which is only used in
- If it's just used on a single standalone page which is only used in
a development environment (e.g. `/devlogin`) create a new entry
point in `tools/webpack.dev-assets.json` or it's used in both
production and development (e.g. `/stats`) create a new entry point
@@ -224,10 +224,11 @@ If you want to test minified files in development, look for the
### How it works in production
A few useful notes are:
* Zulip installs static assets in production in
- Zulip installs static assets in production in
`/home/zulip/prod-static`. When a new version is deployed, before the
server is restarted, files are copied into that directory.
* We use the VFL (versioned file layout) strategy, where each file in
- We use the VFL (versioned file layout) strategy, where each file in
the codebase (e.g. `favicon.ico`) gets a new name
(e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each
deployment, has a manifest file
@@ -239,7 +240,7 @@ server is restarted, files are copied into that directory.
deployment can't find their static assets. It also is necessary for
any incremental rollout strategy where different clients get
different versions of the site.
* Some paths for files (e.g. emoji) are stored in the
- Some paths for files (e.g. emoji) are stored in the
`rendered_content` of past messages, and thus cannot be removed
without breaking the rendering of old messages (or doing a
mass-rerender of old messages).
@@ -269,11 +270,11 @@ the browser console, but for debugging convenience, we have a custom
webpack plugin (`tools/debug-require-webpack-plugin.ts`) that exposes
a version of the `require()` function to the development environment
browser console for this purpose. For example, you can access our
`people` module by evaluating `people =
require("./static/js/people")`, or the third-party `lodash` module
with `_ = require("lodash")`. This mechanism is **not** a stable API
and should not be used for any purpose other than interactive
debugging.
`people` module by evaluating
`people = require("./static/js/people")`, or the third-party `lodash`
module with `_ = require("lodash")`. This mechanism is **not** a
stable API and should not be used for any purpose other than
interactive debugging.
We have one module, `zulip_test`, thats exposed as a global variable
using `expose-loader` for direct use in Puppeteer tests and in the
@@ -281,8 +282,8 @@ production browser console. If you need to access a variable or
function in those scenarios, add it to `zulip_test`. This is also
**not** a stable API.
[Jinja2]: http://jinja.pocoo.org/
[Handlebars]: https://handlebarsjs.com/
[jinja2]: http://jinja.pocoo.org/
[handlebars]: https://handlebarsjs.com/
[trans]: http://jinja.pocoo.org/docs/dev/templates/#i18n
[jconditionals]: http://jinja.pocoo.org/docs/2.9/templates/#list-of-control-structures
[hconditionals]: https://handlebarsjs.com/guide/#block_helpers.html

View File

@@ -66,7 +66,6 @@ export function filter_taken_users(items, pill_widget) {
You can get notifications from the pill code that pills have been
created/remove.
```js
pills.onPillCreate(function () {
update_save_state();

View File

@@ -20,19 +20,19 @@ is great for small installations.
The [Django][django-errors] framework provides much of the
infrastructure needed by our error reporting system:
* The ability to send emails to the server's administrators with any
- The ability to send emails to the server's administrators with any
500 errors, using the `mail_admins` function. We enhance these data
with extra details (like what user was involved in the error) in
`zerver/logging_handlers.py`, and then send them to the
administrator in `zerver/lib/error_notify.py` (which also supports
sending Zulips to a stream about production errors).
* The ability to rate-limit certain errors to avoid sending hundreds
- The ability to rate-limit certain errors to avoid sending hundreds
of emails in an outage (see `_RateLimitFilter` in
`zerver/lib/logging_util.py`)
* A nice framework for filtering passwords and other important user
- A nice framework for filtering passwords and other important user
data from the exception details, which we use in
`zerver/filters.py`.
* Middleware for handling `JsonableError`, our system for allowing
- Middleware for handling `JsonableError`, our system for allowing
code anywhere in Django to report an API-facing `json_error` from
anywhere in a view code path.
@@ -72,7 +72,7 @@ In production, one usually wants to look at `errors.log` for errors
since the main server log can be very verbose, but the main server log
can be extremely valuable for investigating performance problems.
```
```text
2016-05-20 14:50:22.056 INFO [zr] 127.0.0.1 GET 302 528ms (db: 1ms/1q) (+start: 123ms) / (unauth@zulip via ?)
[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0
2016-05-20 14:50:22.272 INFO [zr] 127.0.0.1 GET 200 124ms (db: 3ms/2q) /login/ (unauth@zulip via ?)
@@ -84,18 +84,19 @@ can be extremely valuable for investigating performance problems.
```
The format of this output is:
* Timestamp
* Log level
* Logger name, abbreviated as "zr" for these Zulip request logs
* IP address
* HTTP method
* HTTP status code
* Time to process
* (Optional perf data details, e.g. database time/queries, memcached
- Timestamp
- Log level
- Logger name, abbreviated as "zr" for these Zulip request logs
- IP address
- HTTP method
- HTTP status code
- Time to process
- (Optional perf data details, e.g. database time/queries, memcached
time/queries, Django process startup time, Markdown processing time,
etc.)
* Endpoint/URL from zproject/urls.py
* "email via client" showing user account involved (if logged in) and
- Endpoint/URL from zproject/urls.py
- "email via client" showing user account involved (if logged in) and
the type of client they used ("web", "Android", etc.).
The performance data details are particularly useful for investigating
@@ -124,33 +125,34 @@ might use). In development, this means displaying a highly visible
overlay over the message view area, to make exceptions in testing a
new feature hard to miss.
* Blueslip is implemented in `static/js/blueslip.js`.
* In order to capture essentially any error occurring in the browser,
- Blueslip is implemented in `static/js/blueslip.js`.
- In order to capture essentially any error occurring in the browser,
Blueslip listens for the `error` event on `window`, and has methods
for being manually triggered by Zulip JavaScript code for warnings
and assertion failures.
* Blueslip keeps a log of all the notices it has received during a
- Blueslip keeps a log of all the notices it has received during a
browser session, and includes them in reports to the server, so that
one can see cases where exceptions chained together. You can print
this log from the browser console using `blueslip =
require("./static/js/blueslip"); blueslip.get_log()`.
this log from the browser console using
`blueslip = require("./static/js/blueslip"); blueslip.get_log()`.
Blueslip supports several error levels:
* `throw new Error(…)`: For fatal errors that cannot be easily
- `throw new Error(…)`: For fatal errors that cannot be easily
recovered from. We try to avoid using it, since it kills the
current JS thread, rather than returning execution to the caller.
* `blueslip.error`: For logging of events that are definitely caused
- `blueslip.error`: For logging of events that are definitely caused
by a bug and thus sufficiently important to be reported, but where
we can handle the error without creating major user-facing problems
(e.g. an exception when handling a presence update).
* `blueslip.warn`: For logging of events that are a problem but not
- `blueslip.warn`: For logging of events that are a problem but not
important enough to send an email about in production. They are,
however, highlighted in the JS console in development.
* `blueslip.log` (and `blueslip.info`): Logged to the JS console in
- `blueslip.log` (and `blueslip.info`): Logged to the JS console in
development and also in the blueslip log in production. Useful for
data that might help discern what state the browser was in during an
error (e.g. whether the user was in a narrow).
* `blueslip.debug`: Similar to `blueslip.log`, but are not printed to
- `blueslip.debug`: Similar to `blueslip.log`, but are not printed to
the JS console in development.
## Frontend performance reporting
@@ -159,12 +161,12 @@ In order to make it easier to debug potential performance problems in
the critically latency-sensitive message sending code pathway, we log
and report to the server the following whenever a message is sent:
* The time the user triggered the message (aka the start time).
* The time the `send_message` response returned from the server.
* The time the message was received by the browser from the
- The time the user triggered the message (aka the start time).
- The time the `send_message` response returned from the server.
- The time the message was received by the browser from the
`get_events` protocol (these last two race with each other).
* Whether the message was locally echoed.
* If so, whether there was a disparity between the echoed content and
- Whether the message was locally echoed.
- If so, whether there was a disparity between the echoed content and
the server-rendered content, which can be used for statistics on how
effective our [local echo system](../subsystems/markdown.md) is.
@@ -173,9 +175,9 @@ The code is all in `zerver/lib/report.py` and `static/js/sent_messages.js`.
We have similar reporting for the time it takes to narrow / switch to
a new view:
* The time the action was initiated
* The time when the updated message feed was visible to the user
* The time when the browser was idle again after switching views
- The time the action was initiated
- The time when the updated message feed was visible to the user
- The time when the browser was idle again after switching views
(intended to catch issues where we generate a lot of deferred work).
[django-errors]: https://docs.djangoproject.com/en/2.2/howto/error-reporting/

View File

@@ -13,20 +13,20 @@ While Zulip takes advantage of built-in Django management commands for
things like managing Django migrations, we also have dozens that we've
written for a range of purposes:
* Cron jobs to do regular updates, e.g. `update_analytics_counts.py`,
- Cron jobs to do regular updates, e.g. `update_analytics_counts.py`,
`sync_ldap_user_data`, etc.
* Useful parts of provisioning or upgrading a Zulip development
- Useful parts of provisioning or upgrading a Zulip development
environment or server, e.g. `makemessages`, `compilemessages`,
`populate_db`, `fill_memcached_caches`, etc.
* The actual scripts run by supervisord to run the persistent
- The actual scripts run by supervisord to run the persistent
processes in a Zulip server, e.g. `runtornado` and `process_queue`.
* For a sysadmin to verify a Zulip server's configuration during
- For a sysadmin to verify a Zulip server's configuration during
installation, e.g. `checkconfig`, `send_test_email`.
* As the interface for doing those rare operations that don't have a
- As the interface for doing those rare operations that don't have a
UI yet, e.g. `deactivate_realm`, `reactivate_realm`,
`change_user_email` (for the case where the user doesn't control the
old email address).
* For a sysadmin to easily interact with and script common possible
- For a sysadmin to easily interact with and script common possible
changes they might want to make to the database on a Zulip server.
E.g. `send_password_reset_email`, `export`, `purge_queue`.
@@ -38,14 +38,14 @@ command to write a new one. Some good examples are
is good, but we have a few pieces advice specific to the Zulip
project.
* If you need to access a realm or user, use the `ZulipBaseCommand`
- If you need to access a realm or user, use the `ZulipBaseCommand`
class in `zerver/lib/management.py` so you don't need to write the
tedious code of looking those objects up. This is especially
important for users, since the library handles the issues around
looking up users by email well (if there's a unique user with that
email, just modify it without requiring the user to specify the
realm as well, but if there's a collision, throw a nice error).
* Avoid writing a lot of code in management commands; management
- Avoid writing a lot of code in management commands; management
commands are annoying to unit test, and thus easier to maintain if
all the interesting logic is in a nice function that is unit tested
(and ideally, also used in Zulip's existing code). Look for code in

View File

@@ -49,18 +49,18 @@ by both test suites; as a result, it is the preferred place to add new
tests for Zulip's Markdown system. Some important notes on reading
this file:
* `expected_output` is the expected output for the backend Markdown
- `expected_output` is the expected output for the backend Markdown
processor.
* When the frontend processor doesn't support a feature and it should
- When the frontend processor doesn't support a feature and it should
just be rendered on the backend, we set `backend_only_rendering` to
`true` in the fixtures; this will automatically verify that
`markdown.contains_backend_only_syntax` rejects the syntax, ensuring
it will be rendered only by the backend processor.
* When the two processors disagree, we set `marked_expected_output` in
- When the two processors disagree, we set `marked_expected_output` in
the fixtures; this will ensure that the syntax stays that way. If
the differences are important (i.e. not just whitespace), we should
also open an issue on GitHub to track the problem.
* For mobile push notifications, we need a text version of the
- For mobile push notifications, we need a text version of the
rendered content, since the APNS and GCM push notification systems
don't support richer markup. Mostly, this involves stripping HTML,
but there's some syntax we take special care with. Tests for what
@@ -91,44 +91,44 @@ tests with `tools/test-js-with-node markdown` and backend tests with
First, you will likely find these third-party resources helpful:
* **[Python-Markdown](https://pypi.python.org/pypi/Markdown)** is the Markdown
- **[Python-Markdown](https://pypi.python.org/pypi/Markdown)** is the Markdown
library used by Zulip as a base to build our custom Markdown syntax upon.
* **[Python's XML ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)**
- **[Python's XML ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)**
is the part of the Python standard library used by Python Markdown
and any custom extensions to generate and modify the output HTML.
When changing Zulip's Markdown syntax, you need to update several
places:
* The backend Markdown processor (`zerver/lib/markdown/__init__.py`).
* The frontend Markdown processor (`static/js/markdown.js` and sometimes
- The backend Markdown processor (`zerver/lib/markdown/__init__.py`).
- The frontend Markdown processor (`static/js/markdown.js` and sometimes
`static/third/marked/lib/marked.js`), or `markdown.contains_backend_only_syntax` if
your changes won't be supported in the frontend processor.
* If desired, the typeahead logic in `static/js/composebox_typeahead.js`.
* The test suite, probably via adding entries to `zerver/tests/fixtures/markdown_test_cases.json`.
* The in-app Markdown documentation (`markdown_help_rows` in `static/js/info_overlay.js`).
* The list of changes to Markdown at the end of this document.
- If desired, the typeahead logic in `static/js/composebox_typeahead.js`.
- The test suite, probably via adding entries to `zerver/tests/fixtures/markdown_test_cases.json`.
- The in-app Markdown documentation (`markdown_help_rows` in `static/js/info_overlay.js`).
- The list of changes to Markdown at the end of this document.
Important considerations for any changes are:
* Security: A bug in the Markdown processor can lead to XSS issues.
- Security: A bug in the Markdown processor can lead to XSS issues.
For example, we should not insert unsanitized HTML from a
third-party web application into a Zulip message.
* Uniqueness: We want to avoid users having a bad experience due to
- Uniqueness: We want to avoid users having a bad experience due to
accidentally triggering Markdown syntax or typeahead that isn't
related to what they are trying to express.
* Performance: Zulip can render a lot of messages very quickly, and
- Performance: Zulip can render a lot of messages very quickly, and
we'd like to keep it that way. New regular expressions similar to
the ones already present are unlikely to be a problem, but we need
to be thoughtful about expensive computations or third-party API
requests.
* Database: The backend Markdown processor runs inside a Python thread
- Database: The backend Markdown processor runs inside a Python thread
(as part of how we implement timeouts for third-party API queries),
and for that reason we currently should avoid making database
queries inside the Markdown processor. This is a technical
implementation detail that could be changed with a few days of work,
but is an important detail to know about until we do that work.
* Testing: Every new feature should have both positive and negative
- Testing: Every new feature should have both positive and negative
tests; they're easy to write and give us the flexibility to refactor
frequently.
@@ -177,14 +177,14 @@ chat product; even though you can edit messages to fix formatting
mistakes, you don't want to be doing that often. There are basically
2 types of error rates that are important for a product like Zulip:
* What fraction of the time, if you pasted a short technical email
- What fraction of the time, if you pasted a short technical email
that you wrote to your team and passed it through your Markdown
implementation, would you need to change the text of your email for it
to render in a reasonable way? This is the "accidental Markdown
syntax" problem, common with Markdown syntax like the italics syntax
interacting with talking about `char *`s.
* What fraction of the time do users attempting to use a particular
- What fraction of the time do users attempting to use a particular
Markdown syntax actually succeed at doing so correctly? Syntax like
required a blank line between text and the start of a bulleted list
raise this figure substantially.
@@ -207,71 +207,70 @@ accurate.
### Basic syntax
* Enable `nl2br` extension: this means one newline creates a line
- Enable `nl2br` extension: this means one newline creates a line
break (not paragraph break).
* Allow only `*` syntax for italics, not `_`. This resolves an issue where
- Allow only `*` syntax for italics, not `_`. This resolves an issue where
people were using `_` and hitting it by mistake too often. Asterisks
surrounded by spaces won't trigger italics, either (e.g. with stock Markdown
`You should use char * instead of void * there` would produce undesired
results).
* Allow only `**` syntax for bold, not `__` (easy to hit by mistake if
- Allow only `**` syntax for bold, not `__` (easy to hit by mistake if
discussing Python `__init__` or something).
* Add `~~` syntax for strikethrough.
- Add `~~` syntax for strikethrough.
* Disable special use of `\` to escape other syntax. Rendering `\\` as
- Disable special use of `\` to escape other syntax. Rendering `\\` as
`\` was hugely controversial, but having no escape syntax is also
controversial. We may revisit this. For now you can always put
things in code blocks.
### Lists
* Allow tacking a bulleted list or block quote onto the end of a
- Allow tacking a bulleted list or block quote onto the end of a
paragraph, i.e. without a blank line before it.
* Allow only `*` for bulleted lists, not `+` or `-` (previously
- Allow only `*` for bulleted lists, not `+` or `-` (previously
created confusion with diff-style text sloppily not included in a
code block).
* Disable ordered list syntax: stock Markdown automatically renumbers, which
- Disable ordered list syntax: stock Markdown automatically renumbers, which
can be really confusing when sending a numbered list across multiple
messages.
### Links
* Enable auto-linkification, both for `http://...` and guessing at
- Enable auto-linkification, both for `http://...` and guessing at
things like `t.co/foo`.
* Force links to be absolute. `[foo](google.com)` will go to
- Force links to be absolute. `[foo](google.com)` will go to
`http://google.com`, and not `https://zulip.com/google.com` which
is the default behavior.
* Set `title=`(the URL) on every link tag.
- Set `title=`(the URL) on every link tag.
* Disable link-by-reference syntax,
- Disable link-by-reference syntax,
`[foo][bar]` ... `[bar]: https://google.com`.
* Enable linking to other streams using `#**streamName**`.
- Enable linking to other streams using `#**streamName**`.
### Code
* Enable fenced code block extension, with syntax highlighting.
- Enable fenced code block extension, with syntax highlighting.
* Disable line-numbering within fenced code blocks -- the `<table>`
- Disable line-numbering within fenced code blocks -- the `<table>`
output confused our web client code.
### Other
* Disable headings, both `# foo` and `== foo ==` syntax: they don't
- Disable headings, both `# foo` and `== foo ==` syntax: they don't
make much sense for chat messages.
* Disabled images with `![]()` (images from links are shown as an inline
- Disabled images with `![]()` (images from links are shown as an inline
preview).
* Allow embedding any avatar as a tiny (list bullet size) image. This
- Allow embedding any avatar as a tiny (list bullet size) image. This
is used primarily by version control integrations.
* We added the `~~~ quote` block quote syntax.
- We added the `~~~ quote` block quote syntax.

View File

@@ -11,10 +11,10 @@ the details of the email/mobile push notifications code path.
Here we name a few corner cases worth understanding in designing this
sort of notifications system:
* The **idle desktop problem**: We don't want the presence of a
- The **idle desktop problem**: We don't want the presence of a
desktop computer at the office to eat all notifications because the
user has an "online" client that they may not have used in 3 days.
* The **hard disconnect problem**: A client can lose its connection to
- The **hard disconnect problem**: A client can lose its connection to
the Internet (or be suspended, or whatever) at any time, and this
happens routinely. We want to ensure that races where a user closes
their laptop shortly after a notifiable message is sent does not
@@ -25,16 +25,17 @@ sort of notifications system:
As a reminder, the relevant part of the flow for sending messages is
as follows:
* `do_send_messages` is the synchronous message-sending code path,
- `do_send_messages` is the synchronous message-sending code path,
and passing the following data in its `send_event` call:
* Data about the message's content (E.g. mentions, wildcard
- Data about the message's content (E.g. mentions, wildcard
mentions, and alert words) and encodes it into the `UserMessage`
table's `flags` structure, which is in turn passed into
`send_event` for each user receiving the message.
* Data about user configuration relevant to the message, such as
- Data about user configuration relevant to the message, such as
`push_notify_user_ids` and `stream_notify_user_ids`, are included
alongside `flags` in the per-user data structure.
* The `presence_idle_user_ids` set, containing the subset of
- The `presence_idle_user_ids` set, containing the subset of
recipient users who are mentioned, are PM recipients, have alert
words, or otherwise would normally get a notification, but have not
interacted with a Zulip client in the last few minutes. (Users who
@@ -43,7 +44,7 @@ as follows:
structure ignores users for whom the message is not notifiable,
which is important to avoid this being thousands of `user_ids` for
messages to large streams with few currently active users.
* The Tornado [event queue system](../subsystems/events-system.md)
- The Tornado [event queue system](../subsystems/events-system.md)
processes that data, as well as data about each user's active event
queues, to (1) push an event to each queue needing that message and
(2) for notifiable messages, pushing an event onto the
@@ -51,65 +52,65 @@ as follows:
queues. This important message-processing logic has notable extra
logic not present when processing normal events, both for details
like splicing `flags` to customize event payloads per-user, as well.
* The Tornado system determines whether the user is "offline/idle".
- The Tornado system determines whether the user is "offline/idle".
Zulip's email notifications are designed to not fire when the user
is actively using Zulip to avoid spam, and this is where those
checks are implemented.
* Users in `presence_idle_user_ids` are always considered idle:
- Users in `presence_idle_user_ids` are always considered idle:
the variable name means "users who are idle because of
presence". This is how we solve the idle desktop problem; users
with an idle desktop are treated the same as users who aren't
logged in for this check.
* However, that check does not handle the hard disconnect problem:
- However, that check does not handle the hard disconnect problem:
if a user was present 1 minute before a message was sent, and then
closed their laptop, the user will not be in
`presence_idle_user_ids`, and so without an additional mechanism,
messages sent shortly after a user leaves would never trigger a
notification (!).
* We solve that problem by also notifying if
- We solve that problem by also notifying if
`receiver_is_off_zulip` returns `True`, which checks whether the user has any
current events system clients registered to receive `message`
events. This check is done immediately (handling soft disconnects,
where E.g. the user closes their last Zulip tab and we get the
`DELETE /events/{queue_id}` request).
* The `receiver_is_off_zulip` check is effectively repeated when
- The `receiver_is_off_zulip` check is effectively repeated when
event queues are garbage-collected (in `missedmessage_hook`) by
looking for whether the queue being garbage-collectee was the only
one; this second check solves the hard disconnect problem, resulting in
notifications for these hard-disconnect cases usually coming 10
minutes late.
* The message-edit code path has parallel logic in
- The message-edit code path has parallel logic in
`maybe_enqueue_notifications_for_message_update` for triggering
notifications in cases like a mention added during message
editing.
* The business logic for all these notification decisions made
- The business logic for all these notification decisions made
inside Tornado has extensive automated test suites; e.g.
`test_message_edit_notifications.py` covers all the cases around
editing a message to add/remove a mention.
* We may in the future want to add some sort of system for letting
- We may in the future want to add some sort of system for letting
users see past notifications, to help with explaining and
debugging this system, since it has so much complexity.
* Desktop notifications are the simplest; they are implemented
- Desktop notifications are the simplest; they are implemented
client-side by the web/desktop app's logic
(`static/js/notifications.js`) inspecting the `flags` fields that
were spliced into `message` events by the Tornado system, as well as
the user's notification settings.
* The queue processors for those queues make the final determination
- The queue processors for those queues make the final determination
for whether to send a notification, and do the work to generate an
email (`zerver/lib/email_notifications.py`) or mobile
(`zerver/lib/push_notifications.py`) notification. We'll detail
this process in more detail for each system below, but it's
important to know that it's normal for a message to sit in these
queues for minutes (and in the future, possibly hours).
* Both queue processor code paths do additional filtering before
- Both queue processor code paths do additional filtering before
sending a notification:
* Messages that have already been marked as read by the user before
- Messages that have already been marked as read by the user before
the queue processor runs never trigger a notification.
* Messages that were already deleted never trigger a notification.
* The user-level settings for whether email/mobile notifications are
- Messages that were already deleted never trigger a notification.
- The user-level settings for whether email/mobile notifications are
disabled are rechecked, as the user may have disabled one of these
settings during the queuing period.
* The **Email notifications queue processor**, `MissedMessageWorker`,
- The **Email notifications queue processor**, `MissedMessageWorker`,
takes care to wait for 2 minutes (hopefully in the future this will be a
configuration setting) and starts a thread to batch together multiple
messages into a single email. These features are unnecessary
@@ -119,7 +120,7 @@ as follows:
to GitHub's email notifications, with a clean, simple design that
makes replying from an email client possible (using the [incoming
email integration](../production/email-gateway.md)).
* The **Push notifications queue processor**,
- The **Push notifications queue processor**,
`PushNotificationsWorker`, is a simple wrapper around the
`push_notifications.py` code that actually sends the
notification. This logic is somewhat complicated by having to track
@@ -131,27 +132,27 @@ as follows:
The following important constraints are worth understanding about the
structure of the system, when thinking about changes to it:
* **Bulk database queries** are much more efficient for checking
- **Bulk database queries** are much more efficient for checking
details from the database like "which users receiving this message
are online".
* **Thousands of users**. Zulip supports thousands of users, and we
- **Thousands of users**. Zulip supports thousands of users, and we
want to avoid `send_event()` pushing large amounts of per-user data
to Tornado via RabbitMQ for scalability reasons.
* **Tornado doesn't do database queries**. Because the Tornado system
- **Tornado doesn't do database queries**. Because the Tornado system
is an asynchronous event-driven framework, and our Django database
library is synchronous, database queries are very expensive. So
these queries need to be done in either `do_send_messages` or the
queue processor logic. (For example, this means `presence` data
should be checked in either `do_send_messages` or the queue
processors, not in Tornado).
* **Future configuration**. Notification settings are an area that we
- **Future configuration**. Notification settings are an area that we
expect to only expand with time, with upcoming features like
following a topic (to get notifications for messages only within
that topic in a stream). There are a lot of different workflows
possible with Zulip's threading, and it's important to make it easy
for users to set up Zulip's notification to fit as many of those
workflows as possible.
* **Message editing**. Zulip supports editing messages, and that
- **Message editing**. Zulip supports editing messages, and that
interacts with notifications in ways that require careful handling:
Notifications should have
the latest edited content (users often fix typos 30 seconds after

View File

@@ -7,13 +7,13 @@ workload of usage without performance materially degrading.
First, a few notes on philosophy.
* We consider it an important technical goal for Zulip to be fast,
- We consider it an important technical goal for Zulip to be fast,
because that's an important part of user experience for a real-time
collaboration tool like Zulip. Many UI features in the Zulip webapp
are designed to load instantly, because all the data required for
them is present in the initial HTTP response, and both the Zulip
API and webapp are architected around that strategy.
* The Zulip database model and server implementation are carefully
- The Zulip database model and server implementation are carefully
designed to ensure that every common operation is efficient, with
automated tests designed to prevent the accidental introductions of
inefficient or excessive database queries. We much prefer doing
@@ -29,7 +29,8 @@ important to understand the load profiles for production uses.
Zulip servers typically involve a mixture of two very different types
of load profiles:
* Open communities like open source projects, online classes,
- Open communities like open source projects, online classes,
etc. have large numbers of users, many of whom are idle. (Many of
the others likely stopped by to ask a question, got it answered, and
then didn't need the community again for the next year). Our own
@@ -40,7 +41,7 @@ of load profiles:
deactivation](../subsystems/sending-messages.html#soft-deactivation)
to ensure idle users have minimal impact on both server-side
scalability and request latency.
* Fulltime teams, like your typical corporate Zulip installation,
- Fulltime teams, like your typical corporate Zulip installation,
have users who are mostly active for multiple hours a day and sending a
high volume of messages each. This load profile is most important
for self-hosted servers, since many of those are used exclusively by
@@ -83,20 +84,16 @@ substantial oscillation within a 24 hour period), we expect the rough
sense of them (as well as the list of important endpoints) is not
likely to vary dramatically over time.
``` eval_rst
======================= ============ ============== ===============
Endpoint Average time Request volume Average impact
======================= ============ ============== ===============
POST /users/me/presence 25ms 36% 9000
GET /messages 70ms 3% 2100
GET / 300ms 0.3% 900
GET /events 2ms 44% 880
GET /user_uploads/* 12ms 5% 600
POST /messages/flags 25ms 1.5% 375
POST /messages 40ms 0.5% 200
POST /users/me/* 50ms 0.04% 20
======================= ============ ============== ===============
```
| Endpoint | Average time | Request volume | Average impact |
| ----------------------- | ------------ | -------------- | -------------- |
| POST /users/me/presence | 25ms | 36% | 9000 |
| GET /messages | 70ms | 3% | 2100 |
| GET / | 300ms | 0.3% | 900 |
| GET /events | 2ms | 44% | 880 |
| GET /user_uploads/\* | 12ms | 5% | 600 |
| POST /messages/flags | 25ms | 1.5% | 375 |
| POST /messages | 40ms | 0.5% | 200 |
| POST /users/me/\* | 50ms | 0.04% | 20 |
The "Average impact" above is computed by multiplying request volume
by average time; this tells you roughly that endpoint's **relative**
@@ -110,16 +107,16 @@ memcached to do work.
As one can see, there are two categories of endpoints that are
important for scalability: those with extremely high request volumes,
and those with moderately high request volumes that are also
expensive. It doesn't matter how expensive, for example, `POST
/users/me/subscriptions` is for scalability, because the volume is
negligible.
expensive. It doesn't matter how expensive, for example,
`POST /users/me/subscriptions` is for scalability, because the volume
is negligible.
### Tornado
Zulip's Tornado-based [real-time push
system](../subsystems/events-system.md), and in particular `GET
/events`, accounts for something like 50% of all HTTP requests to a
production Zulip server. Despite `GET /events` being extremely
system](../subsystems/events-system.md), and in particular
`GET /events`, accounts for something like 50% of all HTTP requests to
a production Zulip server. Despite `GET /events` being extremely
high-volume, the typical request takes 1-3ms to process, and doesn't
use the database at all (though it will access `memcached` and
`redis`), so they aren't a huge contributor to the overall CPU usage
@@ -191,13 +188,13 @@ who have a lot of latency to the server.
There are only a few exceptions where we fetch data in a separate AJAX
request after page load:
* Message history is managed separately; this is why the Zulip webapp will
- Message history is managed separately; this is why the Zulip webapp will
first render the entire site except for the middle panel, and then a
moment later render the middle panel (showing the message history).
* A few very rarely accessed data sets like [message edit
- A few very rarely accessed data sets like [message edit
history](https://zulip.com/help/view-a-messages-edit-history) are
only fetched on demand.
* A few data sets that are only required for administrative settings
- A few data sets that are only required for administrative settings
pages are fetched only when loading those parts of the UI.
Requests to `GET /` and `/api/v1/register` that fetch `page_params`
@@ -232,21 +229,21 @@ of active optimization work.
### Fetching message history
Bulk requests for message content and metadata ([`GET
/messages`](https://zulip.com/api/get-messages)) account for ~3% of
total HTTP requests. The zulip webapp has a few major reasons it does
a large number of these requests:
Bulk requests for message content and metadata
([`GET /messages`](https://zulip.com/api/get-messages)) account for
~3% of total HTTP requests. The zulip webapp has a few major reasons
it does a large number of these requests:
* Most of these requests are from users clicking into different views
- Most of these requests are from users clicking into different views
-- to avoid certain subtle bugs, Zulip's webapp currently fetches
content from the server even when it has the history for the
relevant stream/topic cached locally.
* When a browser opens the Zulip webapp, it will eventually fetch and
- When a browser opens the Zulip webapp, it will eventually fetch and
cache in the browser all messages newer than the oldest unread
message in a non-muted context. This can be in total extremely
expensive for users with 10,000s of unread messages, resulting in a
single browser doing 100 of these requests.
* When a new version of the Zulip server is deployed, every browser
- When a new version of the Zulip server is deployed, every browser
will reload within 30 minutes to ensure they are running the latest
code. For installations that deploy often like chat.zulip.org and
zulip.com, this can result in a thundering herd effect for both `/`
@@ -269,11 +266,11 @@ per-request basis. However, we have technical designs for optimizing
the overall frequency with which clients need to make these requests
in two major ways:
* Improving [client-side
- Improving [client-side
caching](https://github.com/zulip/zulip/issues/15131) to allow
caching of narrows that the user has viewed in the current session,
avoiding repeat fetches of message content during a given session.
* Adjusting the behavior for clients with 10,000s of unread messages
- Adjusting the behavior for clients with 10,000s of unread messages
to not fetch as much old message history into the cache. See [this
issue](https://github.com/zulip/zulip/issues/16697) for relevant
design work.

View File

@@ -17,10 +17,10 @@ make to the model.
First a bit of terminology:
* "Narrowing" is the process of filtering to a particular subset of
- "Narrowing" is the process of filtering to a particular subset of
the messages the user has access to.
* The blue cursor box (the "pointer") is around is called the
- The blue cursor box (the "pointer") is around is called the
"selected" message. Zulip ensures that the currently selected
message is always in-view.
@@ -28,7 +28,7 @@ First a bit of terminology:
### Recipient bar: message you clicked
If you enter a narrow by clicking on a message group's *recipient bar*
If you enter a narrow by clicking on a message group's _recipient bar_
(stream/topic or private message recipient list at the top of a group
of messages), Zulip will select the message you clicked on. This
provides a nice user experience where you get to see the stuff near
@@ -78,11 +78,11 @@ How does Zulip decide whether a message has been read by the user?
The algorithm needs to correctly handle a range of ways people might
use the product. The algorithm is as follows:
* Any message which is selected or above a message which is selected
- Any message which is selected or above a message which is selected
is marked as read. So messages are marked as read as you scroll
down the keyboard when the pointer passes over them.
* If the whitespace at the very bottom of the feed is in view, all
- If the whitespace at the very bottom of the feed is in view, all
messages in view are marked as read.
These two simple rules, combined with the pointer logic above, end up
@@ -95,10 +95,10 @@ thread; search views will never mark messages as read.
## Testing and development
In a Zulip development environment, you can use `manage.py
mark_all_messages_unread` to set every user's pointer to 0 and all
messages as unread, for convenience in testing unread count related
logic.
In a Zulip development environment, you can use
`manage.py mark_all_messages_unread` to set every user's pointer to 0
and all messages as unread, for convenience in testing unread count
related logic.
It can be useful to combine this with `manage.py populate_db -n 3000`
(which rebuilds the database with 3000 initial messages) to ensure a

View File

@@ -22,9 +22,9 @@ A client should report to the server every minute a `POST` request to
requests contains a few parameters. The most important is "status",
which had 2 valid values:
* "active" -- this means the user has interacted with the client
- "active" -- this means the user has interacted with the client
recently. We use this for the "green" state in the webapp.
* "idle" -- the user has not interacted with the client recently.
- "idle" -- the user has not interacted with the client recently.
This is important for the case where a user left a Zulip tab open on
their desktop at work and went home for the weekend. We use this
for the "orange" state in the webapp.
@@ -34,7 +34,7 @@ for each user, contains their status and timestamp that we last heard
from that client. There are a few important details to understand
about that data structure:
* It's really important that the timestamp is the last time we heard
- It's really important that the timestamp is the last time we heard
from the client. A client can only interpret the status to display
about another user by doing a simple computation using the (status,
timestamp) pair. E.g. a user who last used Zulip 1 week ago will
@@ -45,13 +45,13 @@ about that data structure:
accurately compute whether that user is offline (even if the last
data from the server was 45 seconds ago, and the user was last
online 4:30 before the client received that server data).
* Users can disable their own presence updates in user settings
- Users can disable their own presence updates in user settings
(`UserProfile.presence_enabled` is the flag storing [this user
preference](https://zulip.com/help/status-and-availability#disable-updating-availability)).
* The `status_from_timestamp` function in `static/js/presence.js` is
- The `status_from_timestamp` function in `static/js/presence.js` is
useful sample code; the `OFFLINE_THRESHOLD_SECS` check is critical
to correct output.
* We provide the data for e.g. whether the user was online on their
- We provide the data for e.g. whether the user was online on their
desktop or the mobile app, but for a basic client, you will likely
only want to parse the "aggregated" key, which shows the summary
answer for "is this user online".

View File

@@ -3,24 +3,24 @@
Zulip uses RabbitMQ to manage a system of internal queues. These are
used for a variety of purposes:
* Asynchronously doing expensive operations like sending email
- Asynchronously doing expensive operations like sending email
notifications which can take seconds per email and thus would
otherwise time out when 100s are triggered at once (E.g. inviting a
lot of new users to a realm).
* Asynchronously doing non-time-critical somewhat expensive operations
- Asynchronously doing non-time-critical somewhat expensive operations
like updating analytics tables (e.g. UserActivityInternal) which
don't have any immediate runtime effect.
* Communicating events to push to clients (browsers, etc.) from the
- Communicating events to push to clients (browsers, etc.) from the
main Zulip Django application process to the Tornado-based events
system. Example events might be that a new message was sent, a user
has changed their subscriptions, etc.
* Processing mobile push notifications and email mirroring system
- Processing mobile push notifications and email mirroring system
messages.
* Processing various errors, frontend tracebacks, and slow database
- Processing various errors, frontend tracebacks, and slow database
queries in a batched fashion.
Needless to say, the RabbitMQ-based queuing system is an important
@@ -35,16 +35,16 @@ custom integration defined in `zerver/lib/queue.py`.
To add a new queue processor:
* Define the processor in `zerver/worker/queue_processors.py` using
- Define the processor in `zerver/worker/queue_processors.py` using
the `@assign_queue` decorator; it's pretty easy to get the template
for an existing similar queue processor. This suffices to test your
queue worker in the Zulip development environment
(`tools/run-dev.py` will automatically restart the queue processors
and start running your new queue processor code). You can also run
a single queue processor manually using e.g. `./manage.py
process_queue --queue=user_activity`.
a single queue processor manually using e.g.
`./manage.py process_queue --queue=user_activity`.
* So that supervisord will know to run the queue processor in
- So that supervisord will know to run the queue processor in
production, you will need to add to the `queues` variable in
`puppet/zulip/manifests/app_frontend_base.pp`; the list there is
used to generate `/etc/supervisor/conf.d/zulip.conf`.
@@ -75,14 +75,14 @@ processor's code path, but it isn't always possible.
If you need to clear a queue (delete all the events in it), run
`./manage.py purge_queue <queue_name>`, for example:
```
```bash
./manage.py purge_queue user_activity
```
You can also use the amqp tools directly. Install `amqp-tools` from
apt and then run:
```
```bash
amqp-delete-queue --username=zulip --password='...' --server=localhost \
--queue=user_presence
```

View File

@@ -1,6 +1,6 @@
# Realms in Zulip
Zulip allows multiple *realms* to be hosted on a single instance.
Zulip allows multiple _realms_ to be hosted on a single instance.
Realms are the Zulip codebases's internal name for what we refer to in
user documentation as an organization (the name "realm" comes from
[Kerberos](https://web.mit.edu/kerberos/)).
@@ -19,8 +19,8 @@ are also relevant reading.
There are two main methods for creating realms.
* Using unique link generator
* Enabling open realm creation
- Using unique link generator
- Enabling open realm creation
#### Using unique link generator
@@ -77,10 +77,10 @@ will try to get the page on your behalf. Since zulipdev.com points
to 127.0.0.1 the proxy server is likely to give you a 503 error. The
workaround is to disable your proxy for `*.zulipdev.com`. The DNS
lookup should still work even if you disable proxy for
*.zulipdev.com. If it doesn't you can add zulipdev.com records in
\*.zulipdev.com. If it doesn't you can add zulipdev.com records in
`/etc/hosts` file. The file should look something like this.
```
```text
127.0.0.1 localhost
127.0.0.1 zulipdev.com

View File

@@ -5,91 +5,92 @@ preparing a new release.
### A week before the release
* For a major release (e.g. 4.0):
* Upgrade all Python dependencies in
- For a major release (e.g. 4.0):
- Upgrade all Python dependencies in
`requirements` to latest upstream versions so they can burn in (use
`pip list --outdated`).
* [Upload strings to
- [Upload strings to
Transifex](../translating/internationalization.html#translation-process)
using `push-translations`. Post a Transifex
[Announcement](https://www.transifex.com/zulip/zulip/announcements/)
notifying translators that we're approaching a release.
* Merge draft updates to the [changelog](../overview/changelog.md)
- Merge draft updates to the [changelog](../overview/changelog.md)
with changes since the last release. While doing so, take notes on
things that might need follow-up work or documentation before we
can happily advertise them in a release blog post.
* Inspect all `TODO/compatibility` comments for whether we can
- Inspect all `TODO/compatibility` comments for whether we can
remove any backwards-compatibility code in this release.
* Create a burn-down list of issues that need to be fixed before we can
- Create a burn-down list of issues that need to be fixed before we can
release, and make sure all of them are being worked on.
* Draft the release blog post (a.k.a. the release notes) in Paper. In
- Draft the release blog post (a.k.a. the release notes) in Paper. In
it, list the important changes in the release, from most to least
notable.
### Final release preparation
* Update the Paper blog post draft with any new commits.
* _Except minor releases:_ Download updated translation strings from
- Update the Paper blog post draft with any new commits.
- _Except minor releases:_ Download updated translation strings from
Transifex and commit them.
* Use `build-release-tarball` to generate a release tarball.
* Test the new tarball extensively, both new install and upgrade from last
- Use `build-release-tarball` to generate a release tarball.
- Test the new tarball extensively, both new install and upgrade from last
release, on both Bionic and Focal.
* Repeat until release is ready.
* Send around the Paper blog post draft for review.
* Move the blog post draft to Ghost. (For a draft in Dropbox Paper,
- Repeat until release is ready.
- Send around the Paper blog post draft for review.
- Move the blog post draft to Ghost. (For a draft in Dropbox Paper,
use "··· > Export > Markdown" to get a pretty good markup
conversion.) Proofread the post, especially for formatting. Tag
the post with "Release announcements" in Ghost.
### Executing the release
* Create the release commit, on master (for major releases) or on the
- Create the release commit, on `main` (for major releases) or on the
release branch (for minor releases):
* Copy the Markdown release notes for the release into
- Copy the Markdown release notes for the release into
`docs/overview/changelog.md`.
* _Except minor releases:_ Adjust the `changelog.md` heading to have
- _Except minor releases:_ Adjust the `changelog.md` heading to have
the stable release series boilerplate.
* Update `ZULIP_VERSION` and `LATEST_RELEASE_VERSION` in `version.py`.
* _Except minor releases:_ Update `API_FEATURE_LEVEL` to a feature
- Update `ZULIP_VERSION` and `LATEST_RELEASE_VERSION` in `version.py`.
- _Except minor releases:_ Update `API_FEATURE_LEVEL` to a feature
level for the final release, and document a reserved range.
* Tag that commit with an unsigned Git tag named the release number.
* Use `build-release-tarball` to generate a final release tarball.
* Push the tag and release commit.
* Copy the tarball to `zulip.org`, and run
- Tag that commit with an unsigned Git tag named the release number.
- Use `build-release-tarball` to generate a final release tarball.
- Push the tag and release commit.
- Copy the tarball to `zulip.org`, and run
`/etc/zulip/ship-release.sh` on it; this will put it in place,
update the latest symlink, and update the SHA256 sums.
* Post the release by [editing the latest tag on
- Post the release by [editing the latest tag on
GitHub](https://github.com/zulip/zulip/tags); use the text from
`changelog.md` for the release notes.
**Note:** This will trigger the [GitHub action](https://github.com/zulip/zulip/blob/master/tools/oneclickapps/README.md)
**Note:** This will trigger the [GitHub action](https://github.com/zulip/zulip/blob/main/tools/oneclickapps/README.md)
for updating DigitalOcean one-click app image. The action uses the latest release
tarball published on `zulip.org` for creating the image.
* Update the [Docker image](https://github.com/zulip/docker-zulip) and
tarball published on `download.zulip.com` for creating the image.
- Update the [Docker image](https://github.com/zulip/docker-zulip) and
do a release of that.
* Update the image of DigitalOcean one click app using
- Update the image of DigitalOcean one click app using
[Fabric](https://github.com/zulip/marketplace-partners) and publish
it to DO marketplace.
* Publish the blog post; check the box to "send by email."
* Email [zulip-announce](https://groups.google.com/g/zulip-announce),
- Publish the blog post; check the box to "send by email."
- Email [zulip-announce](https://groups.google.com/g/zulip-announce),
post to [#announce](https://chat.zulip.org/#narrow/stream/1-announce),
and [send a tweet](https://twitter.com/zulip).
### Post-release
* Following a major release (e.g. 4.0):
* Create a release branch (e.g. `4.x`).
* On the release branch, update `ZULIP_VERSION` in `version.py` to
- Following a major release (e.g. 4.0):
- Create a release branch (e.g. `4.x`).
- On the release branch, update `ZULIP_VERSION` in `version.py` to
the present release with a `+git` suffix, e.g. `4.0+git`.
* On master, update `ZULIP_VERSION` to the future major release with
- On `main`, update `ZULIP_VERSION` to the future major release with
a `-dev+git` suffix, e.g. `5.0-dev+git`. Make a Git tag for this
update commit with a `-dev` suffix, e.g. `5.0-dev`. Push the tag
to both zulip.git and zulip-internal.git to get a correct version
number for future Cloud deployments.
* Consider removing a few old releases from ReadTheDocs; we keep about
- Consider removing a few old releases from ReadTheDocs; we keep about
two years of back-versions.
* Following a minor release (e.g. 3.2), on the release branch:
* Update `ZULIP_VERSION` to the present release with a `+git`
- Following a minor release (e.g. 3.2), on the release branch:
- Update `ZULIP_VERSION` to the present release with a `+git`
suffix, e.g. `3.2+git`.
* Update `LATEST_RELEASE_VERSION` with the released version.
* Cherry-pick the changelog changes back to `master`.
- Update `LATEST_RELEASE_VERSION` with the released version.
- Cherry-pick the changelog changes back to `main`.

View File

@@ -8,53 +8,53 @@ tutorial](../tutorials/new-feature-tutorial.md).
This page documents some important issues related to writing schema
migrations.
* If your database migration is just to reflect new fields in
- If your database migration is just to reflect new fields in
`models.py`, you'll typically want to just:
* Rebase your branch before you start (this may save work later).
* Update the model class definitions in `zerver/models.py`.
* Run `./manage.py makemigrations` to generate a migration file
* Rename the migration file to have a descriptive name if Django
- Rebase your branch before you start (this may save work later).
- Update the model class definitions in `zerver/models.py`.
- Run `./manage.py makemigrations` to generate a migration file
- Rename the migration file to have a descriptive name if Django
generated used a date-based name like `0089_auto_20170710_1353.py`
(which happens when the changes are to multiple models and Django).
* `git add` the new migration file
* Run `tools/provision` to update your local database to apply the
- `git add` the new migration file
- Run `tools/provision` to update your local database to apply the
migrations.
* Commit your changes.
* For more complicated migrations where you need to run custom Python
- Commit your changes.
- For more complicated migrations where you need to run custom Python
code as part of the migration, it's best to read past migrations to
understand how to write them well. `git grep RunPython
zerver/migrations/02*` will find many good examples. Before writing
migrations of this form, you should read Django's docs and the
sections below.
* **Numbering conflicts across branches**: If you've done your schema
understand how to write them well.
`git grep RunPython zerver/migrations/02*` will find many good
examples. Before writing migrations of this form, you should read
Django's docs and the sections below.
- **Numbering conflicts across branches**: If you've done your schema
change in a branch, and meanwhile another schema change has taken
place, Django will now have two migrations with the same
number. There are two easy way to fix this:
* If your migrations were automatically generated using `manage.py
makemigrations`, a good option is to just remove your migration
and rerun the command after rebasing. Remember to `git rebase` to
do this in the the commit that changed `models.py` if you have a
multi-commit branch.
* If you wrote code as part of preparing your migrations, or prefer
- If your migrations were automatically generated using
`manage.py makemigrations`, a good option is to just remove your
migration and rerun the command after rebasing. Remember to
`git rebase` to do this in the the commit that changed `models.py`
if you have a multi-commit branch.
- If you wrote code as part of preparing your migrations, or prefer
this workflow, you can use run `./tools/renumber-migrations`,
which renumbers your migration(s) and fixes up the "dependencies"
entries in your migration(s). The tool could use a bit of work to
prompt unnecessarily less, but it will update the working tree for
you automatically (you still need to do all the `git add`
commands, etc.).
* **Large tables**: For our very largest tables (e.g. Message and
- **Large tables**: For our very largest tables (e.g. Message and
UserMessage), we often need to take precautions when adding columns
to the table, performing data backfills, or building indexes. We
have a `zerver/lib/migrate.py` library to help with adding columns
and backfilling data.
* **Adding indexes** Regular `CREATE INDEX` SQL (corresponding to Django's
- **Adding indexes** Regular `CREATE INDEX` SQL (corresponding to Django's
`AddIndex` operation) locks writes to the affected table. This can be
problematic when dealing with larger tables in particular and we've
generally preferred to use `CREATE INDEX CONCURRENTLY` to allow the index
to be built while the server is active. While in historical migrations
we've used `RunSQL` directly, newer versions of Django add the corresponding
operation `AddIndexConcurrently` and thus that's what should normally be used.
* **Atomicity**. By default, each Django migration is run atomically
- **Atomicity**. By default, each Django migration is run atomically
inside a transaction. This can be problematic if one wants to do
something in a migration that touches a lot of data and would best
be done in batches of e.g. 1000 objects (e.g. a `Message` or
@@ -65,7 +65,7 @@ migrations.
to use the batch update tools in `zerver/lib/migrate.py` (originally
written to work with South) for doing larger database migrations.
* **Accessing code and models in RunPython migrations**. When writing
- **Accessing code and models in RunPython migrations**. When writing
a migration that includes custom python code (aka `RunPython`), you
almost never want to import code from `zerver` or anywhere else in
the codebase. If you imagine the process of upgrading a Zulip
@@ -93,33 +93,35 @@ migrations.
Another important note is that making changes to the data in a table
via `RunPython` code and `ALTER TABLE` operations within a single,
atomic migration don't mix well. If you encounter an error such as
```
```text
django.db.utils.OperationalError: cannot ALTER TABLE "table_name" because it has pending trigger events
```
when testing the migration, the reason is often that these operations
were incorrectly mixed. To resolve this, consider making the migration
non-atomic, splitting it into two migration files (recommended), or replacing the
`RunPython` logic with pure SQL (though this can generally be difficult).
* **Making large migrations work**. Major migrations should have a
- **Making large migrations work**. Major migrations should have a
few properties:
* **Unit tests**. You'll want to carefully test these, so you might
- **Unit tests**. You'll want to carefully test these, so you might
as well write some unit tests to verify the migration works
correctly, rather than doing everything by hand. This often saves
a lot of time in re-testing the migration process as we make
adjustments to the plan.
* **Run in batches**. Updating more than 1K-10K rows (depending on
- **Run in batches**. Updating more than 1K-10K rows (depending on
type) in a single transaction can lock up a database. It's best
to do lots of small batches, potentially with a brief sleep in
between, so that we don't block other operations from finishing.
* **Rerunnability/idempotency**. Good migrations are ones where if
- **Rerunnability/idempotency**. Good migrations are ones where if
operational concerns (e.g. it taking down the Zulip server for
users) interfere with it finishing, it's easy to restart the
migration without doing a bunch of hand investigation. Ideally,
the migration can even continue where it left off, without needing
to redo work.
* **Multi-step migrations**. For really big migrations, one wants
- **Multi-step migrations**. For really big migrations, one wants
to split the transition into into several commits that are each
individually correct, and can each be deployed independently:

View File

@@ -17,13 +17,14 @@ and we generally don't repeat the content discussed there.
This is just a bit of terminology: A "message list" is what Zulip
calls the frontend concept of a (potentially narrowed) message feed.
There are 3 related structures:
* A `message_list_data` just has the sequencing data of which message
- A `message_list_data` just has the sequencing data of which message
IDs go in what order.
* A `message_list` is built on top of `message_list_data` and
- A `message_list` is built on top of `message_list_data` and
additionally contains the data for a visible-to-the-user message list
(E.g. where trailing bookends should appear, a selected message,
etc.).
* A `message_list_view` is built on top of `message_list` and
- A `message_list_view` is built on top of `message_list` and
additionally contains rendering details like a window of up to 400
messages that is present in the DOM at the time, scroll position
controls, etc.
@@ -48,10 +49,10 @@ process described in our
[new application feature tutorial](../tutorials/new-feature-tutorial.md).
This section details the ways in which it is different:
* There is significant custom code inside the `process_message_event`
- There is significant custom code inside the `process_message_event`
function in `zerver/tornado/event_queue.py`. This custom code has a
number of purposes:
* Triggering [email and mobile push
- Triggering [email and mobile push
notifications](../subsystems/notifications.md) for any users who
do not have active clients and have settings of the form "push
notifications when offline". In order to avoid doing any real
@@ -61,41 +62,41 @@ number of purposes:
[queue](../subsystems/queuing.md) to actually send the message.
See `maybe_enqueue_notifications` and related code for this part
of the logic.
* Splicing user-dependent data (E.g. `flags` such as when the user
- Splicing user-dependent data (E.g. `flags` such as when the user
was `mentioned`) into the events.
* Handling the [local echo details](#local-echo).
* Handling certain client configuration options that affect
- Handling the [local echo details](#local-echo).
- Handling certain client configuration options that affect
messages. E.g. determining whether to send the
plaintext/Markdown raw content or the rendered HTML (e.g. the
`apply_markdown` and `client_gravatar` features in our
[events API docs](https://zulip.com/api/register-queue)).
* Following our standard naming convention, input validation is done
- Following our standard naming convention, input validation is done
inside the `check_message` function in `zerver/lib/actions.py`, which is responsible for
validating the user can send to the recipient,
[rendering the Markdown](../subsystems/markdown.md), etc. --
basically everything that can fail due to bad user input.
* The core `do_send_messages` function (which handles actually sending
- The core `do_send_messages` function (which handles actually sending
the message) in `zerver/lib/actions.py` is one of the most optimized and thus complex parts of
the system. But in short, its job is to atomically do a few key
things:
* Store a `Message` row in the database.
* Store one `UserMessage` row in the database for each user who is
- Store a `Message` row in the database.
- Store one `UserMessage` row in the database for each user who is
a recipient of the message (including the sender), with
appropriate `flags` for whether the user was mentioned, an alert
word appears, etc. See
[the section on soft deactivation](#soft-deactivation) for
a clever optimization we use here that is important for large
open organizations.
* Do all the database queries to fetch relevant data for and then
- Do all the database queries to fetch relevant data for and then
send a `message` event to the
[events system](../subsystems/events-system.md) containing the
data it will need for the calculations described above. This
step adds a lot of complexity, because the events system cannot
make queries to the database directly.
* Trigger any other deferred work caused by the current message,
- Trigger any other deferred work caused by the current message,
e.g. [outgoing webhooks](https://zulip.com/api/outgoing-webhooks)
or embedded bots.
* Every query is designed to be a bulk query; we carefully
- Every query is designed to be a bulk query; we carefully
unit-test this system for how many database and memcached queries
it makes when sending messages with large numbers of recipients,
to ensure its performance.
@@ -123,12 +124,12 @@ details on how that works and is tested.
The rest of this section details how Zulip manages locally echoed
messages.
* The core function in the frontend codebase
- The core function in the frontend codebase
`echo.try_deliver_locally`. This checks whether correct local echo
is possible (via `markdown.contains_backend_only_syntax`) and useful
(whether the message would appear in the current view), and if so,
causes Zulip to insert the message into the relevant feed(s).
* Since the message hasn't been confirmed by the server yet, it
- Since the message hasn't been confirmed by the server yet, it
doesn't have a message ID. The frontend makes one up, via
`local_message.next_local_id`, by taking the highest message ID it
has seen and adding the decimal `0.01`. The use of a floating point
@@ -141,14 +142,14 @@ messages.
resort it to its proper place once it is confirmed by the server.
We do it this way to minimize messages jumping around/reordering
visually).
* The `POST /messages` API request to the server to send the message
- The `POST /messages` API request to the server to send the message
is passed two special parameters that clients not implementing local
echo don't use: `queue_id` and `local_id`. The `queue_id` is the ID
of the client's event queue; here, it is used just as a unique
identifier for the specific client (e.g. a browser tab) that sent
the message. And the `local_id` is, by the construction above, a
unique value within that namespace identifying the message.
* The `do_send_messages` backend code path includes the `queue_id` and
- The `do_send_messages` backend code path includes the `queue_id` and
`local_id` in the data it passes to the
[events system](../subsystems/events-system.md). The events
system will extend the `message` event dictionary it delivers to
@@ -156,7 +157,7 @@ messages.
containing the `local_id` that the relevant client used when sending
the message. This allows the client to know that the `message`
event it is receiving is the same message it itself had sent.
* Using that information, rather than adding the "new message" to the
- Using that information, rather than adding the "new message" to the
relevant message feed, it updates the (locally echoed) message's
properties (at the very least, message ID and timestamp) and
rerenders it in any message lists where it appears. This is
@@ -176,30 +177,31 @@ implementation was under 150 lines of code.
This section just has a brief review of the sequence of steps all in
one place:
* User hits send in the compose box.
* Compose box validation runs; if it passes, the browser locally
- User hits send in the compose box.
- Compose box validation runs; if it passes, the browser locally
echoes the message and then sends a request to the `POST /messages`
API endpoint.
* The Django URL routes and middleware run, and eventually call the
- The Django URL routes and middleware run, and eventually call the
`send_message_backend` view function in `zerver/views/messages.py`.
(Alternatively, for an API request to send a message via Zulip's
REST API, things start here).
* `send_message_backend` does some validation before triggering the
- `send_message_backend` does some validation before triggering the
`check_message` + `do_send_messages` backend flow.
* That backend flow saves the data to the database and triggers a
- That backend flow saves the data to the database and triggers a
`message` event in the `notify_tornado` queue (part of the events
system).
* The events system processes, and dispatches that event to all
- The events system processes, and dispatches that event to all
clients subscribed to receive notifications for users who should
receive the message (including the sender). As a side effect, it
adds queue items to the email and push notification queues (which,
in turn, may trigger those notifications).
* Other clients receive the event and display the new message.
* For the client that sent the message, it instead replaces its
- Other clients receive the event and display the new message.
- For the client that sent the message, it instead replaces its
locally echoed message with the final message it received back
from the server (it indicates this to the sender by adding a
display timestamp to the message).
* The `send_message_backend` view function returns
- The `send_message_backend` view function returns
a 200 `HTTP` response; the client receives that response and mostly
does nothing with it other than update some logging details. (This
may happen before or after the client receives the event notifying
@@ -210,17 +212,17 @@ one place:
Message editing uses a very similar principle to how sending messages
works. A few details are worth mentioning:
* `maybe_enqueue_notifications_for_message_update` is an analogue of
- `maybe_enqueue_notifications_for_message_update` is an analogue of
`maybe_enqueue_notifications`, and exists to handle cases like a
user was newly mentioned after the message is edited (since that
should trigger email/push notifications, even if the original
message didn't have one).
* We use a similar technique to what's described in the local echo
- We use a similar technique to what's described in the local echo
section for doing client-side rerendering to update the message feed.
* In the default configuration, Zulip stores the message edit history
- In the default configuration, Zulip stores the message edit history
(which is useful for forensics but also exposed in the UI), in the
`message.edit_history` attribute.
* We support topic editing, including bulk-updates moving several
- We support topic editing, including bulk-updates moving several
messages between topics.
### Inline URL previews
@@ -232,16 +234,16 @@ from the target URL, and for slow websites, this could result in a
significant delay in rendering the message and delivering it to other
users.
* For this case, Zulip's backend Markdown processor will render the
- For this case, Zulip's backend Markdown processor will render the
message without including the URL embeds/previews, but it will add a
deferred work item into the `embed_links` queue.
* The [queue processor](../subsystems/queuing.md) for the
- The [queue processor](../subsystems/queuing.md) for the
`embed_links` queue will fetch the URLs, and then if they return
results, rerun the Markdown processor and notify clients of the
updated message `rendered_content`.
* We reuse the `update_message` framework (used for
- We reuse the `update_message` framework (used for
Zulip's message editing feature) in order to avoid needing custom code
to implement the notification-and-rerender part of this implementation.
@@ -327,10 +329,10 @@ organization for a few weeks, they are tagged as soft-deactivated.
The way this works internally is:
* We (usually) skip creating UserMessage rows for soft-deactivated
- We (usually) skip creating UserMessage rows for soft-deactivated
users when a message is sent to a stream where they are subscribed.
* If/when the user ever returns to Zulip, we can at that time
- If/when the user ever returns to Zulip, we can at that time
reconstruct the UserMessage rows that they missed, and create the rows
at that time (or, to avoid a latency spike if/when the user returns to
Zulip, this work can be done in a nightly cron job). We can construct
@@ -342,7 +344,7 @@ that the messages have not been marked as read by the user). This is
done in the `add_missing_messages` function, which is the core of the
soft-deactivation implementation.
* The “usually” above is because there are a few flags that result
- The “usually” above is because there are a few flags that result
from content in the message (e.g., a message that mentions a user
results in a “mentioned” flag in the UserMessage row), that we need to
keep track of. Since parsing a message can be expensive (>10ms of
@@ -356,13 +358,13 @@ time they were sent without any material performance impact. And then
The end result is the best of both worlds:
* Nobody's view of the world is different because the user was
- Nobody's view of the world is different because the user was
soft-deactivated (resulting in no visible user-experience impact), at
least if one is running the cron job. If one does not run the cron
job, then users returning after being away for a very long time will
potentially have a (very) slow loading experience as potentially
100,000s of UserMessage rows might need to be reconstructed at once.
* On the latency-sensitive message sending and fanout code path, the
- On the latency-sensitive message sending and fanout code path, the
server only needs to do work for users who are currently interacting
with Zulip.
@@ -374,13 +376,14 @@ itll arrive in the couple hundred milliseconds one would expect if
the extra 4500 inactive subscribers didnt exist.
There are a few details that require special care with this system:
* [Email and mobile push
- [Email and mobile push
notifications](../subsystems/notifications.md). We need to make
sure these are still correctly delivered to soft-deactivated users;
making this work required careful work for those code paths that
assumed a `UserMessage` row would always exist for a message that
triggers a notification to a given user.
* Digest emails, which use the `UserMessage` table extensively to
- Digest emails, which use the `UserMessage` table extensively to
determine what has happened in streams the user can see. We can use
the user's subscriptions to construct what messages they should have
access to for this feature.

View File

@@ -5,9 +5,10 @@ help you decide how to correctly implement new settings you're adding
to Zulip.
We have two types of administrative settings in Zulip:
* **Server settings** are set via configuration files, and apply to
- **Server settings** are set via configuration files, and apply to
the whole Zulip installation.
* **Realm settings** (or **organization settings**) are usually
- **Realm settings** (or **organization settings**) are usually
set via the /#organization page in the Zulip web application, and
apply to a single Zulip realm/organization. (Which, for most Zulip
servers, is the only realm on the server).
@@ -15,11 +16,11 @@ We have two types of administrative settings in Zulip:
Philosophically, the goals of the settings system are to make it
convenient for:
* Zulip server administrators to configure
- Zulip server administrators to configure
Zulip's featureset for their server without needing to patch Zulip
* Realm administrators to configure settings for their organization
- Realm administrators to configure settings for their organization
independently without needing to talk with the server administrator.
* Secrets (passwords, API keys, etc.) to be stored in a separate place
- Secrets (passwords, API keys, etc.) to be stored in a separate place
from shareable configuration.
## Server settings
@@ -30,7 +31,7 @@ means that the settings files are Python programs that set a lot of
variables with all-capital names like `EMAIL_GATEWAY_PATTERN`. You can
access these anywhere in the Zulip Django code using e.g.:
```
```python
from django.conf import settings
print(settings.EMAIL_GATEWAY_PATTERN)
```
@@ -38,7 +39,7 @@ print(settings.EMAIL_GATEWAY_PATTERN)
Additionally, if you need to access a Django setting in a shell
script (or just on the command line for debugging), you can use e.g.:
```
```console
$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN
%s@localhost:9991
```
@@ -49,7 +50,7 @@ of settings needed by the Zulip Django app. As a result, there are a
few files involved in the Zulip settings for server administrators.
In a production environment, we have:
* `/etc/zulip/settings.py` (the template is in the Zulip repo at
- `/etc/zulip/settings.py` (the template is in the Zulip repo at
`zproject/prod_settings_template.py`) is the main system
administrator-facing settings file for Zulip. It contains all the
server-specific settings, such as how to send outgoing email, the
@@ -64,7 +65,7 @@ In a production environment, we have:
release](../production/upgrade-or-modify.html#updating-settings-py-inline-documentation)
to pick up new inline documentation.
* `/etc/zulip/zulip-secrets.conf` (generated by
- `/etc/zulip/zulip-secrets.conf` (generated by
`scripts/setup/generate_secrets.py` as part of installation)
contains secrets used by the Zulip installation. These are read
using the standard Python `ConfigParser`, and accessed in
@@ -73,18 +74,18 @@ In a production environment, we have:
stored here, and read using the `get_secret` function in
`zproject/config.py`.
* `zproject/settings.py` is the main Django settings file for Zulip.
- `zproject/settings.py` is the main Django settings file for Zulip.
It imports everything from `zproject/configured_settings.py` and
`zproject/computed_settings.py`.
* `zproject/configured_settings.py` imports everything from
- `zproject/configured_settings.py` imports everything from
`zproject/default_settings.py`, then in a prod environment imports
`/etc/zulip/settings.py` via a symlink.
* `zproject/default_settings.py` has the default values for the settings the
- `zproject/default_settings.py` has the default values for the settings the
user would set in `/etc/zulip/settings.py`.
* `zproject/computed_settings.py` contains all the settings that are
- `zproject/computed_settings.py` contains all the settings that are
constant for all Zulip installations or computed as a function of
`zproject/configured_settings.py` (e.g. configuration for logging,
static assets, middleware, etc.).
@@ -92,10 +93,10 @@ In a production environment, we have:
In a development environment, we have `zproject/settings.py`, and
additionally:
* `zproject/dev_settings.py` has the custom settings for the Zulip development
- `zproject/dev_settings.py` has the custom settings for the Zulip development
environment; these are set after importing `prod_settings_template.py`.
* `zproject/dev-secrets.conf` replaces
- `zproject/dev-secrets.conf` replaces
`/etc/zulip/zulip-secrets.conf`, and is not tracked by Git. This
allows you to configure your development environment to support
features like [authentication
@@ -105,28 +106,28 @@ additionally:
You can see a full list with `git grep development_only=True`, or
add additional settings of this form if needed.
* `zproject/test_settings.py` imports everything from
- `zproject/test_settings.py` imports everything from
`zproject/settings.py` and `zproject/test_extra_settings.py`.
* `zproject/test_extra_settings.py` has the (default) settings used
- `zproject/test_extra_settings.py` has the (default) settings used
for the Zulip tests (both backend and Puppeteer), which are applied on
top of the development environment settings.
When adding a new server setting to Zulip, you will typically add it
in two or three places:
* `zproject/default_settings.py`, with a default value
- `zproject/default_settings.py`, with a default value
for production environments.
* If the settings has a secret key,
- If the settings has a secret key,
you'll add a `get_secret` call in `zproject/computed_settings.py` (and the
user will add the value when they configure the feature).
* In an appropriate section of `zproject/prod_settings_template.py`,
- In an appropriate section of `zproject/prod_settings_template.py`,
with documentation in the comments explaining the setting's
purpose and effect.
* Possibly also `zproject/dev_settings.py` and/or
- Possibly also `zproject/dev_settings.py` and/or
`zproject/test_settings.py`, if the desired value of the setting for
Zulip development and/or test environments is different from the
default for production.
@@ -138,9 +139,9 @@ want those settings.
### Testing non-default settings
You can write tests for settings using e.g. `with
self.settings(TERMS_OF_SERVICE=None)`. However, this only works for
settings which are checked at runtime, not settings which are only
You can write tests for settings using e.g.
`with self.settings(TERMS_OF_SERVICE=None)`. However, this only works
for settings which are checked at runtime, not settings which are only
accessed in initialization of Django (or Zulip) internals
(e.g. `DATABASES`). See the [Django docs on overriding settings in
tests][django-test-settings] for more details.
@@ -159,10 +160,10 @@ process for adding a new realm setting to Zulip.
So for example, the following server settings will eventually be
replaced with realm settings:
* `NAME_CHANGES_DISABLED`
* `INLINE_IMAGE_PREVIEW`
* `ENABLE_GRAVATAR`
* Which authentication methods are allowed should probably appear in
- `NAME_CHANGES_DISABLED`
- `INLINE_IMAGE_PREVIEW`
- `ENABLE_GRAVATAR`
- Which authentication methods are allowed should probably appear in
both places; in server settings indicating the capabilities of the
server, and in the realm settings indicating which methods the realm
administrator wants to allow users to log in with.

View File

@@ -17,8 +17,8 @@ views.
There are two major roles for users in this system:
* The "writing user" is composing a message.
* The "receiving user" is waiting to receive a message (or possibly
- The "writing user" is composing a message.
- The "receiving user" is waiting to receive a message (or possibly
ready to shift their attention elsewhere).
Any Zulip user can play either one of these roles, and sometimes
@@ -28,9 +28,9 @@ being composed by the "writing user."
On a high level the typing indicators system works like this:
* The client for the "writing user" sends requests to the server.
* The server broadcasts events to other users.
* The clients for "receiving users" receive events and conditionally
- The client for the "writing user" sends requests to the server.
- The server broadcasts events to other users.
- The clients for "receiving users" receive events and conditionally
show typing indicators, depending on where the clients are narrowed.
## Writing user
@@ -97,8 +97,8 @@ incoming "typing" events from the server, and the client will
display typing notification only when both of these conditions are
true:
* The "writing user" is still likely typing.
* The "receiving user" is in a view where they'd see the eventual
- The "writing user" is still likely typing.
- The "receiving user" is in a view where they'd see the eventual
message.
The client code starts by processing events, and it maintains data
@@ -118,10 +118,10 @@ The web app client maintains a list of incoming "typists" using
code in `static/js/typing_data.js`. The API here has functions
like the following:
* `add_typist`
* `remove_typist`
* `get_group_typists`
* `get_all_typists`
- `add_typist`
- `remove_typist`
- `get_group_typists`
- `get_all_typists`
One subtle thing that the client has to do here is to maintain
timers for typing notifications. The constant

Some files were not shown because too many files have changed in this diff Show More