From aa6e70382d2c7846e02e5fb433b5a8504ea9af23 Mon Sep 17 00:00:00 2001 From: Anders Kaseorg Date: Fri, 20 Aug 2021 12:53:28 -0700 Subject: [PATCH] docs: Apply sentence single-spacing from Prettier. Signed-off-by: Anders Kaseorg (cherry picked from commit 35c1c8d41b937f7a444fea3866ce88375e7400e2) --- CONTRIBUTING.md | 24 +-- README.md | 2 +- SECURITY.md | 2 +- docs/contributing/accessibility.md | 14 +- docs/contributing/chat-zulip-org.md | 28 +-- docs/contributing/code-reviewing.md | 40 ++-- docs/contributing/code-style.md | 22 +-- docs/contributing/gsoc-ideas.md | 186 +++++++++---------- docs/contributing/summer-with-zulip.md | 6 +- docs/contributing/version-control.md | 14 +- docs/contributing/zulipbot-usage.md | 6 +- docs/development/authentication.md | 36 ++-- docs/development/remote.md | 8 +- docs/development/setup-advanced.md | 12 +- docs/development/setup-vagrant.md | 78 ++++---- docs/development/test-install.md | 4 +- docs/development/using.md | 10 +- docs/documentation/api.md | 50 ++--- docs/documentation/integrations.md | 4 +- docs/documentation/openapi.md | 14 +- docs/documentation/overview.md | 32 ++-- docs/documentation/user.md | 8 +- docs/git/cloning.md | 12 +- docs/git/fixing-commits.md | 2 +- docs/git/overview.md | 2 +- docs/git/pull-requests.md | 4 +- docs/git/setup.md | 4 +- docs/git/using.md | 2 +- docs/git/working-copies.md | 6 +- docs/git/zulip-tools.md | 10 +- docs/overview/architecture-overview.md | 28 +-- docs/overview/changelog.md | 168 ++++++++--------- docs/overview/directory-structure.md | 12 +- docs/overview/release-lifecycle.md | 16 +- docs/production/authentication-methods.md | 122 ++++++------ docs/production/deployment.md | 62 +++---- docs/production/email-gateway.md | 22 +-- docs/production/email.md | 30 +-- docs/production/expensive-migrations.md | 8 +- docs/production/export-and-import.md | 72 +++---- docs/production/giphy-gif-integration.md | 8 +- docs/production/install-existing-server.md | 14 +- docs/production/install.md | 24 +-- docs/production/management-commands.md | 22 +-- docs/production/mobile-push-notifications.md | 36 ++-- docs/production/multiple-organizations.md | 28 +-- docs/production/password-strength.md | 10 +- docs/production/postgresql.md | 10 +- docs/production/requirements.md | 46 ++--- docs/production/security-model.md | 36 ++-- docs/production/settings.md | 16 +- docs/production/ssl-certificates.md | 42 ++--- docs/production/troubleshooting.md | 30 +-- docs/production/upgrade-or-modify.md | 88 ++++----- docs/production/upload-backends.md | 16 +- docs/production/video-calls.md | 4 +- docs/subsystems/analytics.md | 22 +-- docs/subsystems/caching.md | 58 +++--- docs/subsystems/client.md | 4 +- docs/subsystems/dependencies.md | 136 +++++++------- docs/subsystems/django-upgrades.md | 10 +- docs/subsystems/email.md | 22 +-- docs/subsystems/emoji.md | 34 ++-- docs/subsystems/events-system.md | 98 +++++----- docs/subsystems/full-text-search.md | 6 +- docs/subsystems/hashchange-system.md | 20 +- docs/subsystems/hotspots.md | 2 +- docs/subsystems/html-css.md | 38 ++-- docs/subsystems/input-pills.md | 6 +- docs/subsystems/logging.md | 26 +-- docs/subsystems/management-commands.md | 12 +- docs/subsystems/markdown.md | 48 ++--- docs/subsystems/notifications.md | 10 +- docs/subsystems/performance.md | 66 +++---- docs/subsystems/pointer.md | 8 +- docs/subsystems/presence.md | 26 +-- docs/subsystems/queuing.md | 18 +- docs/subsystems/realms.md | 16 +- docs/subsystems/release-checklist.md | 12 +- docs/subsystems/schema-migrations.md | 34 ++-- docs/subsystems/sending-messages.md | 78 ++++---- docs/subsystems/settings.md | 24 +-- docs/subsystems/typing-indicators.md | 44 ++--- docs/subsystems/widgets.md | 82 ++++---- docs/testing/continuous-integration.md | 10 +- docs/testing/linters.md | 40 ++-- docs/testing/manual-testing.md | 42 ++--- docs/testing/mypy.md | 76 ++++---- docs/testing/philosophy.md | 40 ++-- docs/testing/testing-with-django.md | 78 ++++---- docs/testing/testing-with-node.md | 22 +-- docs/testing/testing-with-puppeteer.md | 20 +- docs/testing/testing.md | 22 +-- docs/testing/typescript.md | 18 +- docs/translating/chinese.md | 2 +- docs/translating/internationalization.md | 36 ++-- docs/translating/russian.md | 2 +- docs/translating/translating.md | 20 +- docs/tutorials/new-feature-tutorial.md | 20 +- docs/tutorials/reading-list.md | 2 +- docs/tutorials/shell-tips.md | 4 +- docs/tutorials/writing-views.md | 26 +-- static/shared/README.md | 2 +- tools/droplets/README.md | 2 +- 104 files changed, 1528 insertions(+), 1528 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b3684cbae..79b0658e57 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -75,7 +75,7 @@ to help. and do the Git tutorial (coming soon) if you are unfamiliar with Git, getting help in [#git help](https://chat.zulip.org/#narrow/stream/44-git-help) if - you run into any troubles. Be sure to check out the + you run into any troubles. Be sure to check out the [extremely useful Zulip-specific tools page](https://zulip.readthedocs.io/en/latest/git/zulip-tools.html). ### Picking an issue @@ -96,7 +96,7 @@ on. - For the main server and web repository, we recommend browsing recently opened issues to look for issues you are confident you can fix correctly in a way that clearly communicates why your changes - are the correct fix. Our GitHub workflow bot, zulipbot, limits + are the correct fix. Our GitHub workflow bot, zulipbot, limits users who have 0 commits merged to claiming a single issue labeled with "good first issue" or "help wanted". - We also partition all of our issues in the main repo into areas like @@ -127,9 +127,9 @@ Other notes: [good first issue](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) and [help wanted](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) - labels. Avoid issues with the "difficult" label unless you + labels. Avoid issues with the "difficult" label unless you understand why it is difficult and are confident you can resolve the - issue correctly and completely. Issues without one of these labels + issue correctly and completely. Issues without one of these labels are fair game if Tim has written a clear technical design proposal in the issue, or it is a bug that you can reproduce and you are confident you can fix the issue correctly. @@ -172,7 +172,7 @@ labels. ## What makes a great Zulip contributor? -Zulip has a lot of experience working with new contributors. In our +Zulip has a lot of experience working with new contributors. In our experience, these are the best predictors of success: - Posting good questions. This generally means explaining your current @@ -216,7 +216,7 @@ if appropriate. **Reporting security issues**. Please do not report security issues publicly, including on public streams on chat.zulip.org. You can - email security@zulip.com. We create a CVE for every security + email security@zulip.com. We create a CVE for every security issue in our released software. ## User feedback @@ -257,7 +257,7 @@ mind: publicly viewable patches you can link to from your resume, regardless of whether you are selected. - To apply, you'll have to submit at least one pull request to a Zulip - repository. Most students accepted to one of our programs have + repository. Most students accepted to one of our programs have several merged pull requests (including at least one larger PR) by the time of the application deadline. - The main criteria we use is quality of your best contributions, and @@ -274,7 +274,7 @@ important parts of the project. We hope you apply! ### Google Summer of Code The largest outreach program Zulip participates in is GSoC (14 -students in 2017; 11 in 2018; 17 in 2019; 18 in 2020). While we don't control how +students in 2017; 11 in 2018; 17 in 2019; 18 in 2020). While we don't control how many slots Google allocates to Zulip, we hope to mentor a similar number of students in future summers. @@ -282,9 +282,9 @@ If you're reading this well before the application deadline and want to make your application strong, we recommend getting involved in the community and fixing issues in Zulip now. Having good contributions and building a reputation for doing good work is the best way to have -a strong application. About half of Zulip's GSoC students for Summer +a strong application. About half of Zulip's GSoC students for Summer 2017 had made significant contributions to the project by February -2017, and about half had not. Our +2017, and about half had not. Our [GSoC project ideas page][gsoc-guide] has lots more details on how Zulip does GSoC, as well as project ideas (though the project idea list is maintained only during the GSoC application period, so if @@ -293,9 +293,9 @@ out-of-date). We also have in some past years run a Zulip Summer of Code (ZSoC) program for students who we didn't have enough slots to accept for -GSoC but were able to find funding for. Student expectations are the +GSoC but were able to find funding for. Student expectations are the same as with GSoC, and it has no separate application process; your -GSoC application is your ZSoC application. If we'd like to select you +GSoC application is your ZSoC application. If we'd like to select you for ZSoC, we'll contact you when the GSoC results are announced. [gsoc-guide]: https://zulip.readthedocs.io/en/latest/contributing/gsoc-ideas.html diff --git a/README.md b/README.md index 85136eaa34..c8732dc70c 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ You might be interested in: - **Contributing code**. Check out our [guide for new contributors](https://zulip.readthedocs.io/en/latest/overview/contributing.html) - to get started. Zulip prides itself on maintaining a clean and + to get started. Zulip prides itself on maintaining a clean and well-tested codebase, and a stock of hundreds of [beginner-friendly issues][beginner-friendly]. diff --git a/SECURITY.md b/SECURITY.md index 3411b57b44..91d853aea2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,7 +8,7 @@ so you should subscribe if you are running Zulip in production. We love responsible reports of (potential) security issues in Zulip, whether in the latest release or our development branch. -Our security contact is security@zulip.com. Reporters should expect a +Our security contact is security@zulip.com. Reporters should expect a response within 24 hours. Please include details on the issue and how you'd like to be credited diff --git a/docs/contributing/accessibility.md b/docs/contributing/accessibility.md index 783f43d791..9886250583 100644 --- a/docs/contributing/accessibility.md +++ b/docs/contributing/accessibility.md @@ -3,7 +3,7 @@ ## Guidelines In order to accommodate all users, Zulip strives to implement accessibility -best practices in its user interface. There are many aspects to accessibility; +best practices in its user interface. There are many aspects to accessibility; here are some of the more important ones to keep in mind. - All images should have alternative text attributes for the benefit of users @@ -20,7 +20,7 @@ here are some of the more important ones to keep in mind. There are many different standards for accessibility, but the most relevant one for Zulip is the W3C's [WCAG](https://www.w3.org/TR/WCAG20/) (Web Content -Accessibility Guidelines), currently at version 2.0. Whenever practical, we +Accessibility Guidelines), currently at version 2.0. Whenever practical, we should strive for compliance with the AA level of this specification. (The W3C itself [recommends not trying](https://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html#uc-conf-req1-head) @@ -30,11 +30,11 @@ as it is not possible for some content.) ## Tools There are tools available to automatically audit a web page for compliance -with many of the WCAG guidelines. Here are some of the more useful ones: +with many of the WCAG guidelines. Here are some of the more useful ones: - [Accessibility Developer Tools][chrome-webstore] This open source Chrome extension from Google adds an accessibility audit to - the "Audits" tab of the Chrome Developer Tools. The audit is performed + the "Audits" tab of the Chrome Developer Tools. The audit is performed against the page's DOM via JavaScript, allowing it to identify some issues that a static HTML inspector would miss. - [aXe](https://www.deque.com/products/axe/) An open source Chrome and Firefox @@ -42,14 +42,14 @@ with many of the WCAG guidelines. Here are some of the more useful ones: extension. - [Wave](https://wave.webaim.org/) This web application takes a URL and loads it in a frame, reporting on all the issues it finds with links to more - information. Has the advantage of not requiring installation, but requires + information. Has the advantage of not requiring installation, but requires a URL which can be directly accessed by an external site. - [Web Developer](https://chrispederick.com/work/web-developer/) This browser extension has many useful features, including a convenient link for opening the current URL in Wave to get an accessibility report. Note that these tools cannot catch all possible accessibility problems, and -sometimes report false positives as well. They are a useful aid in quickly +sometimes report false positives as well. They are a useful aid in quickly identifying potential problems and checking for regressions, but their recommendations should not be blindly obeyed. @@ -57,7 +57,7 @@ recommendations should not be blindly obeyed. Problems with Zulip's accessibility should be reported as [GitHub issues](https://github.com/zulip/zulip/issues) with the "accessibility" -label. This label can be added by entering the following text in a separate +label. This label can be added by entering the following text in a separate comment on the issue: > @zulipbot add "area: accessibility" diff --git a/docs/contributing/chat-zulip-org.md b/docs/contributing/chat-zulip-org.md index 08f25d23b5..b21d9d9d96 100644 --- a/docs/contributing/chat-zulip-org.md +++ b/docs/contributing/chat-zulip-org.md @@ -4,10 +4,10 @@ forum for the Zulip community. You can go through the simple signup process at that link, and then -you will soon be talking to core Zulip developers and other users. To +you will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luck finding core developers roughly between 17:00 UTC and 6:00 UTC, but the sun never -sets on the Zulip community. Most questions get a reply within +sets on the Zulip community. Most questions get a reply within minutes to a few hours, depending on the time of day. ## Community norms @@ -16,13 +16,13 @@ minutes to a few hours, depending on the time of day. [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) or as a PM to yourself to avoid disturbing others. - When asking for help, provide the details needed for others to help - you. E.g. include the full traceback in a code block (not a + you. E.g. include the full traceback in a code block (not a screenshot), a link to the code or a WIP PR you're having trouble debugging, etc. - Ask questions on streams rather than PMing core contributors. You'll get answers faster since other people can help, and it makes it possible for other developers to learn from reading the discussion. -- Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's +- Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's usually easy to see which message you're replying to, so you don't need to mention your conversation partner in every reply. Mentioning other users is great for timely questions or making sure @@ -33,9 +33,9 @@ minutes to a few hours, depending on the time of day. For example, avoid using a pronoun like her or his in sentences like "Every developer should clean [their] keyboard at least once a week." - Follow the community [code of conduct](../code-of-conduct.md). -- Participate! Zulip is a friendly and welcoming community, and we +- Participate! Zulip is a friendly and welcoming community, and we love meeting new people, hearing about what brought them to Zulip, - and getting their feedback. If you're not sure where to start, + and getting their feedback. If you're not sure where to start, introduce yourself and your interests in [#new members](https://chat.zulip.org/#narrow/stream/95-new-members), using your name as the topic. @@ -44,7 +44,7 @@ minutes to a few hours, depending on the time of day. The chat.zulip.org community sends several thousand messages every single week, about half of them to streams that we have included in -the default streams for new users for discoverability. Keeping up +the default streams for new users for discoverability. Keeping up with **everything** happening in the Zulip project is both difficult and rarely a useful goal. @@ -56,7 +56,7 @@ streams that are only of occasional interest. The chat.zulip.org server is frequently deployed off of `main` from the Zulip Git repository, so please point out anything you notice that -seems wrong! We catch many bugs that escape code review this way. +seems wrong! We catch many bugs that escape code review this way. The chat.zulip.org server is a development and testing server, not a production service, so don't use it for anything mission-critical, @@ -74,7 +74,7 @@ everyone, even non-developers: posting feedback on Zulip. - [#design](https://chat.zulip.org/#narrow/stream/101-design) is where we discuss UI and feature design and collect feedback on potential design - changes. We love feedback, so don't hesitate to speak up! + changes. We love feedback, so don't hesitate to speak up! - [#user community](https://chat.zulip.org/#narrow/stream/138-user-community) is for Zulip users to discuss their experiences using and adopting Zulip. - [#production help](https://chat.zulip.org/#narrow/stream/31-production-help) @@ -86,17 +86,17 @@ everyone, even non-developers: There are dozens of streams for development discussions in the Zulip community (e.g. one for each app, etc.); check out the [Streams page](https://chat.zulip.org/#streams/all) to see the -descriptions for all of them. Relevant to almost everyone are these: +descriptions for all of them. Relevant to almost everyone are these: - [#checkins](https://chat.zulip.org/#narrow/stream/65-checkins) is for progress updates on what you're working on and its status; usually - folks post with their name as the topic. Everyone is welcome to + folks post with their name as the topic. Everyone is welcome to participate! - [#development help](https://chat.zulip.org/#narrow/stream/49-development-help) is for asking for help with any Zulip server/webapp development work (use the app streams for help working on one of the apps). - [#code review](https://chat.zulip.org/#narrow/stream/91-code-review) - is for getting feedback on your work. We encourage all developers + is for getting feedback on your work. We encourage all developers to comment on work posted here, even if you're new to the Zulip project; reviewing other PRs is a great way to develop experience, and even just manually testing a proposed new feature and posting @@ -112,14 +112,14 @@ descriptions for all of them. Relevant to almost everyone are these: There are also official private streams, including large ones for established community contributors (and for GSoC mentors), and small streams where Kandra Labs staff can discuss customer support, -production server operations, and security issues. Because our +production server operations, and security issues. Because our community values are to work in the open, these private streams are relatively low traffic. ## Searching for past conversations When [searching][] for previous discussions of a given topic, we -recommend using the `streams:public keyword` set of operators. This +recommend using the `streams:public keyword` set of operators. This will search the full history of public streams in the organization for `keyword` (including messages sent before you joined and on public streams you're not subscribed to). diff --git a/docs/contributing/code-reviewing.md b/docs/contributing/code-reviewing.md index a1f294c39b..9e85f7012c 100644 --- a/docs/contributing/code-reviewing.md +++ b/docs/contributing/code-reviewing.md @@ -1,8 +1,8 @@ # Reviewing Zulip code -Code review is a key part of how Zulip does development! If you've +Code review is a key part of how Zulip does development! If you've been contributing to Zulip's code, we'd love for you to do reviews. -This is a guide to how. (With some thoughts for writing code too.) +This is a guide to how. (With some thoughts for writing code too.) ## Protocol for authors @@ -26,7 +26,7 @@ to dive right into reviewing the PR's core functionality. ### Responding to a review feedback Once you've received a review and resolved any feedback, it's critical -to update the GitHub thread to reflect that. Best practices are to: +to update the GitHub thread to reflect that. Best practices are to: - Make sure that CI passes and the PR is rebased onto recent `main`. - Post comments on each feedback thread explaining at least how you @@ -38,7 +38,7 @@ to update the GitHub thread to reflect that. Best practices are to: - Post a summary comment in the main feed for the PR, explaining that this is ready for another review, and summarizing any changes from the previous version, details on how you tested the changes, new - screenshots/etc. More detail is better than less, as long as you + screenshots/etc. More detail is better than less, as long as you take the time to write clearly. If you resolve the feedback, but the PR has merge conflicts, CI @@ -48,7 +48,7 @@ will assume it isn't ready for review and move on to other work. If you need help or think an open discussion topic requires more feedback or a more complex discussion, move the discussion to a topic -in the Zulip development community server. Be sure to provide links +in the Zulip development community server. Be sure to provide links from the GitHub PR to the conversation (and vice versa) so that it's convenient to read both conversations together. @@ -74,7 +74,7 @@ those are really helpful contributions. Doing code reviews is an important part of making the project grow. It's also an important skill to develop for participating in -open-source projects and working in the industry in general. If +open-source projects and working in the industry in general. If you're contributing to Zulip and have been working in our code for a little while, we would love for some of your time contributing to come in the form of doing code reviews! @@ -86,7 +86,7 @@ the first couple of weeks as you're getting going) doing code reviews. ### Fast replies are key For the author of a PR, getting feedback quickly is really important -for making progress quickly and staying productive. That means that +for making progress quickly and staying productive. That means that if you get @-mentioned on a PR with a request for you to review it, it helps the author a lot if you reply promptly. @@ -99,9 +99,9 @@ review the PR. People in the Zulip project live and work in many timezones, and code reviewers also need focused chunks of time to write code and do other -things, so an immediate reply isn't always possible. But a good +things, so an immediate reply isn't always possible. But a good benchmark is to try to always reply **within one workday**, at least -with a short initial reply, if you're working regularly on Zulip. And +with a short initial reply, if you're working regularly on Zulip. And sooner is better. ## Things to look for @@ -123,25 +123,25 @@ sooner is better. - *Technical design.* There are a lot of considerations here: security, migration paths/backwards compatibility, cost of new dependencies, interactions with features, speed of performance, API - changes. Security is especially important and worth thinking about + changes. Security is especially important and worth thinking about carefully with any changes to security-sensitive code like views. - *User interface and visual design.* If frontend changes are involved, the reviewer will check out the code, play with the new UI, and verify it for both quality and consistency with the rest of - the Zulip UI. We highly encourage posting screenshots to save + the Zulip UI. We highly encourage posting screenshots to save reviewers time in getting a feel for what the feature looks like -- you'll get a quicker response that way. - *Error handling.* The code should always check for invalid user - input. User-facing error messages should be clear and when possible + input. User-facing error messages should be clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct the problem). - *Testing.* The tests should validate that the feature works correctly, and specifically test for common error conditions, bad user input, and potential bugs that are likely for the type of - change being made. Tests that exclude whole classes of potential + change being made. Tests that exclude whole classes of potential bugs are preferred when possible (e.g., the common test suite `test_markdown.py` between the Zulip server's [frontend and backend Markdown processors](../subsystems/markdown.md), or the `GetEventsTest` test for @@ -158,7 +158,7 @@ sooner is better. - *Duplicated code.* Code duplication is a huge source of bugs in large projects and makes the codebase difficult to understand, so we - avoid significant code duplication wherever possible. Sometimes + avoid significant code duplication wherever possible. Sometimes avoiding code duplication involves some refactoring of existing code; if so, that should usually be done as its own series of commits (not squashed into other changes or left as a thing to do @@ -170,11 +170,11 @@ sooner is better. discussion for the feature itself. - *Completeness.* For refactorings, verify that the changes are - complete. Usually one can check that efficiently using `git grep`, + complete. Usually one can check that efficiently using `git grep`, and it's worth it, as we very frequently find issues by doing so. -- *Documentation updates.* If this changes how something works, does it - update the documentation in a corresponding way? If it's a new +- *Documentation updates.* If this changes how something works, does it + update the documentation in a corresponding way? If it's a new feature, is it documented, and documented in the right place? - *Good comments.* It's often worth thinking about whether explanation @@ -185,7 +185,7 @@ sooner is better. lots of clever tricks. - *Coding style.* See the Zulip [code-style] documentation for - details. Our goal is to have as much of this as possible verified + details. Our goal is to have as much of this as possible verified via the linters and tests, but there's always going to be unusual forms of Python/JavaScript style that our tools don't check for. @@ -202,11 +202,11 @@ Some points specific to the Zulip server codebase: the various error conditions. - *Testing -- Frontend.* If the feature involves frontend changes, - there should be frontend tests. See the [test + there should be frontend tests. See the [test writing][test-writing] documentation for more details. - *mypy annotations.* New functions should be annotated using [mypy] - and existing annotations should be updated. Use of `Any`, `ignore`, + and existing annotations should be updated. Use of `Any`, `ignore`, and unparameterized containers should be limited to cases where a more precise type cannot be specified. diff --git a/docs/contributing/code-style.md b/docs/contributing/code-style.md index 83d7b66682..48aa63fe4d 100644 --- a/docs/contributing/code-style.md +++ b/docs/contributing/code-style.md @@ -2,7 +2,7 @@ One can summarize Zulip's coding philosophy as a relentless focus on making the codebase easy to understand and difficult to make dangerous -mistakes in. The majority of work in any large software development +mistakes in. The majority of work in any large software development project is understanding the existing code so one can debug or modify it, and investments in code readability usually end up paying for themselves when someone inevitably needs to debug or improve the code. @@ -15,10 +15,10 @@ comments/docstrings, and commit messages (roughly in order of priority better than writing a comment explaining how the bad interface works). This page documents code style policies that every Zulip developer -should understand. We aim for this document to be short and focused +should understand. We aim for this document to be short and focused only on details that cannot be easily enforced another way (e.g. through linters, automated tests, subsystem design that makes classes -of mistakes unlikely, etc.). This approach minimizes the cognitive +of mistakes unlikely, etc.). This approach minimizes the cognitive load of ensuring a consistent coding style for both contributors and maintainers. @@ -86,8 +86,8 @@ for bar in bars: # Make use of foo ``` -...which makes a database query for every Bar. While this may be fast -locally in development, it may be quite slow in production! Instead, +...which makes a database query for every Bar. While this may be fast +locally in development, it may be quite slow in production! Instead, tell Django's [QuerySet API](https://docs.djangoproject.com/en/dev/ref/models/querysets/) to _prefetch_ the data in the initial query: @@ -263,9 +263,9 @@ code a lot uglier, in which case it's fine to go up to 120 or so. ### JavaScript and TypeScript Our JavaScript and TypeScript code is formatted with -[Prettier](https://prettier.io/). You can ask Prettier to reformat +[Prettier](https://prettier.io/). You can ask Prettier to reformat all code via our [linter tool](../testing/linters.md) with -`tools/lint --only=prettier --fix`. You can also [integrate it with your +`tools/lint --only=prettier --fix`. You can also [integrate it with your editor](https://prettier.io/docs/en/editors.html). Combine adjacent on-ready functions, if they are logically related. @@ -306,7 +306,7 @@ call a helper function instead. ### HTML / CSS -Our CSS is formatted with [Prettier](https://prettier.io/). You can +Our CSS is formatted with [Prettier](https://prettier.io/). You can ask Prettier to reformat all code via our [linter tool](../testing/linters.md) with `tools/lint --only=prettier --fix`. You can also [integrate it with your @@ -324,10 +324,10 @@ type changes in the future. - Our Python code is formatted with [Black](https://github.com/psf/black) and - [isort](https://pycqa.github.io/isort/). The [linter + [isort](https://pycqa.github.io/isort/). The [linter tool](../testing/linters.md) enforces this by running Black and isort in check mode, or in write mode with - `tools/lint --only=black,isort --fix`. You may find it helpful to + `tools/lint --only=black,isort --fix`. You may find it helpful to [integrate Black](https://black.readthedocs.io/en/stable/editor_integration.html) and @@ -355,7 +355,7 @@ type changes in the future. Clear, readable code is important for [tests](../testing/testing.md); familiarize yourself with our testing frameworks so that you can write -clean, readable tests. Comments about anything subtle about what is +clean, readable tests. Comments about anything subtle about what is being verified are appreciated. ### Third party code diff --git a/docs/contributing/gsoc-ideas.md b/docs/contributing/gsoc-ideas.md index 7f62aab16d..87891c671b 100644 --- a/docs/contributing/gsoc-ideas.md +++ b/docs/contributing/gsoc-ideas.md @@ -10,7 +10,7 @@ integrations, all open source. Zulip has gained a considerable amount of traction since it was [released as open source software][oss-release] in late 2015, with code contributions from [over 700 people](https://zulip.com/team) -from all around the world. Thousands of people use Zulip every single +from all around the world. Thousands of people use Zulip every single day, and your work on Zulip will have impact on the daily experiences of a large and rapidly growing number of people. @@ -19,7 +19,7 @@ of a large and rapidly growing number of people. As an organization, we value high-quality, responsive mentorship and making sure our product quality is extremely high -- you can expect to experience disciplined code reviews by highly experienced -engineers. Since Zulip is a team chat product, your GSoC experience +engineers. Since Zulip is a team chat product, your GSoC experience with the Zulip project will be highly interactive. As part of that commitment, Zulip has over 160,000 words of @@ -31,17 +31,17 @@ the way that it does. ### Our history with Google Open Source Programs Zulip has been a GSoC mentoring organization since 2016, and we aim -for 15-20 GSoC students each summer. We have some of the highest +for 15-20 GSoC students each summer. We have some of the highest standards of any GSoC organization; successful applications generally have dozens of commits integrated into Zulip or other open source -projects by the time we review their application. See [our +projects by the time we review their application. See [our contributing guide](../overview/contributing.md) for details on getting involved with GSoC. Zulip participated in GSoC 2016 and mentored three successful students officially (plus 4 more who did their proposed projects unofficially). We had 14 (+3) students in 2017, 10 (+3) students in 2018, 17 (+1) in -2019, and 18 in 2020. We've also mentored five Outreachy interns and +2019, and 18 in 2020. We've also mentored five Outreachy interns and hundreds of Google Code-In participants (several of who are major contributors to the project today). @@ -55,7 +55,7 @@ the summer). [Our guide for having a great summer with Zulip](../contributing/summer-with-zulip.md) is focused on what one should know once doing a summer project with -Zulip. But it has a lot of useful advice on how we expect students to +Zulip. But it has a lot of useful advice on how we expect students to interact, above and beyond what is discussed in Google's materials. [What makes a great Zulip contributor](../overview/contributing.html#what-makes-a-great-zulip-contributor) @@ -69,7 +69,7 @@ We also recommend reviewing Finally, keep your eye on [the GSoC timeline](https://developers.google.com/open-source/gsoc/timeline). The -student application deadline is April 13, 2021. However, as is +student application deadline is April 13, 2021. However, as is discussed in detail later in this document, we recommend against working on a proposal until 2 weeks before the deadline. @@ -114,7 +114,7 @@ Your application should include the following: We expect applicants to either have experience with the technologies relevant to their project or have strong general programming -experience. We also expect applicants to be excited about learning +experience. We also expect applicants to be excited about learning how to do disciplined, professional software engineering, where they can demonstrate through reasoning and automated tests that their code is correct. @@ -166,19 +166,19 @@ post.](https://www.harihareswara.net/sumana/2016/10/12/0) ## Mentors Zulip has dozens of longtime contributors who sign up to mentoring -projects. We usually decide who will mentor which projects based in +projects. We usually decide who will mentor which projects based in part on who is a good fit for the needs of each student as well as technical expertise as well as who has available time during the -summer. You can reach us via +summer. You can reach us via [#GSoC](https://chat.zulip.org/#narrow/stream/14-GSoC) on [the Zulip development community server](../contributing/chat-zulip-org.md), (compose a new stream message with your name as the topic). -Zulip operates under group mentorship. That means you should +Zulip operates under group mentorship. That means you should generally post in public streams on chat.zulip.org, not send private -messages, for assistance. Our preferred approach is to just post in +messages, for assistance. Our preferred approach is to just post in an appropriate public stream on chat.zulip.org and someone will help -you. We list the Zulip contributors who are experts for various +you. We list the Zulip contributors who are experts for various projects by name below; they will likely be able to provide you with the best feedback on your proposal (feel free to @-mention them in your Zulip post). In practice, this allows project leadership to @@ -188,11 +188,11 @@ However, the first and most important thing to do for building a strong application is to show your skills by contributing to a large open source project like Zulip, to show that you can work effectively in a large codebase (it doesn't matter what part of Zulip, and we're -happy to consider work in other open source projects). The quality of +happy to consider work in other open source projects). The quality of your best work is more important to us than the quantity; so be sure to test your work before submitting it for review and follow our coding guidelines (and don't worry if you make mistakes in your first -few contributions! Everyone makes mistakes getting started. Just +few contributions! Everyone makes mistakes getting started. Just make sure you don't make the same mistakes next time). Once you have several PRs merged (or at least one significant PR @@ -206,8 +206,8 @@ online. These are the seeds of ideas; you will need to do research on the Zulip codebase, read issues on GitHub, and talk with developers to put -together a complete project proposal. It's also fine for you to come -up with your own project ideas. As you'll see below, you can put +together a complete project proposal. It's also fine for you to come +up with your own project ideas. As you'll see below, you can put together a great project around one of the [area labels](https://github.com/zulip/zulip/labels) on GitHub; each has a cluster of problems in one part of the Zulip project that we'd @@ -217,13 +217,13 @@ We don't believe in labeling projects by difficulty (e.g. a project that involves writing a lot of documentation will be hard for some great programmers, and a UI design project might be hard for a great backend programmer, while a great writer might have trouble doing -performance work). To help you find a great project, we list the +performance work). To help you find a great project, we list the skills needed, and try to emphasize where strong skills with particular tools are likely to be important for a given project. For all of our projects, an important skill to develop is a good command of Git; read [our Git guide](../git/overview.md) in full to -learn how to use it well. Of particular importance is mastering using +learn how to use it well. Of particular importance is mastering using Git rebase so that you can construct commits that are clearly correct and explain why they are correct. We highly recommend investing in learning a [graphical Git client](../git/setup.md) and learning to @@ -246,13 +246,13 @@ set of 8 issues may not be the right ones to invest in. For 2021, we are particularly interested in GSoC students who have strong skills at visual design, HTML/CSS, mobile development, -performance optimization, or Electron. So if you're a student with +performance optimization, or Electron. So if you're a student with those skills and are looking for an organization to join, we'd love to talk to you! The Zulip project has a huge surface area, so even when we're focused on something, a huge amount of essential work goes into other parts of -the project. Every area of Zulip could benefit from the work of a +the project. Every area of Zulip could benefit from the work of a student with strong programming skills; so don't feel discouraged if the areas mentioned above are not your main strength. @@ -273,12 +273,12 @@ CSS](https://github.com/zulip/zulip/). Zulip has a [nice framework](../documentation/api.md) for writing API documentation built by past GSoC students based on the OpenAPI standard with built-in automated tests of the data both the Python - and curl examples. However, the documentation isn't yet what we're + and curl examples. However, the documentation isn't yet what we're hoping for: there are a few dozen endpoints that are missing, several of which are quite important, the visual design isn't perfect (especially for e.g. `GET /events`), many template could be - deleted with a bit of framework effort, etc. See the [API docs area - label][api-docs-area] for many specific projects in the area. Our + deleted with a bit of framework effort, etc. See the [API docs area + label][api-docs-area] for many specific projects in the area. Our goal for the summer is for 1-2 students to resolve all open issues related to the REST API documentation. @@ -288,46 +288,46 @@ CSS](https://github.com/zulip/zulip/). Zulip, including [default stream groups](https://github.com/zulip/zulip/issues/13670), [Mute User](https://github.com/zulip/zulip/issues/168), and [public - access](https://github.com/zulip/zulip/issues/13172). Expert: Tim - Abbott. Many of these issues have open PRs with substantial work + access](https://github.com/zulip/zulip/issues/13172). Expert: Tim + Abbott. Many of these issues have open PRs with substantial work towards the goal, but each of them is likely to have dozens of adjacent or follow-up tasks. - Fill in gaps, fix bugs, and improve the framework for Zulip's - library of native integrations. We have about 100 integrations, but - there are a handful of important integrations that are missing. The + library of native integrations. We have about 100 integrations, but + there are a handful of important integrations that are missing. The [the integrations label on GitHub](https://github.com/zulip/zulip/labels/area%3A%20integrations) lists some of the priorities here (many of which are great preparatory projects); once those are cleared, we'll likely have - many more. **Skills required**: Strong Python experience, will to - do careful manual testing of third-party products. Fluent English, + many more. **Skills required**: Strong Python experience, will to + do careful manual testing of third-party products. Fluent English, usability sense and/or technical writing skills are all pluses. Expert: Eeshan Garg. - Optimize performance and scalability, either for the web frontend or - the server. Zulip is already one of the faster webapps out there, + the server. Zulip is already one of the faster webapps out there, but there are a bunch of ideas for how to make it substantially - faster. This is likely a particularly challenging project to do + faster. This is likely a particularly challenging project to do well, since there are a lot of subtle interactions to understand. **Skill recommended**: Strong debugging, communication, and code - reading skills are most important here. JavaScript experience; some + reading skills are most important here. JavaScript experience; some Python/Django experience, some skill with CSS, ideally experience using the Chrome Timeline profiling tools (but you can pick this up - as you go) can be useful depending on what profiling shows. Our + as you go) can be useful depending on what profiling shows. Our [backend scalability design doc](../subsystems/performance.md) and the [production issue label][prod-label] (where performance/scalability issues tend to be filed) may be helpful - reading for the backend part of this. Expert: Steve Howell. + reading for the backend part of this. Expert: Steve Howell. [prod-label]: https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22area%3A+production%22 - Extract JavaScript logic modules from the Zulip webapp that we'd - like to be able to share with the Zulip webapp. This work can have + like to be able to share with the Zulip webapp. This work can have big benefits it terms of avoiding code duplication for complex - logic. We have prototyped for a few modules by migrating them to + logic. We have prototyped for a few modules by migrating them to `static/shared/`; this project will involve closely collaborating - with the mobile team to prioritize the modules to migrate. **Skills + with the mobile team to prioritize the modules to migrate. **Skills recommended**: JavaScript experience, careful refactoring, API design, React. @@ -338,12 +338,12 @@ CSS](https://github.com/zulip/zulip/). permissions (and implementing the enforcement logic), adding an OAuth system for presenting those controls to users, as well as making the /integrations page UI have buttons to create a bot, - rather than sending users to the administration page. **Skills + rather than sending users to the administration page. **Skills recommended**: Strong Python/Django; JavaScript, CSS, and design - sense helpful. Understanding of implementing OAuth providers, + sense helpful. Understanding of implementing OAuth providers, e.g. having built a prototype with [the Django OAuth toolkit](https://django-oauth-toolkit.readthedocs.io/en/latest/) - would be great to demonstrate as part of an application. The + would be great to demonstrate as part of an application. The [Zulip integration writing guide](../documentation/integrations.md) and [integration documentation](https://zulip.com/integrations/) @@ -351,69 +351,69 @@ CSS](https://github.com/zulip/zulip/). and [the integrations label on GitHub](https://github.com/zulip/zulip/labels/area%3A%20integrations) has a bunch of good starter issues to demonstrate your skills if - you're interested in this area. Expert: Eeshan Garg. + you're interested in this area. Expert: Eeshan Garg. - Extend Zulip's meta-integration that converts the Slack incoming webhook API to post messages into Zulip. Zulip has several dozen native integrations (https://zulip.com/integrations/), but Slack has a - ton more. We should build an interface to make all of Slack’s + ton more. We should build an interface to make all of Slack’s numerous third-party integrations work with Zulip as well, by basically building a Zulip incoming webhook interface that accepts the Slack API (if you just put in a Zulip server URL as your "Slack - server"). **Skills required**: Strong Python experience; experience - with the Slack API a plus. Work should include documenting the - system and advertising it. Expert: Tim Abbott. + server"). **Skills required**: Strong Python experience; experience + with the Slack API a plus. Work should include documenting the + system and advertising it. Expert: Tim Abbott. - Visual and user experience design work on the core Zulip web UI. We're particularly excited about students who are interested in making our CSS clean and readable as part of working on the UI. **Skills required**: Design, HTML and CSS skills; JavaScript and - illustration experience are helpful. A great application would + illustration experience are helpful. A great application would include PRs making small, clean improvements to the Zulip UI - (whether logged-in or logged-out pages). Expert: Aman Agrawal. + (whether logged-in or logged-out pages). Expert: Aman Agrawal. - Build support for outgoing webhooks and slash commands into Zulip to - improve its chat-ops capabilities. There's an existing + improve its chat-ops capabilities. There's an existing [pull request](https://github.com/zulip/zulip/pull/1393) with a lot of work on the outgoing webhooks piece of this feature that would need to be cleaned up and finished, and then we need to build support for slash commands, some example integrations, and a full set of - documentation and tests. Recommended reading includes Slack's + documentation and tests. Recommended reading includes Slack's documentation for these features, the Zulip message sending code path, and the linked pull request. **Skills required**: Strong - Python/Django skills. Expert: Steve Howell. + Python/Django skills. Expert: Steve Howell. - Build a system for managing Zulip bots entirely on the web. Right now, there's a somewhat cumbersome process where you download the API bindings, create a bot with an API key, put it in - configuration files, etc. We'd like to move to a model where a bot + configuration files, etc. We'd like to move to a model where a bot could easily progress from being a quick prototype to being a third-party extension to - being built into Zulip. And then for built-in bots, one should be able to click a few + being built into Zulip. And then for built-in bots, one should be able to click a few buttons of configuration on the web to set them up and include them in - your organization. We've developed a number of example bots + your organization. We've developed a number of example bots in the [`zulip_bots`](https://github.com/zulip/python-zulip-api/tree/main/zulip_bots) PyPI package. **Skills recommended**: Python and JavaScript/CSS, plus devops skills (Linux deployment, Docker, Puppet etc.) are all useful here. Experience writing tools using various popular APIs is helpful for - being able to make good choices. Expert: Steve Howell. + being able to make good choices. Expert: Steve Howell. - Improve the UI and visual design of the existing Zulip settings and - administration pages while fixing bugs and adding new settings. The + administration pages while fixing bugs and adding new settings. The pages have improved a great deal during recent GSoCs, but because - they have a ton of surface area, there's a lot to do. You can get a + they have a ton of surface area, there's a lot to do. You can get a great sense of what needs to be done by playing with the settings/administration/streams overlays in a development - environment. You can get experience working on the subsystem by + environment. You can get experience working on the subsystem by working on some of [our open settings/admin issues](https://github.com/zulip/zulip/labels/area%3A%20admin). **Skills recommended**: JavaScript, HTML, CSS, and an eye for visual - design. Expert: Shubham Dhama. + design. Expert: Shubham Dhama. - Build out the administration pages for Zulip to add new permissions and other settings more features that will make Zulip better for - larger organizations. We get constant requests for these kinds of - features from Zulip users. The Zulip bug tracker has plentiful open + larger organizations. We get constant requests for these kinds of + features from Zulip users. The Zulip bug tracker has plentiful open issues( [settings (admin/org)](https://github.com/zulip/zulip/labels/area%3A%20settings%20%28admin%2Forg%29), [settings @@ -422,32 +422,32 @@ CSS](https://github.com/zulip/zulip/). (user)](https://github.com/zulip/zulip/labels/area%3A%20settings%20%28user%29), [stream settings](https://github.com/zulip/zulip/labels/area%3A%20stream%20settings) - ) in the space of improving the Zulip administrative UI. Many are + ) in the space of improving the Zulip administrative UI. Many are little bite-size fixes in those pages, which are great for getting a feel for things, but a solid project here would be implementing 5-10 of the major missing features as full-stack development projects. The first part of this project will be refactoring the admin UI interfaces to require writing less semi-duplicate code for each feature. **Skills recommended**: A good mix of Python/Django and - HTML/CSS/JavaScript skill is ideal. The system for adding new + HTML/CSS/JavaScript skill is ideal. The system for adding new features is [well documented](../tutorials/new-feature-tutorial.md). Expert: Shubham Dhama. -- Write cool new features for Zulip. Play around with the software, +- Write cool new features for Zulip. Play around with the software, browse Zulip's issues for things that seem important, and suggest - something you’d like to build! A great project can combine 3-5 - significant features. Experts: Depends on the features! + something you’d like to build! A great project can combine 3-5 + significant features. Experts: Depends on the features! -- Work on Zulip's development and testing infrastructure. Zulip is a +- Work on Zulip's development and testing infrastructure. Zulip is a project that takes great pride in building great tools for development, but there's always more to do to make the experience - delightful. Significantly, a full 10% of Zulip's open issues are + delightful. Significantly, a full 10% of Zulip's open issues are ideas for how to improve the project, and are [in](https://github.com/zulip/zulip/labels/area%3A%20tooling) [these](https://github.com/zulip/zulip/labels/area%3A%20testing-coverage) [four](https://github.com/zulip/zulip/labels/area%3A%20testing-infrastructure) [labels](https://github.com/zulip/zulip/labels/area%3A%20provision) - for tooling improvements. A good place to start is + for tooling improvements. A good place to start is [backend test coverage](https://github.com/zulip/zulip/issues/7089). This is a somewhat unusual project, in that it would likely consist @@ -459,14 +459,14 @@ CSS](https://github.com/zulip/zulip/). A possible specific larger project in this space is working on adding [mypy](../testing/mypy.md) stubs - for Django in mypy to make our type checking more powerful. Read + for Django in mypy to make our type checking more powerful. Read [our mypy blog post](https://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/) - for details on how mypy works and is integrated into Zulip. This + for details on how mypy works and is integrated into Zulip. This specific project is ideal for a strong contributor interested in type systems. **Skills required**: Python, some DevOps, and a passion for checking - your work carefully. A strong applicant for this will have + your work carefully. A strong applicant for this will have completed several projects in these areas. Experts: Anders Kaseorg (provision, testing), Steve Howell (tooling, testing). @@ -476,12 +476,12 @@ CSS](https://github.com/zulip/zulip/). [python](https://github.com/zulip/python-zulip-api), [JavaScript](https://github.com/zulip/zulip-js), [PHP](https://packagist.org/packages/mrferos/zulip-php), and - [Haskell](https://hackage.haskell.org/package/hzulip)). The + [Haskell](https://hackage.haskell.org/package/hzulip)). The JavaScript bindings are a particularly high priority, since they are a project that hasn't gotten a lot of attention since being adopted from its original author, and we'd like to convert them to - Typescript. **Skills required**: Experience with the target - language and API design. Expert: Depends on language. + Typescript. **Skills required**: Experience with the target + language and API design. Expert: Depends on language. - Develop [**@zulipbot**](https://github.com/zulip/zulipbot), the GitHub workflow bot for the Zulip organization and its repositories. By utilizing the @@ -505,12 +505,12 @@ Experts: Greg Price, Chris Bobbe. The highest priority for the Zulip project overall is improving the Zulip React Native mobile app. -- Work on issues and polish for the app. You can see the open issues - [here](https://github.com/zulip/zulip-mobile/issues). There are a +- Work on issues and polish for the app. You can see the open issues + [here](https://github.com/zulip/zulip-mobile/issues). There are a few hundred open issues across the project, and likely many more problems that nobody has found yet; in the short term, it needs - polish, bug finding/squashing, and debugging. So browse the open - issues, play with the app, and get involved! Goals include parity + polish, bug finding/squashing, and debugging. So browse the open + issues, play with the app, and get involved! Goals include parity with the webapp (in terms of what you can do), parity with Slack (in terms of the visuals), world-class scrolling and narrowing performance, and a great codebase. @@ -518,15 +518,15 @@ Zulip React Native mobile app. A good project proposal here will bundle together a few focus areas that you want to make really great (e.g. the message composing, editing, and reacting experience), that you can work on over the -summer. We'd love to have multiple students working on this area if +summer. We'd love to have multiple students working on this area if we have enough strong applicants. **Skills required**: Strong programming experience, especially in reading the documentation of unfamiliar projects and communicating - what you learned. JavaScript and React experience are great pluses, + what you learned. JavaScript and React experience are great pluses, as are iOS or Android development/design experience is useful as - well. You'll need to learn React Native as part of getting - involved. There's tons of good online tutorials, courses, etc. + well. You'll need to learn React Native as part of getting + involved. There's tons of good online tutorials, courses, etc. ### Electron desktop app @@ -535,13 +535,13 @@ Code: Experts: Anders Kaseorg, Akash Nimare, Abhighyan Khaund. - Contribute to our [Electron-based desktop client - application](https://github.com/zulip/zulip-desktop). There's + application](https://github.com/zulip/zulip-desktop). There's plenty of feature/UI work to do, but focus areas for us include things to (1) improve the release process for the app, using - automated testing, TypeScript, etc. and (2) polish the UI. Browse + automated testing, TypeScript, etc. and (2) polish the UI. Browse the open issues and get involved! -**Skills required**: JavaScript experience, Electron experience. You +**Skills required**: JavaScript experience, Electron experience. You can learn electron as part of your application! Good preparation for desktop app projects is to (1) try out the app @@ -556,10 +556,10 @@ Experts: Aman Agrawal, Neil Pilgrim. - Work on Zulip Terminal, the official terminal client for Zulip. zulip-terminal is already a basic usable client, but it needs a lot - of work to approach the webapp's quality level. We would be happy - to accept multiple strong students to work on this project. Our + of work to approach the webapp's quality level. We would be happy + to accept multiple strong students to work on this project. Our goal for this summer is to improve its quality enough that we can - upgrade it from an alpha to an advertised feature. **Skills + upgrade it from an alpha to an advertised feature. **Skills required**: Python 3 development skills, good communication and project management skills, good at reading code and testing. @@ -569,7 +569,7 @@ Code: [zulip-archive](https://github.com/zulip/zulip-archive) Experts: Rein Zustand, Steve Howell - Work on zulip-archive, which provides a Google-indexable read-only - archive of Zulip conversations. The issue tracker for the project + archive of Zulip conversations. The issue tracker for the project has a great set of introductory/small projects; the overall goal is to make the project super convenient to use for our OSS communities. **Skills useful**: Python 3, reading feedback from users, CSS, @@ -584,16 +584,16 @@ two before the application deadline. That way, the whole developer community -- not just the mentors and administrators -- have a chance to give you feedback and help you improve your proposal. -Where should you publish your draft? We prefer Dropbox Paper or +Where should you publish your draft? We prefer Dropbox Paper or Google Docs, since those platforms allow people to look at the text without having to log in or download a particular app, and you can -update the draft as you improve your idea. In either case, you should +update the draft as you improve your idea. In either case, you should post the draft for feedback in chat.zulip.org. Rough is fine! The ideal first draft to get feedback from the community on should include primarily (1) links to your contributions to Zulip (or other projects) and (2) a paragraph or two explaining -what you plan to work on. Your friends are likely better able to help +what you plan to work on. Your friends are likely better able to help you improve the sections of your application explaining who you are, and this helps the community focus feedback on the areas you can most improve (e.g. either doing more contributions or adjusting the project diff --git a/docs/contributing/summer-with-zulip.md b/docs/contributing/summer-with-zulip.md index 3e01d8185e..ff0e592830 100644 --- a/docs/contributing/summer-with-zulip.md +++ b/docs/contributing/summer-with-zulip.md @@ -88,14 +88,14 @@ materials](https://developers.google.com/open-source/gsoc/resources/manual). - If you work in one a smaller Zulip project (e.g. `zulip-terminal`), follow the project on GitHub so you can - keep track of what's happening there. For folks working in + keep track of what's happening there. For folks working in `zulip/zulip`, doing that will send you too many notifications. So instead, we recommend that you join Zulip's GitHub teams that relate to your projects and/or interests, so that you see new - issues and PRs coming in that are relevant to your work. When we + issues and PRs coming in that are relevant to your work. When we label an issue or PR with one of our area labels, `zulipbot` will automatically mention the relevant teams for that area, - subscribing you to those issues/PR threads. You can browse the + subscribing you to those issues/PR threads. You can browse the area teams here: https://github.com/orgs/zulip/teams (You need to be a member of the Zulip organization to see them; ask Tim for an invite if needed). diff --git a/docs/contributing/version-control.md b/docs/contributing/version-control.md index 2d5fa63f7b..b37198016c 100644 --- a/docs/contributing/version-control.md +++ b/docs/contributing/version-control.md @@ -58,21 +58,21 @@ Other considerations: tests" commit on top of it. Zulip expects you to structure the commits in your pull requests to form -a clean history before we will merge them. It's best to write your +a clean history before we will merge them. It's best to write your commits following these guidelines in the first place, but if you don't, you can always fix your history using `git rebase -i` (more on that [here](../git/fixing-commits.md)). Never mix multiple changes together in a single commit, but it's great to include several related changes, each in their own commit, in a -single pull request. If you notice an issue that is only somewhat +single pull request. If you notice an issue that is only somewhat related to what you were working on, but you feel that it's too minor to create a dedicated pull request, feel free to append it as an additional commit in the pull request for your main project (that commit should have a clear explanation of the bug in its commit -message). This way, the bug gets fixed, but this independent change -is highlighted for reviewers. Or just create a dedicated pull request -for it. Whatever you do, don't squash unrelated changes together in a +message). This way, the bug gets fixed, but this independent change +is highlighted for reviewers. Or just create a dedicated pull request +for it. Whatever you do, don't squash unrelated changes together in a single commit; the reviewer will ask you to split the changes out into their own commits. @@ -100,8 +100,8 @@ The first line of the commit message is the **summary**. The summary: ### Good summaries: -Below is an example of a good commit summary line. It starts with the -prefix "provision:", using lowercase "**p**". Next, "Improve performance of +Below is an example of a good commit summary line. It starts with the +prefix "provision:", using lowercase "**p**". Next, "Improve performance of install npm." starts with a capital "**I**", uses imperative tense, and ends with a period. diff --git a/docs/contributing/zulipbot-usage.md b/docs/contributing/zulipbot-usage.md index d3df0d5657..85b80eaaa0 100644 --- a/docs/contributing/zulipbot-usage.md +++ b/docs/contributing/zulipbot-usage.md @@ -6,7 +6,7 @@ repositories in order to create a better workflow for Zulip contributors. Its purpose is to work around various limitations in GitHub's permissions and notifications systems to make it possible to have a -much more democractic workflow for our contributors. It allows anyone +much more democractic workflow for our contributors. It allows anyone to self-assign or label an issue, not just the core contributors trusted with full write access to the repository (which is the only model GitHub supports). @@ -59,14 +59,14 @@ issues and pull requests within your fields of expertise on the the Zulip server [area label teams](https://github.com/orgs/zulip/teams?utf8=✓&query=Server) (Note: this link only works for members of the Zulip organization; -we'll happily add you if you're interested). These teams correspond +we'll happily add you if you're interested). These teams correspond to the repository's [area labels](https://github.com/zulip/zulip/labels), although some teams are associated with multiple labels; for example, the **area: message-editing** and **area: message view** labels are both related to the [Server message view](https://github.com/orgs/zulip/teams/server-message-view) -team. Feel free to join as many area label teams as as you'd like! +team. Feel free to join as many area label teams as as you'd like! After your request to join an area label team is approved, you'll receive notifications for any issues labeled with the team's corresponding area diff --git a/docs/development/authentication.md b/docs/development/authentication.md index 28014302c9..86fa7733f0 100644 --- a/docs/development/authentication.md +++ b/docs/development/authentication.md @@ -14,9 +14,9 @@ through the real flow. The steps to do this are a variation of the steps discussed in the production documentation, including the comments in -`zproject/prod_settings_template.py`. The differences here are driven +`zproject/prod_settings_template.py`. The differences here are driven by the fact that `dev_settings.py` is in Git, so it is inconvenient -for local [settings configuration](../subsystems/settings.md). As a +for local [settings configuration](../subsystems/settings.md). As a result, in the development environment, we allow setting certain settings in the untracked file `zproject/dev-secrets.conf` (which is also serves as `/etc/zulip/zulip-secrets.conf`). @@ -28,7 +28,7 @@ methods supported by Zulip. Zulip's default EmailAuthBackend authenticates users by verifying control over their email address, and then allowing them to set a -password for their account. There are two development environment +password for their account. There are two development environment details worth understanding: - All of our authentication flows in the development environment have @@ -40,7 +40,7 @@ details worth understanding: `manage.py print_initial_password username@example.com`, that prints out **default** passwords for the development environment users. Note that if you change a user's password in the development - environment, those passwords will no longer work. It also prints + environment, those passwords will no longer work. It also prints out the user's **current** API key. ### Google @@ -51,9 +51,9 @@ console](https://console.developers.google.com) and navigate to "APIs to your dev environment. - Navigate to "APIs & services" > "Library", and find the "Identity - Toolkit API". Choose "Enable". + Toolkit API". Choose "Enable". -- Return to "Credentials", and select "Create credentials". Choose +- Return to "Credentials", and select "Create credentials". Choose "OAuth client ID", and follow prompts to create a consent screen, etc. For "Authorized redirect URIs", fill in `http://zulipdev.com:9991/complete/google/` . @@ -70,7 +70,7 @@ to your dev environment. Specify `http://zulipdev.com:9991/complete/github/` as the callback URL. - You should get a page with settings for your new application, - showing a client ID and a client secret. In `dev-secrets.conf`, set + showing a client ID and a client secret. In `dev-secrets.conf`, set `social_auth_github_key` to the client ID and `social_auth_github_secret` to the client secret. @@ -81,7 +81,7 @@ to your dev environment. Specify `http://zulipdev.com:9991/complete/gitlab` as the callback URL. - You should get a page containing the Application ID and Secret for - your new application. In `dev-secrets.conf`, enter the Application + your new application. In `dev-secrets.conf`, enter the Application ID as `social_auth_gitlab_key` and the Secret as `social_auth_gitlab_secret`. @@ -137,7 +137,7 @@ to your dev environment. Some OAuth providers (such as Facebook) require HTTPS on the callback URL they post back to, which isn't supported directly by the Zulip -development environment. If you run a +development environment. If you run a [remote Zulip development server](../development/remote.md), we have instructions for [an nginx reverse proxy with SSL](../development/remote.html#using-an-nginx-reverse-proxy) @@ -148,18 +148,18 @@ that you can use for your development efforts. Before Zulip 2.0, one of the more common classes of bug reports with Zulip's authentication was users having trouble getting [LDAP authentication](../production/authentication-methods.html#ldap-including-active-directory) -working. The root cause was because setting up a local LDAP server +working. The root cause was because setting up a local LDAP server for development was difficult, which meant most developers were unable to work on fixing even simple issues with it. We solved this problem for our unit tests long ago by using the -popular [fakeldap](https://github.com/zulip/fakeldap) library. And in +popular [fakeldap](https://github.com/zulip/fakeldap) library. And in 2018, we added convenient support for using fakeldap in the Zulip development environment as well, so that you can go through all the actual flows for LDAP configuration. - To enable fakeldap, set `FAKE_LDAP_MODE` in -`zproject/dev_settings.py` to one of the following options. For more +`zproject/dev_settings.py` to one of the following options. For more information on these modes, refer to [our production docs](../production/authentication-methods.html#ldap-including-active-directory): - `a`: If users' email addresses are in LDAP and used as username. @@ -185,7 +185,7 @@ contain data one might want to sync, including avatars and custom profile fields. We also have configured `AUTH_LDAP_USER_ATTR_MAP` in -`zproject/dev_settings.py` to sync several of those fields. For +`zproject/dev_settings.py` to sync several of those fields. For example: - Modes `a` and `b` will set the user's avatar on account creation and @@ -203,7 +203,7 @@ example: For our automated tests, we generally configure custom LDAP data for each individual test, because that generally means one can understand exactly what data is being used in the test without looking at other -resources. It also gives us more freedom to edit the development +resources. It also gives us more freedom to edit the development environment directory without worrying about tests. ## Two factor authentication @@ -213,13 +213,13 @@ Zulip uses [django-two-factor-auth][0] as a beta 2FA integration. To enable 2FA, set `TWO_FACTOR_AUTHENTICATION_ENABLED` in settings to `True`, then log in to Zulip and add an OTP device from the settings page. Once the device is added, password based authentication will ask -for a one-time-password. In the development environment, this +for a one-time-password. In the development environment, this one-time-password will be printed to the console when you try to -log in. Just copy-paste it into the form field to continue. +log in. Just copy-paste it into the form field to continue. Direct development logins don't prompt for 2FA one-time-passwords, so to test 2FA in development, make sure that you log in using a -password. You can get the passwords for the default test users using +password. You can get the passwords for the default test users using `./manage.py print_initial_password`. ## Password form implementation @@ -227,7 +227,7 @@ password. You can get the passwords for the default test users using By default, Zulip uses `autocomplete=off` for password fields where we enter the current password, and `autocomplete="new-password"` for password fields where we create a new account or change the existing -password. This prevents the browser from auto-filling the existing +password. This prevents the browser from auto-filling the existing password. Visit for more details. diff --git a/docs/development/remote.md b/docs/development/remote.md index b5751cf929..d99521c1af 100644 --- a/docs/development/remote.md +++ b/docs/development/remote.md @@ -32,14 +32,14 @@ networks. ## Setting up user accounts You will need a non-root user account with sudo privileges to set up -the Zulip development environment. If you have one already, continue +the Zulip development environment. If you have one already, continue to the next section. You can create a new user with sudo privileges by running the following commands as root: - You can create a `zulipdev` user by running the command `adduser zulipdev`. Run through the prompts to assign a password and -user information. (You can pick any username you like for this user +user information. (You can pick any username you like for this user account.) - You can add the user to the sudo group by running the command `usermod -aG sudo zulipdev`. @@ -93,7 +93,7 @@ developing on your laptop). To properly secure your remote development environment, you can [port forward](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding) using ssh instead of running the development environment on an exposed -interface. For example, if you're running Zulip on a remote server +interface. For example, if you're running Zulip on a remote server such as a DigitalOcean Droplet or an AWS EC2 instance, you can set up port-forwarding to access Zulip by running the following command in your terminal: @@ -282,7 +282,7 @@ Next, read the following to learn more about developing for Zulip: For some applications (e.g. developing an OAuth2 integration for Facebook), you may need your Zulip development to have a valid SSL -certificate. While `run-dev.py` doesn't support that, you can do this +certificate. While `run-dev.py` doesn't support that, you can do this with an `nginx` reverse proxy sitting in front of `run-dev.py.`. The following instructions assume you have a Zulip Droplet working and diff --git a/docs/development/setup-advanced.md b/docs/development/setup-advanced.md index 863e1fdfd6..9214436ba1 100644 --- a/docs/development/setup-advanced.md +++ b/docs/development/setup-advanced.md @@ -132,7 +132,7 @@ installation method described here. 1. [Visual Studio Code Remote - WSL](https://code.visualstudio.com/docs/remote/wsl) is recommended for editing files when developing with WSL. -1. You're done! You can pick up the [documentation on using the +1. You're done! You can pick up the [documentation on using the Zulip development environment](../development/setup-vagrant.html#step-4-developing), ignoring the parts about `vagrant` (since you're not using it). @@ -253,7 +253,7 @@ expected. 1. If you get the error `Hyper-V could not initialize memory`, this is likely because your system has insufficient free memory to start - the virtual machine. You can generally work around this error by + the virtual machine. You can generally work around this error by closing all other running programs and running `vagrant up --provider=hyperv` again. You can reopen the other programs after the provisioning is completed. If it still isn't @@ -276,7 +276,7 @@ these platforms reliably and easily, so we no longer maintain manual installation instructions for these platforms. If `tools/provision` doesn't yet support a newer release of Debian or -Ubuntu that you're using, we'd love to add support for it. It's +Ubuntu that you're using, we'd love to add support for it. It's likely only a few lines of changes to `tools/lib/provision.py` and `scripts/lib/setup-apt-repo` if you'd like to do it yourself and submit a pull request, or you can ask for help in @@ -291,7 +291,7 @@ that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal. This section documents how to set up the Zulip development environment -in a Cloud9 workspace. If you don't have an existing Cloud9 account, +in a Cloud9 workspace. If you don't have an existing Cloud9 account, you can sign up [here](https://aws.amazon.com/cloud9/). - Create a Workspace, and select the blank template. @@ -313,7 +313,7 @@ There's a NPM package, `zulip-cloud9`, that provides a wrapper around the Zulip development server for use in the Cloud9 environment. Note: `npm i -g zulip-cloud9` does not work in zulip's virtual -environment. Although by default, any packages installed in workspace +environment. Although by default, any packages installed in workspace folder (i.e. the top level folder) are added to `$PATH`. ```bash @@ -326,7 +326,7 @@ If you get error of the form `bash: cannot find command zulip-dev`, you need to start a new terminal. Your development server would be running at -`https://-.c9users.io` on port 8080. You +`https://-.c9users.io` on port 8080. You dont need to add `:8080` to your URL, since the Cloud9 proxy should automatically forward the connection. You might want to visit [zulip-cloud9 repo](https://github.com/cPhost/zulip-cloud9) and it's diff --git a/docs/development/setup-vagrant.md b/docs/development/setup-vagrant.md index d631410d9e..1a69f04410 100644 --- a/docs/development/setup-vagrant.md +++ b/docs/development/setup-vagrant.md @@ -65,7 +65,7 @@ to GitHub working on your machine. Follow our [Git guide][set-up-git] in order to install Git, set up a GitHub account, create an SSH key to access code on GitHub -efficiently, etc. Be sure to create an SSH key and add it to your +efficiently, etc. Be sure to create an SSH key and add it to your GitHub account using [these instructions](https://help.github.com/en/articles/generating-an-ssh-key). @@ -109,7 +109,7 @@ Adding user christie to group docker Done. ``` -You will need to reboot for this change to take effect. If it worked, +You will need to reboot for this change to take effect. If it worked, you will see `docker` in your list of groups: ```console @@ -124,7 +124,7 @@ If you had previously installed and removed an older version of Docker, an [Ubuntu bug](https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1844894) may prevent Docker from being automatically enabled and started after -installation. You can check using the following: +installation. You can check using the following: ```console $ systemctl status docker @@ -229,7 +229,7 @@ Now you are ready for [Step 2: Get Zulip code](#step-2-get-zulip-code). (Note: The **GitHub Desktop client** for Windows has a bug where it will automatically set `git config core.symlink false` on a repository if you use it to clone a repository, which will break the Zulip -development environment, because we use symbolic links. For that +development environment, because we use symbolic links. For that reason, we recommend avoiding using GitHub Desktop client to clone projects and to instead follow these instructions exactly.) @@ -298,24 +298,24 @@ does the following: - runs the `tools/provision` script inside the virtual machine/container, which downloads all required dependencies, sets up the python environment for the Zulip development server, and initializes a default test - database. We call this process "provisioning", and it is documented + database. We call this process "provisioning", and it is documented in some detail in our [dependencies documentation](../subsystems/dependencies.md). You will need an active internet connection during the entire process. (See [Specifying a proxy](#specifying-a-proxy) if you need a proxy to access the internet.) `vagrant up` can fail while -provisioning if your Internet connection is unreliable. To retry, you +provisioning if your Internet connection is unreliable. To retry, you can use `vagrant provision` (`vagrant up` will just boot the guest -without provisioning after the first time). Other common issues are +without provisioning after the first time). Other common issues are documented in the [Troubleshooting and common errors](#troubleshooting-and-common-errors) -section. If that doesn't help, please visit +section. If that doesn't help, please visit [#provision help](https://chat.zulip.org/#narrow/stream/21-provision-help) in the [Zulip development community server](../contributing/chat-zulip-org.md) for real-time help. On Windows, you will see the message -`The system cannot find the path specified.` several times. This is +`The system cannot find the path specified.` several times. This is normal and is not a problem. Once `vagrant up` has completed, connect to the development @@ -335,7 +335,7 @@ Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-54-generic x86_64) Congrats, you're now inside the Zulip development environment! You can confirm this by looking at the command prompt, which starts -with `(zulip-py3-venv)vagrant@`. If it just starts with `vagrant@`, your +with `(zulip-py3-venv)vagrant@`. If it just starts with `vagrant@`, your provisioning failed and you should look at the [troubleshooting section](#troubleshooting-and-common-errors). @@ -410,7 +410,7 @@ development environment on the virtual machine/container. Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your -browser. More details on how this works are available below. +browser. More details on how this works are available below. Zulip's whitespace rules are all enforced by linters, so be sure to run `tools/lint` often to make sure you're following our coding style @@ -439,10 +439,10 @@ guide][rtd-git-guide]. If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or running tests, this is -probably not because Zulip's `main` branch is broken. Instead, this +probably not because Zulip's `main` branch is broken. Instead, this is likely because we've recently merged changes to the development environment provisioning process that you need to apply to your -development environment. To update your environment, you'll need to +development environment. To update your environment, you'll need to re-provision your vagrant machine using `vagrant provision` (this just runs `tools/provision` from your Zulip checkout inside the Vagrant guest); this should complete in about a minute. @@ -461,16 +461,16 @@ help. If you ever want to recreate your development environment again from scratch (e.g. to test a change you've made to the provisioning process, or because you think something is broken), you can do so -using `vagrant destroy` and then `vagrant up`. This will usually be +using `vagrant destroy` and then `vagrant up`. This will usually be much faster than the original `vagrant up` since the base image is already cached on your machine (it takes about 5 minutes to run with a fast Internet connection). Any additional programs (e.g. Zsh, emacs, etc.) or configuration that you may have installed in the development environment will be lost -when you recreate it. To address this, you can create a script called +when you recreate it. To address this, you can create a script called `tools/custom_provision` in your Zulip Git checkout; and place any -extra setup commands there. Vagrant will run `tools/custom_provision` +extra setup commands there. Vagrant will run `tools/custom_provision` every time you run `vagrant provision` (or create a Vagrant guest via `vagrant up`). @@ -542,7 +542,7 @@ which can help you optimize your development workflow). ### Troubleshooting and common errors -Below you'll find a list of common errors and their solutions. Most +Below you'll find a list of common errors and their solutions. Most issues are resolved by just provisioning again (by running `./tools/provision` (from `/srv/zulip`) inside the Vagrant guest or equivalently `vagrant provision` from outside). @@ -560,7 +560,7 @@ When reporting your issue, please include the following information: - installation method (Vagrant or direct) - whether or not you are using a proxy - a copy of Zulip's `vagrant` provisioning logs, available in - `/var/log/provision.log` on your virtual machine. If you choose to + `/var/log/provision.log` on your virtual machine. If you choose to post just the error output, please include the **beginning of the error output**, not just the last few lines. @@ -569,7 +569,7 @@ usually helpful. #### Vagrant guest doesn't show (zulip-py3-venv) at start of prompt -This is caused by provisioning failing to complete successfully. You +This is caused by provisioning failing to complete successfully. You can see the errors in `var/log/provision.log`; it should end with something like this: @@ -609,7 +609,7 @@ vagrant halt vagrant up ``` -to reboot the guest. After this, you can do `vagrant provision` and +to reboot the guest. After this, you can do `vagrant provision` and `vagrant ssh`. #### ssl read error @@ -648,7 +648,7 @@ ssh_exchange_identification: Connection closed by remote host ``` It usually means the Vagrant guest is not running, which is usually -solved by rebooting the Vagrant guest via `vagrant halt; vagrant up`. See +solved by rebooting the Vagrant guest via `vagrant halt; vagrant up`. See [Vagrant was unable to communicate with the guest machine](#vagrant-was-unable-to-communicate-with-the-guest-machine) for more details. @@ -670,9 +670,9 @@ Then Vagrant was not able to create a symbolic link. First, if you are using Windows, **make sure you have run Git BASH (or Cygwin) as an administrator**. By default, only administrators can -create symbolic links on Windows. Additionally [UAC][windows-uac], a +create symbolic links on Windows. Additionally [UAC][windows-uac], a Windows feature intended to limit the impact of malware, can prevent -even administrator accounts from creating symlinks. [Turning off +even administrator accounts from creating symlinks. [Turning off UAC][disable-uac] will allow you to create symlinks. You can also try some of the solutions mentioned [here](https://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-home-premium-as-a-regular-user). @@ -682,7 +682,7 @@ some of the solutions mentioned If you ran Git BASH as administrator but you already had VirtualBox running, you might still get this error because VirtualBox is not -running as administrator. In that case: close the Zulip VM with +running as administrator. In that case: close the Zulip VM with `vagrant halt`; close any other VirtualBox VMs that may be running; exit VirtualBox; and try again with `vagrant up --provision` from a Git BASH running as administrator. @@ -728,7 +728,7 @@ The virtual machine needs to be shut down when you run this command. If you get an error message on Windows about lack of Windows Home support for Hyper-V when running `vagrant up`, the problem is that Windows is incorrectly attempting to use Hyper-V rather than -Virtualbox as the virtualization provider. You can fix this by +Virtualbox as the virtualization provider. You can fix this by explicitly passing the virtualbox provider to `vagrant up`: ```console @@ -783,7 +783,7 @@ the timeout ("config.vm.boot_timeout") value. ``` This has a range of possible causes, that usually amount to a bug in -Virtualbox or Vagrant. If you see this error, you usually can fix it +Virtualbox or Vagrant. If you see this error, you usually can fix it by rebooting the guest via `vagrant halt; vagrant up`. #### Vagrant up fails with subprocess.CalledProcessError @@ -793,14 +793,14 @@ The `vagrant up` command basically does the following: - Downloads an Ubuntu image and starts it using a Vagrant provider. - Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs `tools/provision`, which has a lot of subcommands that are - executed via Python's `subprocess` module. These errors mean that + executed via Python's `subprocess` module. These errors mean that one of those subcommands failed. To debug such errors, you can log in to the Vagrant guest machine by running `vagrant ssh`, which should present you with a standard shell -prompt. You can debug interactively by using e.g. +prompt. You can debug interactively by using e.g. `cd zulip && ./tools/provision`, and then running the individual -subcommands that failed. Once you've resolved the problem, you can +subcommands that failed. Once you've resolved the problem, you can rerun `tools/provision` to proceed; the provisioning system is designed to recover well from failures. @@ -817,11 +817,11 @@ Zulip development environment setup succeeded! If the `(zulip-py3-venv)` part is missing, this is because your installation failed the first time before the Zulip virtualenv was -created. You can fix this by just closing the shell and running +created. You can fix this by just closing the shell and running `vagrant ssh` again, or using `source /srv/zulip-py3-venv/bin/activate`. Finally, if you encounter any issues that weren't caused by your -Internet connection, please report them! We try hard to keep Zulip +Internet connection, please report them! We try hard to keep Zulip development environment provisioning free of bugs. ##### `pip install` fails during `vagrant up` on Ubuntu @@ -876,9 +876,9 @@ First, ensure that hardware virtualization support (VT-x or AMD-V) is enabled in your BIOS. If the error persists, you may have run into an incompatibility -between VirtualBox and Hyper-V on Windows. To disable Hyper-V, open +between VirtualBox and Hyper-V on Windows. To disable Hyper-V, open command prompt as administrator, run -`bcdedit /set hypervisorlaunchtype off`, and reboot. If you need to +`bcdedit /set hypervisorlaunchtype off`, and reboot. If you need to enable it later, run `bcdedit /deletevalue hypervisorlaunchtype`, and reboot. @@ -894,7 +894,7 @@ default: OSError: [Errno 26] Text file busy: 'baremetrics' This error is caused by a [bug](https://www.virtualbox.org/ticket/19004) in recent versions of -the VirtualBox Guest Additions for Linux on Windows hosts. You can +the VirtualBox Guest Additions for Linux on Windows hosts. You can check the running version of VirtualBox Guest Additions with this command: @@ -922,10 +922,10 @@ vagrant reload --provision ### Specifying an Ubuntu mirror Bringing up a development environment for the first time involves -downloading many packages from the Ubuntu archive. The Ubuntu cloud +downloading many packages from the Ubuntu archive. The Ubuntu cloud images use the global mirror `http://archive.ubuntu.com/ubuntu/` by default, but you may find that you can speed up the download by using -a local mirror closer to your location. To do this, create +a local mirror closer to your location. To do this, create `~/.zulip-vagrant-config` and add a line like this, replacing the URL as appropriate: @@ -962,7 +962,7 @@ NO_PROXY localhost,127.0.0.1,.example.com,.zulipdev.com ``` You'll want to **double-check** your work for mistakes (a common one -is using `https://` when your proxy expects `http://`). Invalid proxy +is using `https://` when your proxy expects `http://`). Invalid proxy configuration can cause confusing/weird exceptions; if you're using a proxy and get an error, the first thing you should investigate is whether you entered your proxy configuration correctly. @@ -978,7 +978,7 @@ then do a `vagrant reload`. ### Using a different port for Vagrant You can also change the port on the host machine that Vagrant uses by -adding to your `~/.zulip-vagrant-config` file. E.g. if you set: +adding to your `~/.zulip-vagrant-config` file. E.g. if you set: ```text HOST_PORT 9971 @@ -1009,7 +1009,7 @@ described here are ignored). Our default Vagrant settings allocate 2 cpus with 2GiB of memory for the guest, which is sufficient to run everything in the development -environment. If your host system has more CPUs, or you have enough +environment. If your host system has more CPUs, or you have enough RAM that you'd like to allocate more than 2GiB to the guest, you can improve performance of the Zulip development environment by allocating more resources. diff --git a/docs/development/test-install.md b/docs/development/test-install.md index 91e9cac4fe..9a4dee8008 100644 --- a/docs/development/test-install.md +++ b/docs/development/test-install.md @@ -12,7 +12,7 @@ the installation process in a clean environment each time. ## Configuring Using the test installer framework requires a Linux operating system; -it will not work on WSL, for instance. It requires at least 3G of +it will not work on WSL, for instance. It requires at least 3G of RAM, in order to accommodate the VMs and the steps which build the release assets. @@ -43,7 +43,7 @@ as the last step; for example, Next, unpack that file into a local directory; we will make any changes we want in our source checkout and copy them into this directory. The test installer needs the release directory to be named -`zulip-server`, so we rename it and move it appropriately. In the +`zulip-server`, so we rename it and move it appropriately. In the first line, you'll need to substitute the actual path that you got for the tarball, above: ```bash diff --git a/docs/development/using.md b/docs/development/using.md index e7b75590d3..5dc5582ecc 100644 --- a/docs/development/using.md +++ b/docs/development/using.md @@ -2,9 +2,9 @@ Using the development environment ================================= This page describes the basic edit/refresh workflows for working with -the Zulip development environment. Generally, the development +the Zulip development environment. Generally, the development environment will automatically update as soon as you save changes -using your editor. Details for work on the [server](#server), +using your editor. Details for work on the [server](#server), [webapp](#web), and [mobile apps](#mobile) are below. If you're working on authentication methods or need to use the [Zulip @@ -23,7 +23,7 @@ the development environment][authentication-dev-server]. if no changes are required. - After making changes, you'll often want to run the [linters](../testing/linters.md) and relevant [test - suites](../testing/testing.md). Consider using our [Git pre-commit + suites](../testing/testing.md). Consider using our [Git pre-commit hook](../git/zulip-tools.html#set-up-git-repo-script) to automatically lint whenever you make a commit. - All of our test suites are designed to support quickly testing just @@ -43,7 +43,7 @@ the development environment][authentication-dev-server]. - The main Django/Tornado server processes are run on top of Django's [manage.py runserver][django-runserver], which will automatically restart them when you save changes to Python code - they use. You can watch this happen in the `run-dev.py` console + they use. You can watch this happen in the `run-dev.py` console to make sure the backend has reloaded. - The Python queue workers will also automatically restart when you save changes, as long as they haven't crashed (which can happen if @@ -73,7 +73,7 @@ the development environment][authentication-dev-server]. something other than the login process. - You can test the login or registration process by clicking the links for the normal login page. -- Most changes will take effect automatically. Details: +- Most changes will take effect automatically. Details: - If you change CSS files, your changes will appear immediately via webpack hot module replacement. - If you change JavaScript code (`static/js`) or Handlebars diff --git a/docs/documentation/api.md b/docs/documentation/api.md index c63d75fbfe..302374e9d1 100644 --- a/docs/documentation/api.md +++ b/docs/documentation/api.md @@ -4,7 +4,7 @@ This document explains the system for documenting [Zulip's REST API](https://zulip.com/api/rest). Zulip's API documentation is an essential resource both for users and -for the developers of Zulip's mobile and terminal apps. Our vision is +for the developers of Zulip's mobile and terminal apps. Our vision is for the documentation to be sufficiently good that developers of Zulip's apps should never need to look at the server's implementation to answer questions about the API's semantics. @@ -15,13 +15,13 @@ and remains so as Zulip's API evolves. In particular, the top goal for this system is that all mistakes in verifiable content (i.e. not the English explanations) should cause -the Zulip test suite to fail. This is incredibly important, because +the Zulip test suite to fail. This is incredibly important, because once you notice one error in API documentation, you no longer trust it to be correct, which ends up wasting the time of its users. Since it's very difficult to not make little mistakes when writing any untested code, the only good solution to this is a way to test -the documentation. We found dozens of errors in the process of adding +the documentation. We found dozens of errors in the process of adding the validation Zulip has today. Our API documentation is defined by a few sets of files: @@ -32,12 +32,12 @@ Our API documentation is defined by a few sets of files: - The top-level templates live under `templates/zerver/api/*`, and are written using the Markdown framework that powers our [user docs](../documentation/user.md), with some special extensions for - rendering nice code blocks and example responses. We expect to + rendering nice code blocks and example responses. We expect to eventually remove most of these files where it is possible to fully generate the documentation from the OpenAPI files. - The text for the Python examples comes from a test suite for the Python API documentation (`zerver/openapi/python_examples.py`; run via - `tools/test-api`). The `generate_code_example` macro will magically + `tools/test-api`). The `generate_code_example` macro will magically read content from that test suite and render it as the code example. This structure ensures that Zulip's API documentation is robust to a wide range of possible typos and other bugs in the API @@ -52,7 +52,7 @@ Our API documentation is defined by a few sets of files: - We have an extensive set of tests designed to validate that the data in this file is correct, `zerver/tests/test_openapi.py` compares every endpoint's accepted parameters in `views` code with those - declared in `zulip.yaml`. And [backend test + declared in `zulip.yaml`. And [backend test suite](../testing/testing-with-django.md) and checks that every API response served during our extensive backend test suite matches one the declared OpenAPI schema for that endpoint. @@ -94,11 +94,11 @@ relevant feature in `/help/`. ### Usage examples We display usage examples in three languages: Python, JavaScript and -`curl`; we may add more in the future. Every endpoint should have +`curl`; we may add more in the future. Every endpoint should have Python and `curl` documentation; `JavaScript` is optional as we don't -consider that API library to be fully supported. The examples are +consider that API library to be fully supported. The examples are defined using a special Markdown extension -(`zerver/openapi/markdown_extension.py`). To use this extension, one +(`zerver/openapi/markdown_extension.py`). To use this extension, one writes a Markdown file block that looks something like this: ```md @@ -121,7 +121,7 @@ writes a Markdown file block that looks something like this: For the Python examples, you'll write the example in `zerver/openapi/python_examples.py`, and it'll be run and verified -automatically in Zulip's automated test suite. The code there will +automatically in Zulip's automated test suite. The code there will look something like this: ``` python @@ -139,14 +139,14 @@ def render_message(client: Client) -> None: ``` This is an actual Python function which will be run as part of the -`tools/test-api` test suite. The `validate_against_opanapi_schema` +`tools/test-api` test suite. The `validate_against_opanapi_schema` function will verify that the result of that request is as defined in the examples in `zerver/openapi/zulip.yaml`. To run as part of the testsuite, the `render_message` function needs to be called from `test_messages` (or one of the other functions at -the bottom of the file). The final function, `test_the_api`, is what -actually runs the tests. Tests with the `openapi_test_function` +the bottom of the file). The final function, `test_the_api`, is what +actually runs the tests. Tests with the `openapi_test_function` decorator that are not called will fail tests, as will new endpoints that are not covered by an `openapi_test_function`-decorated test. @@ -164,7 +164,7 @@ wherever that string appears in the API documentation. ### Parameters We have a separate Markdown extension to document the parameters that -an API endpoint supports. You'll see this in files like +an API endpoint supports. You'll see this in files like `templates/zerver/api/render-message.md` via the following Markdown directive (implemented in `zerver/lib/markdown/api_arguments_table_generator.py`): @@ -193,17 +193,17 @@ code: ## Step by step guide This section offers a step-by-step process for adding documentation -for a new API endpoint. It assumes you've read and understood the +for a new API endpoint. It assumes you've read and understood the above. 1. Start by adding [OpenAPI format](../documentation/openapi.md) - data to `zerver/openapi/zulip.yaml` for the endpoint. If you + data to `zerver/openapi/zulip.yaml` for the endpoint. If you copy-paste (which is helpful to get the indentation structure right), be sure to update all the content that you copied to correctly describe your endpoint! In order to do this, you need to figure out how the endpoint in - question works by reading the code! To understand how arguments + question works by reading the code! To understand how arguments are specified in Zulip backend endpoints, read our [REST API tutorial][rest-api-tutorial], paying special attention to the details of `REQ` and `has_request_variables`. @@ -221,7 +221,7 @@ above. mistakes in how arguments are declared. - `test-backend`: The full Zulip backend test suite will fail if any actual API responses generated by the tests don't match your - defined OpenAPI schema. Use `test-backend --rerun` for a fast + defined OpenAPI schema. Use `test-backend --rerun` for a fast edit/refresh cycle when debugging. [rest-api-tutorial]: ../tutorials/writing-views.html#writing-api-rest-endpoints @@ -229,7 +229,7 @@ above. 1. Add a function for the endpoint you'd like to document to `zerver/openapi/python_examples.py`, decorated with `@openapi_test_function`. `render_message` is a good example to - follow. There are generally two key pieces to your test: (1) doing + follow. There are generally two key pieces to your test: (1) doing an API query and (2) verifying its result has the expected format using `validate_against_openapi_schema`. @@ -248,11 +248,11 @@ above. function will be called when running `test-api`. 1. Capture the JSON response returned by the API call (the test - "fixture"). The easiest way to do this is add an appropriate print + "fixture"). The easiest way to do this is add an appropriate print statement (usually `json.dumps(result, indent=4, sort_keys=True)`), - and then run `tools/test-api`. You can also use + and then run `tools/test-api`. You can also use to format the JSON - fixtures. Add the fixture to the `example` subsection of the + fixtures. Add the fixture to the `example` subsection of the `responses` section for the endpoint in `zerver/openapi/zulip.yaml`. @@ -266,7 +266,7 @@ above. code example on our `/api` page. 1. Finally, write the Markdown file for your API endpoint under - `templates/zerver/api/`. This is usually pretty easy to template + `templates/zerver/api/`. This is usually pretty easy to template off existing endpoints; but refer to the system explanations above for details. @@ -275,7 +275,7 @@ above. 1. Test your endpoint, pretending to be a new user in a hurry, by visiting it via the links on `http://localhost:9991/api` (the API docs are rendered from the Markdown source files on page load, so - just reload to see an updated version as you edit). You should + just reload to see an updated version as you edit). You should make sure that copy-pasting the code in your examples works, and post an example of the output in the pull request. @@ -291,7 +291,7 @@ above. Given that our documentation is written in large part using the OpenAPI format, why maintain a custom Markdown system for displaying -it? There's several major benefits to this system: +it? There's several major benefits to this system: - It is extremely common for API documentation to become out of date as an API evolves; this automated testing system helps make it diff --git a/docs/documentation/integrations.md b/docs/documentation/integrations.md index d7bf9295d0..09371f0cde 100644 --- a/docs/documentation/integrations.md +++ b/docs/documentation/integrations.md @@ -20,7 +20,7 @@ Usually, this involves a few steps: - Make sure you've added your integration to `zerver/lib/integrations.py` in both the `WEBHOOK_INTEGRATIONS` section (or `INTEGRATIONS` if not a webhook), and the - `DOC_SCREENSHOT_CONFIG` sections. These registries configure your + `DOC_SCREENSHOT_CONFIG` sections. These registries configure your integration to appear on the `/integrations` page and make it possible to automatically generate the screenshot of a sample message (which is important for the screenshots to be updated as @@ -88,7 +88,7 @@ Here are a few common macros used to document Zulip's integrations: required to set up the URL and you can't use this macro, be sure to use the `{{ api_url }}` template variable, so that your integration documentation will provide the correct URL for whatever server it is - deployed on. If special configuration is required to set the `SITE` + deployed on. If special configuration is required to set the `SITE` variable, you should document that too. - `{!append-stream-name.md!}` macro - Recommends appending `&stream=stream_name` diff --git a/docs/documentation/openapi.md b/docs/documentation/openapi.md index 22a195a4ca..2d9cd18973 100644 --- a/docs/documentation/openapi.md +++ b/docs/documentation/openapi.md @@ -1,14 +1,14 @@ # OpenAPI configuration -[OpenAPI][openapi-spec] is a popular format for describing an API. An +[OpenAPI][openapi-spec] is a popular format for describing an API. An OpenAPI file can be used by various tools to generate documentation for the API or even basic client-side bindings for dozens of programming languages. -Zulip's API is described in `zerver/openapi/zulip.yaml`. Our aim is +Zulip's API is described in `zerver/openapi/zulip.yaml`. Our aim is for that file to fully describe every endpoint in the Zulip API, and for the Zulip test suite to fail should the API every change without a -corresponding adjustment to the documentation. In particular, +corresponding adjustment to the documentation. In particular, essentially all content in Zulip's [REST API documentation](../documentation/api.md) is generated from our OpenAPI file. @@ -116,7 +116,7 @@ The [Definitions Object](https://swagger.io/specification/#definitionsObject) contains schemas referenced by other objects. For example, `MessageResponse`, the response from the `/messages` endpoint, -contains three required parameters. Two are strings, and one is an +contains three required parameters. Two are strings, and one is an integer. ```yaml @@ -151,7 +151,7 @@ should be organized here: You can edit YAML files in any text editor. Indentation defines blocks, so whitespace is important (as it is in Python.) TAB -characters are not permitted. If your editor has an option to replace +characters are not permitted. If your editor has an option to replace tabs with spaces, this is helpful. You can also use the @@ -176,8 +176,8 @@ correct. headings. - A single `|` (pipe) character begins a multi-line description on the - next line. Single spaced lines (one newline at the end of each) are - joined. Use an extra blank line for a paragraph break. We prefer + next line. Single spaced lines (one newline at the end of each) are + joined. Use an extra blank line for a paragraph break. We prefer to use this format for all descriptions because it doesn't require extra effort to expand. diff --git a/docs/documentation/overview.md b/docs/documentation/overview.md index a35e0ee69b..e8cec5c15c 100644 --- a/docs/documentation/overview.md +++ b/docs/documentation/overview.md @@ -8,14 +8,14 @@ Zulip has three major documentation systems: - Core website documentation: Complete webpages for complex topics, written in HTML, JavaScript, and CSS (using the Django templating - system). These roughly correspond to the documentation someone - might look at when deciding whether to use Zulip. We don't expect + system). These roughly correspond to the documentation someone + might look at when deciding whether to use Zulip. We don't expect to ever have more than about 10 pages written using this system. - User-facing documentation: Our scalable system for documenting Zulip's huge collection of specific features without a lot of - overhead or duplicated code/syntax, written in Markdown. We have - several hundred pages written using this system. There are 3 + overhead or duplicated code/syntax, written in Markdown. We have + several hundred pages written using this system. There are 3 branches of this documentation: - User documentation (with a target audience of individual Zulip users), @@ -30,10 +30,10 @@ These three systems are documented in detail. What you are reading right now is part of the collection of documentation targeted at developers and people running their own -Zulip servers. These docs are written in +Zulip servers. These docs are written in [CommonMark Markdown](https://commonmark.org/) with a small bit of rST. We've chosen Markdown because it is -[easy to write](https://commonmark.org/help/). The source for Zulip's +[easy to write](https://commonmark.org/help/). The source for Zulip's developer documentation is at `docs/` in the Zulip Git repository, and they are served in production at [zulip.readthedocs.io](https://zulip.readthedocs.io/en/latest/). @@ -48,7 +48,7 @@ documentation using: ``` and then opening `http://127.0.0.1:9991/docs/index.html` in your -browser. The raw files are available at +browser. The raw files are available at `file:///path/to/zulip/docs/_build/html/index.html` in your browser (so you can also use e.g. `firefox docs/_build/html/index.html` from the root of your Zulip checkout). @@ -73,10 +73,10 @@ dependencies). Zulip has around 10 HTML documentation pages under `templates/zerver` for specific major topics, like the features list, client apps, -integrations, hotkeys, API bindings, etc. These documents often have +integrations, hotkeys, API bindings, etc. These documents often have somewhat complex HTML and JavaScript, without a great deal of common patterns between them other than inheriting from the `portico.html` -template. We generally avoid adding new pages to this collection +template. We generally avoid adding new pages to this collection unless there's a good reason, but we don't intend to migrate them, either, since this system gives us the flexibility to express these important elements of the product clearly. @@ -91,16 +91,16 @@ to do the things one does a lot in each type of documentation. ### General user documentation Zulip's [help center](https://zulip.com/help/) documentation is -designed to explain how the product works to end users. We aim for +designed to explain how the product works to end users. We aim for this to be clear, concise, correct, and readable to nontechnical -audiences where possible. See our guide on [writing user +audiences where possible. See our guide on [writing user documentation](user.md). ### Integrations documentation Zulip's [integrations documentation](https://zulip.com/integrations) is user-facing documentation explaining to end users how to setup each -of Zulip's more than 100 integrations. There is a detailed [guide on +of Zulip's more than 100 integrations. There is a detailed [guide on documenting integrations](integrations.md), including style guidelines to ensure that the documentation is high quality and consistent. @@ -111,7 +111,7 @@ guide](https://zulip.com/api/integrations-overview). Zulip's [API documentation](https://zulip.com/api/) is intended to make it easy for a technical user to write automation tools that interact -with Zulip. This documentation also serves as our main mechanism for +with Zulip. This documentation also serves as our main mechanism for Zulip server developers to communicate with client developers about how the Zulip API works. @@ -137,17 +137,17 @@ There's an exclude list for the link testing at this horrible path: which is relevant for flaky links. - The API docs are tested by `tools/test-api`, which does some basic -payload verification. Note that this test does not check for broken +payload verification. Note that this test does not check for broken links (those are checked by `test-help-documentation`). - `tools/test-help-documentation` checks `/help/`, `/api/`, `/integrations/`, and the core website ("portico") documentation for - broken links. Note that the "portico" documentation check has a + broken links. Note that the "portico" documentation check has a manually maintained whitelist of pages, so if you add a new page to this site, you will need to edit `PorticoDocumentationSpider` to add it. - `tools/test-backend test_docs.py` tests various internal details of - the variable substitution logic, as well as rendering. It's + the variable substitution logic, as well as rendering. It's essential when editing the documentation framework, but not something you'll usually need to interact with when editing documentation. diff --git a/docs/documentation/user.md b/docs/documentation/user.md index e9847afc21..b1ceff918a 100644 --- a/docs/documentation/user.md +++ b/docs/documentation/user.md @@ -40,7 +40,7 @@ are usually linked from `static/images/help/`. This means that you can contribute to the Zulip user documentation by just adding to or editing the collection of Markdown files under -`templates/zerver/help`. If you have the Zulip development environment +`templates/zerver/help`. If you have the Zulip development environment set up, you simply need to reload your browser on `http://localhost:9991/help/foo` to see the latest version of `foo.md` rendered. @@ -115,7 +115,7 @@ documentation. Images and screenshots should be included in user documentation only if they will help guide the user in how to do something (e.g. if the image will make it much clearer which element on the page the user -should interact with). For instance, an image of an element should +should interact with). For instance, an image of an element should not be included if the element the user needs to interact with is the only thing on the page, but images can be included to show the end result of an interaction with the UI. @@ -126,7 +126,7 @@ instructions for something simple look long and complicated. When taking screenshots, the image should never include the whole Zulip browser window in a screenshot; instead, it should only show -relevant parts of the app. In addition, the screenshot should always +relevant parts of the app. In addition, the screenshot should always come *after* the text that describes it, never before. Images are often a part of a numbered step and must be indented four @@ -235,7 +235,7 @@ should be formatted as a continuation of a numbered step. Our Markdown processor supports easily creating a tab switcher widget design to easily show the instructions for different [platforms](https://zulip.com/help/logging-out) in user docs, -languages in API docs, etc. To create a tab switcher, write: +languages in API docs, etc. To create a tab switcher, write: ```md {start_tabs} diff --git a/docs/git/cloning.md b/docs/git/cloning.md index ff27626020..50c875077a 100644 --- a/docs/git/cloning.md +++ b/docs/git/cloning.md @@ -32,10 +32,10 @@ Checking connectivity... done. ``` (The `--config pull.rebase` option configures Git so that `git pull` -will behave like `git pull --rebase` by default. Using +will behave like `git pull --rebase` by default. Using `git pull --rebase` to update your changes to resolve merge conflicts is expected by essentially all of open source projects, including -Zulip. You can also set that option after cloning using +Zulip. You can also set that option after cloning using `git config --add pull.rebase true`, or just be careful to always run `git pull --rebase`, never `git pull`). @@ -115,13 +115,13 @@ will run tests for new refs you push to GitHub and email you the outcome Running CI against your fork can help save both your and the Zulip maintainers time by making it easy to test a change fully before -submitting a pull request. We generally recommend a workflow where as +submitting a pull request. We generally recommend a workflow where as you make changes, you use a fast edit-refresh cycle running individual -tests locally until your changes work. But then once you've gotten +tests locally until your changes work. But then once you've gotten the tests you'd expect to be relevant to your changes working, push a branch to run the full test suite in GitHub Actions before -you create a pull request. While you wait for GitHub Actions jobs -to run, you can start working on your next task. When the tests finish, +you create a pull request. While you wait for GitHub Actions jobs +to run, you can start working on your next task. When the tests finish, you can create a pull request that you already know passes the tests. GitHub Actions will run all the jobs by default on your forked repository. diff --git a/docs/git/fixing-commits.md b/docs/git/fixing-commits.md index a2c789ddbf..c4f46d388b 100644 --- a/docs/git/fixing-commits.md +++ b/docs/git/fixing-commits.md @@ -32,4 +32,4 @@ Sometimes, you want to make one commit out of a bunch of commits. To do this, 2. Reorder the lines containing the commits and save ## Pushing commits after tidying them -1. `git push origin +my-feature-branch` (Note the `+` there and substitute your actual branch name.) +1. `git push origin +my-feature-branch` (Note the `+` there and substitute your actual branch name.) diff --git a/docs/git/overview.md b/docs/git/overview.md index b7a935149f..a3c4d8eed4 100644 --- a/docs/git/overview.md +++ b/docs/git/overview.md @@ -22,7 +22,7 @@ with these details in mind: We use this strategy in order to avoid the extra commits that appear when another branch is merged, that clutter the commit history (it's - popular with other large projects such as Django). This makes + popular with other large projects such as Django). This makes Zulip's commit history more readable, but a side effect is that many pull requests we merge will be reported by GitHub's UI as *closed* instead of *merged*, since GitHub has poor support for diff --git a/docs/git/pull-requests.md b/docs/git/pull-requests.md index 9a8edd2eb7..183275660b 100644 --- a/docs/git/pull-requests.md +++ b/docs/git/pull-requests.md @@ -20,7 +20,7 @@ requests early and often. This allows you to share your code to make it easier to get feedback and help with your changes. Prefix the titles of work-in-progress pull requests with **[WIP]**, which in our project means that you don't think your pull request is ready to be -merged (e.g. it might not work or pass tests). This sets expectations +merged (e.g. it might not work or pass tests). This sets expectations correctly for any feedback from other developers, and prevents your work from being merged before you're confident in it. @@ -30,7 +30,7 @@ work from being merged before you're confident in it. It is important to [work on a feature branch](using.html#work-on-a-feature-branch) when creating a pull -request. Your new pull request will be inextricably linked with your +request. Your new pull request will be inextricably linked with your branch while it is open, so you will need to reserve your branch only for changes related to your issue, and avoid introducing extraneous changes for other issues or from upstream. diff --git a/docs/git/setup.md b/docs/git/setup.md index 8fd57e572b..a0f198c90a 100644 --- a/docs/git/setup.md +++ b/docs/git/setup.md @@ -15,14 +15,14 @@ You'll also need a GitHub account, which you can sign up for [here][github-join]. We highly recommend you create an SSH key if you don't already have -one and [add it to your GitHub account][github-help-add-ssh-key]. If +one and [add it to your GitHub account][github-help-add-ssh-key]. If you don't, you'll have to type your GitHub username and password every time you interact with GitHub, which is usually several times a day. We also highly recommend the following: - [Configure Git][gitbook-config] with your name and email and - [aliases][gitbook-aliases] for commands you'll use often. We + [aliases][gitbook-aliases] for commands you'll use often. We recommend using your full name (not just your first name), since that's what we'll use to give credit to your work in places like the Zulip release notes. diff --git a/docs/git/using.md b/docs/git/using.md index 5e23b009e3..8c9f090714 100644 --- a/docs/git/using.md +++ b/docs/git/using.md @@ -324,7 +324,7 @@ messages][github-help-closing-issues] for details. Note in particular that GitHub's regular expressions for this feature are sloppy, so phrases like `Partially fixes #1234` will automatically -close the issue. Phrases like `Fixes part of #1234` are a good +close the issue. Phrases like `Fixes part of #1234` are a good alternative. Make as many commits as you need to to address the issue or implement your feature. diff --git a/docs/git/working-copies.md b/docs/git/working-copies.md index 5c42a03228..ba7b8826f8 100644 --- a/docs/git/working-copies.md +++ b/docs/git/working-copies.md @@ -3,8 +3,8 @@ When you work on Zulip code, there are three copies of the Zulip Git repository that you are generally concerned with: -- The `upstream` remote. This is the [official Zulip - repository](https://github.com/zulip/zulip) on GitHub. You probably +- The `upstream` remote. This is the [official Zulip + repository](https://github.com/zulip/zulip) on GitHub. You probably don't have write access to this repository. - The **origin** remote: Your personal remote repository on GitHub. You'll use this to share your code and create [pull requests](../git/pull-requests.md). @@ -44,7 +44,7 @@ working copies: - `git push`: This pushes code from your local repository to one of the remotes. - `git remote`: This helps you configure short names for remotes. - `git pull`: This pulls code, but by default creates a merge commit - (which you definitely don't want). However, if you've followed our + (which you definitely don't want). However, if you've followed our [cloning documentation](../git/cloning.md), this will do `git pull --rebase` instead, which is the only mode you'll want to use when working on Zulip. diff --git a/docs/git/zulip-tools.md b/docs/git/zulip-tools.md index 95023dc5cc..2e5a2172f3 100644 --- a/docs/git/zulip-tools.md +++ b/docs/git/zulip-tools.md @@ -5,7 +5,7 @@ time when working with Git on the Zulip project. ## Set up Git repo script -**Extremely useful**. In the `tools` directory of +**Extremely useful**. In the `tools` directory of [zulip/zulip][github-zulip-zulip] you'll find a bash script `setup-git-repo`. This script installs a pre-commit hook, which will run each time you `git commit` to automatically run @@ -41,7 +41,7 @@ described above in that it does not create a branch for the pull request checkout. **This tool checks for uncommitted changes, but it will move the - current branch using `git reset --hard`. Use with caution.** + current branch using `git reset --hard`. Use with caution.** First, make sure you are working in a branch you want to move (in this example, we'll use the local `main` branch). Then run the script @@ -118,10 +118,10 @@ HEAD is now at 5a1e982 tools: Update clean-branches to clean review branches. ## Push to a pull request `tools/push-to-pull-request` is primarily useful for maintainers who -are merging other users' commits into a Zulip repository. After doing +are merging other users' commits into a Zulip repository. After doing `reset-to-pull-request` or `fetch-pull-request` and making some changes, you can push a branch back to a pull request with e.g. -`tools/push-to-pull-request 1234`. This is useful for a few things: +`tools/push-to-pull-request 1234`. This is useful for a few things: - Getting CI to run and enabling you to use the GitHub "Merge" buttons to merge a PR after you make some corrections to a PR, without @@ -139,7 +139,7 @@ next batch of changes. Note that in order to do this you need permission to do such a push, which GitHub offers by default to users with write access to the -repository. For multiple developers collaborating on a PR, you can +repository. For multiple developers collaborating on a PR, you can achieve this by granting other users permission to write to your fork. ## Delete unimportant branches diff --git a/docs/overview/architecture-overview.md b/docs/overview/architecture-overview.md index 73b727cc42..4886aed66d 100644 --- a/docs/overview/architecture-overview.md +++ b/docs/overview/architecture-overview.md @@ -4,7 +4,7 @@ Zulip architectural overview Key codebases ------------- -The main Zulip codebase is at . It +The main Zulip codebase is at . It contains the Zulip backend (written in Python 3.x and Django), the webapp (written in JavaScript and TypeScript) and our library of incoming webhook [integrations](https://zulip.com/integrations) @@ -43,7 +43,7 @@ Usage assumptions and concepts Zulip is a real-time team chat application meant to provide a great experience for a wide range of organizations, from companies to volunteer projects to groups of friends, ranging in size from a small -team to 10,000s of users. It has [hundreds of +team to 10,000s of users. It has [hundreds of features](https://zulip.com/features) both larger and small, and supports dedicated apps for iOS, Android, Linux, Windows, and macOS, all modern web browsers, several cross-protocol chat clients, and @@ -51,8 +51,8 @@ numerous dedicated [Zulip API](https://zulip.com/api) clients (e.g. bots). A server can host multiple Zulip *realms* (organizations), each on its -own (sub)domain. While most installations host only one organization, some -such as zulip.com host thousands. Each organization is a private +own (sub)domain. While most installations host only one organization, some +such as zulip.com host thousands. Each organization is a private chamber with its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip realms. The administrators of an organization have a great deal of control over @@ -69,7 +69,7 @@ Components ### Django and Tornado Zulip is primarily implemented in the -[Django](https://www.djangoproject.com/) Python web framework. We +[Django](https://www.djangoproject.com/) Python web framework. We also make use of [Tornado](https://www.tornadoweb.org) for the real-time push system. @@ -86,10 +86,10 @@ connection from every running client. For this reason, it's responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in Tornado because we don't want to delay delivery to thousands of other connections (as this would make Zulip -very much not real-time). For instance, we avoid doing cache or +very much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those blocking requests carry a very high performance penalty for a single-threaded, -asynchronous server system. (In principle, we could do non-blocking +asynchronous server system. (In principle, we could do non-blocking requests to those services, but the Django-based database libraries we use in most of our codebase using don't support that, and in any case, our architecture doesn't require Tornado to do that). @@ -142,7 +142,7 @@ from outside. `uWSGI` via `unix:/home/zulip/deployments/uwsgi-socket`. - By default (i.e. if `LOCAL_UPLOADS_DIR` is set), nginx will serve user-uploaded content like avatars, custom emoji, and uploaded - files. However, one can configure Zulip to store these in a cloud + files. However, one can configure Zulip to store these in a cloud storage service like Amazon S3 instead. Note that we do not use `nginx` in the development environment, opting @@ -168,7 +168,7 @@ memcached is used to cache database model objects. `zerver/lib/cache.py` and `zerver/lib/cache_helpers.py` manage putting things into memcached, and invalidating the cache when values change. The memcached configuration is in -`puppet/zulip/files/memcached.conf`. See our +`puppet/zulip/files/memcached.conf`. See our [caching guide](../subsystems/caching.md) to learn how this works in detail. @@ -191,7 +191,7 @@ replace RabbitMQ with Redis, with some loss of functionality). The answer is likely yes, but it wouldn't improve Zulip. Operationally, our current setup is likely easier to develop and run -in production than a pure Redis system would be. Meanwhile, the +in production than a pure Redis system would be. Meanwhile, the perceived benefit for using Redis is usually to reduce memory consumption by running fewer services, and no such benefit would materialize: @@ -221,7 +221,7 @@ and the Tornado push system. Two simple wrappers around `pika` (the Python RabbitMQ client) are in `zulip/zerver/lib/queue.py`. There's an asynchronous client for use in -Tornado and a more general client for use elsewhere. Most of the +Tornado and a more general client for use elsewhere. Most of the processes started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle them; they are defined in `zerver/worker/queue_processors.py`. @@ -243,7 +243,7 @@ list of stopwords used by a PostgreSQL extension. In a development environment, configuration of that PostgreSQL extension is handled by `tools/postgresql-init-dev-db` (invoked by -`tools/provision`). That file also manages setting up the +`tools/provision`). That file also manages setting up the development PostgreSQL user. `tools/provision` also invokes `tools/rebuild-dev-database` @@ -267,13 +267,13 @@ component of the Zulip server (e.g. ## Glossary This section gives names for some of the elements in the Zulip UI used -in Zulip development conversations. In general, our goal is to +in Zulip development conversations. In general, our goal is to minimize the set of terminology listed here by giving elements self-explanatory names. - **bankruptcy**: When a user has been off Zulip for several days and has hundreds of unread messages, they are prompted for whether - they want to mark all their unread messages as read. This is + they want to mark all their unread messages as read. This is called "declaring bankruptcy" (in reference to the concept in finance). diff --git a/docs/overview/changelog.md b/docs/overview/changelog.md index bea92e7e00..2ab3bacb1b 100644 --- a/docs/overview/changelog.md +++ b/docs/overview/changelog.md @@ -97,7 +97,7 @@ up-to-date list of raw changes. composing messages, and is now the default view. The previous default view, "All messages", is still available, and the default view can now be configured via "Display settings". -- Completed API documentation for Zulip's real-time events system. It +- Completed API documentation for Zulip's real-time events system. It is now possible to write a decent Zulip client with minimal interaction with the Zulip server development team. - Added new organization settings: wildcard mention policy. @@ -108,7 +108,7 @@ up-to-date list of raw changes. - This release contains more than 30 independent changes to the [Zulip API](https://zulip.com/api/changelog), largely to support new features or make the API (and thus its documentation) clearer and - easier for clients to implement. Other new API features support + easier for clients to implement. Other new API features support better error handling for the mobile and terminal apps. - The frontend internationalization library was switched from i18next to FormatJS. @@ -123,7 +123,7 @@ up-to-date list of raw changes. - Changed the Tornado service to use 127.0.0.1:9800 instead of 127.0.0.1:9993 as its default network address, to simplify support - for multiple Tornado processes. Since Tornado only listens on + for multiple Tornado processes. Since Tornado only listens on localhost, this change should have no visible effect unless another service is using port 9800. - Zulip's top-level puppet classes have been renamed, largely from @@ -142,14 +142,14 @@ up-to-date list of raw changes. one no longer needs to use `supervisorctl` directly for these tasks. - As this is a major release, we recommend [carefully updating the inline documentation in your - `/etc/zulip/settings.py`][update-settings-docs]. Notably, we rewrote the + `/etc/zulip/settings.py`][update-settings-docs]. Notably, we rewrote the template to be better organized and more readable in this release. - The webapp will now display a warning in the UI if the Zulip server has not been upgraded in more than 18 months. template to be better organized and more readable. - The next time users log in to Zulip with their password after upgrading to this release, they will be logged out of all active - browser sessions (i.e. the web and desktop apps). This is a side + browser sessions (i.e. the web and desktop apps). This is a side effect of improved security settings (increasing the minimum entropy used when salting passwords from 71 bits to 128 bits). - We've removed the partial Thumbor integration from Zulip. The @@ -308,7 +308,7 @@ up-to-date list of raw changes. ### 3.3 -- December 1, 2020 - Guest users should not be allowed to post to streams marked “Only - organization full members can post.” This flaw has existed since + organization full members can post.” This flaw has existed since the feature was added in Zulip Server 3.0. - Permit outgoing mail from postfix; this resolves a bug introduced in Zulip Server 3.2 which prevented Zulip from sending outgoing mail if @@ -347,7 +347,7 @@ up-to-date list of raw changes. ### 3.1 -- July 30, 2020 -- Removed unused `short_name` field from the User model. This field +- Removed unused `short_name` field from the User model. This field had no purpose and could leak the local part of email addresses when email address visibility was restricted. - Fixed a bug where loading spinners would sometimes not be displayed. @@ -361,7 +361,7 @@ up-to-date list of raw changes. releases to fail. - Added a Thinkst Canary integration (and renamed the old one, which was actually an integration for canarytokens.org). -- Reformatted the frontend codebase using prettier. This change was +- Reformatted the frontend codebase using prettier. This change was included in this maintenance release to ensure backporting patches from `main` remains easy. @@ -369,18 +369,18 @@ up-to-date list of raw changes. #### Highlights -- Added support for Ubuntu 20.04 Focal. This release drops support +- Added support for Ubuntu 20.04 Focal. This release drops support for Ubuntu 16.04 Xenial and Debian 9 Stretch. - Redesigned the top navbar/search area to be much cleaner and show useful data like subscriber counts and stream descriptions in default views. - Added a new "recent topics" widget, which lets one browse recent - and ongoing conversations at a glance. We expect this widget to + and ongoing conversations at a glance. We expect this widget to replace "All messages" as the default view in Zulip in the next major release. - Redesigned "Notification settings" to have an intuitive table format and display any individual streams with non-default settings. -- Added support for moving topics between streams. This was by far +- Added support for moving topics between streams. This was by far Zulip's most-requested feature. - Added automatic theme detection using prefers-color-scheme. - Added support for GitLab and Sign in with Apple authentication. @@ -415,8 +415,8 @@ up-to-date list of raw changes. those in past major releases. - Previous versions had a rare bug that made it possible to create two user accounts with the same email address, preventing either from - logging in. A migration in this release adds a database constraint - that will fix this bug. The new migration will fail if any such + logging in. A migration in this release adds a database constraint + that will fix this bug. The new migration will fail if any such duplicate accounts already exist; you can check whether this will happen be running the following in a [management shell][manage-shell]: ```python @@ -425,23 +425,23 @@ up-to-date list of raw changes. .values('realm_id', 'email_lower').annotate(Count('id')).filter(id__count__gte=2) ``` If the command returns any accounts, you need to address the - duplicate accounts before upgrading. Zulip Cloud only had two + duplicate accounts before upgrading. Zulip Cloud only had two accounts affected by this bug, so we expect the vast majority of installations will have none. - This release switches Zulip to install PostgreSQL 12 from the upstream PostgreSQL repository by default, rather than using the default - PostgreSQL version included with the operating system. Existing Zulip + PostgreSQL version included with the operating system. Existing Zulip installations will continue to work with PostgreSQL 10; this detail is - configured in `/etc/zulip/zulip.conf`. We have no concrete plans to + configured in `/etc/zulip/zulip.conf`. We have no concrete plans to start requiring PostgreSQL 12, though we do expect it to improve - performance. Installations that would like to upgrade can follow + performance. Installations that would like to upgrade can follow [our new PostgreSQL upgrade guide][postgresql-upgrade]. - The format of the `JWT_AUTH_KEYS` setting has changed to include an [algorithms](https://pyjwt.readthedocs.io/en/latest/algorithms.html) list: `{"subdomain": "key"}` becomes `{"subdomain": {"key": "key", "algorithms": ["HS256"]}}`. - Added a new organization owner permission above the previous - organization administrator. All existing organization + organization administrator. All existing organization administrators are automatically converted into organization owners. Certain sensitive administrative settings are now only editable by organization owners. @@ -502,7 +502,7 @@ up-to-date list of raw changes. - Improved left sidebar popovers to clearly identify administrative actions. - Rewrote substantial parts of the Zulip installer to be more robust. - Replaced the chevron menu indicators in sidebars with vertical ellipses. -- Removed the right sidebar "Group PMs" widget. It's functionality is +- Removed the right sidebar "Group PMs" widget. It's functionality is available in the left sidebar "Private messages" widget. - Removed the Google Hangouts integration, due to Google's support for it being discontinued. @@ -550,7 +550,7 @@ up-to-date list of raw changes. - Added webhook support for AnsibleTower 9.x.y. - Essentially rewrote our API documentation using the OpenAPI format, with extensive validation to ensure its accuracy as we modify the API. -- Removed New User Bot and Feedback Bot. Messages they had sent are +- Removed New User Bot and Feedback Bot. Messages they had sent are migrated to have been sent by Notification Bot. - Removed the "pointer" message ID from Zulip, a legacy concept dating to 2012 that predated tracking unread messages in Zulip and has @@ -582,7 +582,7 @@ up-to-date list of raw changes. - Replaced our CasperJS frontend integration test system with Puppeteer. - Extracted the typeahead and Markdown libraries for reuse in the mobile apps. -- Removed the legacy websockets-based system for sending messages. This +- Removed the legacy websockets-based system for sending messages. This system was always a hack, was only ever used for one endpoint, and did not provide a measureable latency benefit over HTTP/2. @@ -626,7 +626,7 @@ Administrators of servers originally installed with Zulip 1.9 or older should audit for unexpected [organization administrators][audit-org-admin] following this upgrade, as it is possible CVE-2020-14215 caused a user to incorrectly join as an -organization administrator in the past. See the release blog post for +organization administrator in the past. See the release blog post for details. [audit-org-admin]: https://zulip.com/help/change-a-users-role @@ -639,7 +639,7 @@ details. an exception restoring backups on fresh Zulip servers that had been generated on systems that had been upgraded from older Zulip releases. - Removed fetching GitHub contributor data from static asset build - process. This makes `upgrade-zulip-from-git` much more reliable. + process. This makes `upgrade-zulip-from-git` much more reliable. - Updated translation data from Transifex. - Support for Ubuntu 16.04 Xenial and Debian 9 Stretch is now deprecated. @@ -648,7 +648,7 @@ details. - CVE-2020-9444: Prevent reverse tabnapping attacks. - CVE-2020-9445: Remove unused and insecure modal_link feature. - CVE-2020-10935: Fix XSS vulnerability in local link rewriting. -- Blocked access from Zulip Desktop versions below 5.0.0. This +- Blocked access from Zulip Desktop versions below 5.0.0. This behavior can be adjusted by editing `DESKTOP_*_VERSION` in `/home/zulip/deployments/current/version.py`. - Restructured server initialization to simplify initialization of @@ -710,56 +710,56 @@ details. #### Highlights -- Added support for Debian buster. Removed support for EOL Ubuntu Trusty. +- Added support for Debian buster. Removed support for EOL Ubuntu Trusty. - Added support for SAML authentication. - Removed our dependency on `tsearch_extras`, making it possible to run a production Zulip server against any PostgreSQL database (including those where one cannot install extensions, like Amazon RDS). - Significantly improved the email->Zulip gateway, and added [nice - setup documentation](../production/email-gateway.md). It now + setup documentation](../production/email-gateway.md). It now should be possible to subscribe a Zulip stream to an email list and have a good experience. - Added an option for hiding access to user email addresses from - other users. While counterproductive for most corporate + other users. While counterproductive for most corporate communities, for open source projects and other volunteer organizations, this can be a critical anti-spam feature. - Added a new setting controlling which unread messages are counted in the favicon, title, and desktop app. - Support for showing inline previews of linked webpages has moved - from alpha to beta. See the upgrade notes below for some changes in + from alpha to beta. See the upgrade notes below for some changes in how it is configured. - Added support for importing an organization from Mattermost (similar - to existing Slack/HipChat/Gitter import tools). Slack import now + to existing Slack/HipChat/Gitter import tools). Slack import now supports importing data only included in corporate exports, including private messages and shared channels. - Added Markdown support and typeahead for mentioning topics. - Email notifications have been completely redesigned with a minimal, readable style inspired by GitHub's email notifications. - We merged significant preparatory work for supporting RHEL/CentOS in - production. We're now interested in beta testers for this feature. + production. We're now interested in beta testers for this feature. - Reorganized Zulip's documentation for sysadmins, and added [new documentation](../production/upgrade-or-modify.html#modifying-zulip) on maintaining a fork of Zulip. - Added new `streams:public` search operator that searches the public history of all streams in the organization (even before you joined). - Added support for sending email and mobile push notifications for - wildcard mentions (@all and @everyone). Previously, they only + wildcard mentions (@all and @everyone). Previously, they only triggered desktop notifications; now, that's configurable. #### Upgrade notes for 2.1.0 - The defaults for Zulip's now beta inline URL preview setting have changed. Previously, the server-level `INLINE_URL_EMBED_PREVIEW` setting was -disabled, and organization-level setting was enabled. Now, the +disabled, and organization-level setting was enabled. Now, the server-level setting is enabled by default, and the organization-level -setting is disabled. As a result, organization administrators can -configure this feature entirely in the UI. However, servers that had +setting is disabled. As a result, organization administrators can +configure this feature entirely in the UI. However, servers that had previously [enabled previews of linked websites](https://zulip.com/help/allow-image-link-previews) will lose the setting and need to re-enable it. - We rewrote the Google authentication backend to use the `python-social-auth` system we use for other third-party - authentication systems. For this release, the old variable names + authentication systems. For this release, the old variable names still work, but users should update the following setting names in their configuration as we will desupport the old names in a future release: @@ -775,20 +775,20 @@ lose the setting and need to re-enable it. `AUTH_LDAP_REVERSE_EMAIL_SEARCH` and `AUTH_LDAP_USERNAME_ATTR`. See the [LDAP configuration instructions](../production/authentication-methods.html#ldap-including-active-directory) - for details. You can use the usual `manage.py query_ldap` method to + for details. You can use the usual `manage.py query_ldap` method to verify whether your configuration is working correctly. - The Zulip web and desktop apps have been converted to directly count all unread messages, replacing an old system that just counted the - (recent) messages fully fetched by the webapp. This one-time + (recent) messages fully fetched by the webapp. This one-time transition may cause some users to notice old messages that were - sent months or years ago "just became unread". What actually + sent months or years ago "just became unread". What actually happened is the user never read these messages, and the Zulip webapp - was not displaying that. Generally, the fix is for users to simply + was not displaying that. Generally, the fix is for users to simply mark those messages as read as usual. - Previous versions of Zulip's installer would generate the secrets - `local_database_password` and `initial_password_salt`. These + `local_database_password` and `initial_password_salt`. These secrets don't do anything, as they only modify behavior of a Zulip - development environment. We recommend deleting those lines from + development environment. We recommend deleting those lines from `/etc/zulip/zulip-secrets.conf` when you upgrade to avoid confusion. - This release has a particularly expensive database migration, changing the `UserMessage.id` field from an `int` to a `bigint` to @@ -800,7 +800,7 @@ lose the setting and need to re-enable it. number of rows for your server with `UserMessage.objects.count()`. We expect that most Zulip servers can happily just use the normal - upgrade process with a few minutes of downtime. Zulip servers with + upgrade process with a few minutes of downtime. Zulip servers with over 1M messages may want to first upgrade to [this commit](https://github.com/zulip/zulip/commit/b008515d63841e1c0a16ad868d3d67be3bfc20ca) using `upgrade-zulip-from-git`, following the instructions to avoid @@ -894,12 +894,12 @@ lose the setting and need to re-enable it. - Fixed numbered list handling of blank lines between blocks. - Fixed performance issues that made users soft-deactivated for over a year unable to return to the app. -- Fixed missing -X GET/POST parameters in API docs curl examples. The +- Fixed missing -X GET/POST parameters in API docs curl examples. The API documentation for curl examples is now automatically generated with automated tests for the examples to prevent future similar bugs. - Fixed multi-line /me messages only working for the sender. - Fixed password strength meter not updating on paste. -- Fixed numerous errors and omissions in the API documentation. Added +- Fixed numerous errors and omissions in the API documentation. Added a test suite comparing the API documentation to the implementation. - Fixed copy/paste of blocks of messages in Firefox. - Fixed problems with exception reporting when memcached is down. @@ -1020,7 +1020,7 @@ lose the setting and need to re-enable it. status" messages. - Added a built-in /poll slash command for lightweight polls. - Added experimental support for using Zoom as the video chat - provider. We now support Jitsi, Google Hangouts, and Zoom. + provider. We now support Jitsi, Google Hangouts, and Zoom. - Added support for branding the top-left corner of the logged in app with an organization's logo. - Zulip's "Guest users" feature is no longer experimental. @@ -1028,7 +1028,7 @@ lose the setting and need to re-enable it. Our HipChat and Slack import tools are now well-tested with millions of messages, 10,000s of users, and 100,000s of uploaded files. - Added a built-in tool for backups and restoration. -- Deprecated support for Ubuntu Trusty. Zulip 2.0.x will continue to +- Deprecated support for Ubuntu Trusty. Zulip 2.0.x will continue to support Ubuntu Trusty, but Zulip 2.1.0 will remove support for installing on Trusty. @@ -1044,7 +1044,7 @@ and is enabled by default in that case. To disable it, set #### Full feature changelog - Added support for CentOS 7 in the development environment - provisioning process. This is an important step towards production + provisioning process. This is an important step towards production CentOS/RHEL 7 support. - Added a new invitation workflow with reusable links. - Added a new Azure Active Directory authentication integration. @@ -1078,7 +1078,7 @@ and is enabled by default in that case. To disable it, set - Added "silent mentions" syntax (`@_**Tim Abbott**`), which show visually, but don't trigger a notification to the target user. - Added support for using lightbox in compose preview. -- Changes in date no longer force a repeated recipient bar. This +- Changes in date no longer force a repeated recipient bar. This fixes a common source of confusion for new users. - Suppressed notifications when quoting a message mentioning yourself. - Message editing now has the compose widgets for emoji, video calls, etc. @@ -1116,7 +1116,7 @@ and is enabled by default in that case. To disable it, set This release migrates Zulip off a deprecated Google+ API (necessary for Google authentication to continue working past March 7), and -contains a few bug fixes for the installer and Slack import. It has +contains a few bug fixes for the installer and Slack import. It has minimal changes for existing servers not using Google authentication. - Updated the Google auth integration to stop using a deprecated and @@ -1145,10 +1145,10 @@ Zulip installations; it has minimal changes for existing servers. #### Highlights - Support for Ubuntu bionic and Debian stretch (our first non-Ubuntu - platform!). We expect to deprecate support for installing a new + platform!). We expect to deprecate support for installing a new Zulip server on Ubuntu Trusty in the coming months, in preparation for Trusty’s end-of-life in April 2019. -- New data import tools for HipChat and Gitter. The Slack importer +- New data import tools for HipChat and Gitter. The Slack importer is now out of beta. - Zulip Python process startup time is about 30% faster; this effort resulted in upstream contributions to fix significant performance @@ -1172,18 +1172,18 @@ Zulip installations; it has minimal changes for existing servers. Netlify, and Zabbix; Zulip now has over 100 native integrations (in addition to hundreds more available via Zapier and IFTTT). - New translations for Ukrainian, Portuguese, Indonesian, Dutch, and - Finnish. Zulip now has complete or nearly-complete translations + Finnish. Zulip now has complete or nearly-complete translations for German, Spanish, French, Portuguese, Russian, Ukrainian, - Czech, Finnish, and Turkish. Partial translations for Chinese, + Czech, Finnish, and Turkish. Partial translations for Chinese, Dutch, Korean, Polish, Japanese, and Indonesian cover the majority of the total strings in the project. #### Upgrade notes for 1.9.0 - Zulip 1.9 contains a significant database migration that can take - several minutes to run. The upgrade process automatically minimizes + several minutes to run. The upgrade process automatically minimizes disruption by running this migration first, before beginning the - user-facing downtime. However, if you'd like to watch the downtime + user-facing downtime. However, if you'd like to watch the downtime phase of the upgrade closely, we recommend [running them first manually](../production/expensive-migrations.md) and as well as the usual trick of @@ -1198,7 +1198,7 @@ Zulip installations; it has minimal changes for existing servers. - Added the new `SOCIAL_AUTH_SUBDOMAIN` setting, which all servers using both GitHub authentication and hosting multiple Zulip organizations should set (see [the docs for details](../production/multiple-organizations.html#authentication)). -- Added automatic thumbnailing of images, powered by thumbor. The new +- Added automatic thumbnailing of images, powered by thumbor. The new THUMBOR_URL setting controls this feature; it is disabled by default in this release, because the mobile apps don't support it yet. - Added documentation on alternative production deployment options. @@ -1257,13 +1257,13 @@ Zulip installations; it has minimal changes for existing servers. user accounts on a server. - Emails and several other onboarding strings are now tagged for translation. -- Optimized the performance of importing Zulip by about 30%. This +- Optimized the performance of importing Zulip by about 30%. This significantly decreases the load spike when restarting a Zulip server. - Optimized the performance of development provisioning; a no-op provision now completes in about 3.5s. - Migrated our static asset pipeline to webpack. - Our steady work on codebase quality and our automated test suite - continues. Backend test coverage is now an incredible 98%. + continues. Backend test coverage is now an incredible 98%. ## Zulip 1.8.x series @@ -1298,9 +1298,9 @@ Zulip installations; it has minimal changes for existing servers. **Security and privacy:** - Several important security fixes since 1.7.0, which were released already in 1.7.1 and 1.7.2. -- The security model for private streams has changed. Now +- The security model for private streams has changed. Now organization administrators can remove users, edit descriptions, and - rename private streams they are not subscribed to. See Zulip's + rename private streams they are not subscribed to. See Zulip's security model documentation for details. - On Xenial, the local uploads backend now does the same security checks that the S3 backend did before serving files to users. @@ -1378,7 +1378,7 @@ Zulip installations; it has minimal changes for existing servers. - Improved handling of browser undo in compose. - Improved saved drafts system to garbage-collect old drafts and sort by last modification, not creation. -- Removed the legacy "Zulip labs" autoscroll_forever setting. It was +- Removed the legacy "Zulip labs" autoscroll_forever setting. It was enabled mostly by accident. - Removed some long-deprecated Markdown syntax for mentions. - Added support for clicking on a mention to see a user's profile. @@ -1412,7 +1412,7 @@ Zulip installations; it has minimal changes for existing servers. - Fixed numerous issues in the "stream settings" UI. - Fixed numerous subtle bugs with the stream creation UI. - Changes the URL scheme for stream narrows to encode the stream ID, - so that they can be robust to streams being renamed. The change is + so that they can be robust to streams being renamed. The change is backwards-compatible; existing narrow URLs still work. @@ -1482,13 +1482,13 @@ This major release has no special upgrade notes. ### 1.7.2 -- 2018-04-12 This is a security release, with a handful of cherry-picked changes -since 1.7.1. All Zulip server admins are encouraged to upgrade +since 1.7.1. All Zulip server admins are encouraged to upgrade promptly. - CVE-2018-9986: Fix XSS issues with frontend Markdown processor. - CVE-2018-9987: Fix XSS issue with muting notifications. - CVE-2018-9990: Fix XSS issue with stream names in topic typeahead. -- CVE-2018-9999: Fix XSS issue with user uploads. The fix for this +- CVE-2018-9999: Fix XSS issue with user uploads. The fix for this adds a Content-Security-Policy for the `LOCAL_UPLOADS_DIR` storage backend for user-uploaded files. @@ -1498,7 +1498,7 @@ reporting CVE-2018-9986 and CVE-2018-9990. ### 1.7.1 -- 2017-11-21 This is a security release, with a handful of cherry-picked changes -since 1.7.0. All Zulip server admins are encouraged to upgrade +since 1.7.0. All Zulip server admins are encouraged to upgrade promptly. This release includes fixes for the upgrade process, so server admins @@ -1508,7 +1508,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1. the invitation system allowed an authorized user of one realm to create an account on any other realm. - The Korean translation is now complete, a huge advance from almost - nothing in 1.7.0. The French translation is now nearly complete, + nothing in 1.7.0. The French translation is now nearly complete, and several other languages have smaller updates. - The installer now sets LC_ALL to a known locale, working around an issue where some dependencies fail to install in some locales. @@ -1554,8 +1554,8 @@ running a version from before 1.7 should upgrade directly to 1.7.1. **Backend and scaling** -- Zulip now runs exclusively on Python 3. This is the culmination of - an 18-month migration effort. We are very excited about this! +- Zulip now runs exclusively on Python 3. This is the culmination of + an 18-month migration effort. We are very excited about this! - We’ve added an automatic "soft deactivation" process, which dramatically improves performance for organizations with a large number of inactive users, without any impact on those users’ @@ -1569,9 +1569,9 @@ running a version from before 1.7 should upgrade directly to 1.7.1. #### Upgrade notes for 1.7.0 - Zulip 1.7 contains some significant database migrations that can - take several minutes to run. The upgrade process automatically + take several minutes to run. The upgrade process automatically minimizes disruption by running these first, before beginning the - user-facing downtime. However, if you'd like to watch the downtime + user-facing downtime. However, if you'd like to watch the downtime phase of the upgrade closely, we recommend [running them first manually](../production/expensive-migrations.md) and as well as the usual trick of @@ -1583,12 +1583,12 @@ running a version from before 1.7 should upgrade directly to 1.7.1. to have its own subdomain. This change should have no effect for the vast majority of Zulip - servers that only have one organization. If you manage a server + servers that only have one organization. If you manage a server that hosts multiple organizations, you'll want to read [our guide on multiple organizations](../production/multiple-organizations.md). - We simplified the configuration for our password strength checker to - be much more intuitive. If you were using the + be much more intuitive. If you were using the `PASSWORD_MIN_ZXCVBN_QUALITY` setting, [it has been replaced](https://github.com/zulip/zulip/commit/a116303604e362796afa54b5d923ea5312b2ea23) by the more intuitive `PASSWORD_MIN_GUESSES`. @@ -1769,10 +1769,10 @@ Zulip apps. - Added support for users deleting realm emoji they themselves uploaded. - Added support for organization administrators deleting messages. - Extended data available to mobile apps to cover the entire API. -- Redesigned bots UI. Now can change owners and reactivate bots. +- Redesigned bots UI. Now can change owners and reactivate bots. - Redesigned the visuals of code blocks to be prettier. - Changed right sidebar presence UI to only show recently active users - in large organizations. This has a huge performance benefit. + in large organizations. This has a huge performance benefit. - Changed color for private messages to look better. - Converted realm emoji to be uploaded, not links, for better robustness. - Switched the default password hasher for new passwords to Argon2. @@ -1814,13 +1814,13 @@ Zulip apps. - Trailing whitespace is now stripped in code blocks, avoiding unnecessary scrollbars. - Most API payloads now refer to users primarily by user ID, with - email available for backwards-compatibility. In the future, we may + email available for backwards-compatibility. In the future, we may remove email support. -- Cleaned up Zulip's supervisord configuration. A side effect is the +- Cleaned up Zulip's supervisord configuration. A side effect is the names of the log files have changed for all the queue workers. - Refactored various endpoints to use a single code path for security hardening. -- Removed support for the `MANDRILL_CLIENT` setting. It hadn't been +- Removed support for the `MANDRILL_CLIENT` setting. It hadn't been used in years. - Changed `NOREPLY_EMAIL_ADDRESS` setting to `Name ` format. @@ -1888,7 +1888,7 @@ Zulip apps. - Added a configuration option to disable websockets. - Added support for removing one's own Zulip account. - Added support for realm admins which auth backends are supported. -- Added new organization type concept. This will be used to control +- Added new organization type concept. This will be used to control whether Zulip is optimized around protecting user privacy vs. administrative control. - Added #**streamName** syntax for linking to a stream. @@ -1967,9 +1967,9 @@ Zulip apps. ### 1.4.0 - 2016-08-25 - Migrated Zulip's python dependencies to be installed via a virtualenv, - instead of the via apt. This is a major change to how Zulip + instead of the via apt. This is a major change to how Zulip is installed that we expect will simplify upgrades in the future. -- Fixed unnecessary loading of zxcvbn password strength checker. This +- Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached network transfer for loading Zulip. - Added support for using Ubuntu Xenial in production. @@ -1986,7 +1986,7 @@ Zulip apps. - Added support for pinning streams to the top of the left sidebar. - Added search box for filtering user list when creating a new stream. - Added realm setting to disable message editing. -- Added realm setting to time-limit message editing. Default is 10m. +- Added realm setting to time-limit message editing. Default is 10m. - Added realm setting for default language. - Added year to timestamps in message interstitials for old messages. - Added GitHub authentication (and integrated python-social-auth, so it's @@ -1999,7 +1999,7 @@ Zulip apps. - Added automatic configuration of PostgreSQL/memcached settings based on the server's available RAM. - Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo. -- Added preliminary support for Python 3. All of Zulip's test suites now +- Added preliminary support for Python 3. All of Zulip's test suites now pass using Python 3.4. - Added support for `Name ` format when inviting users. - Added numerous special-purpose settings options. @@ -2010,10 +2010,10 @@ Zulip apps. - Improved error messages for various empty narrows. - Improved missed message emails to better support directly replying. - Increased backend test coverage of Python code to 85.5%. -- Increased mypy static type coverage of Python code to 95%. Also +- Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properly handle Unicode. - Fixed major i18n-related frontend performance regression on - /#subscriptions page. Saves several seconds of load time with 1k + /#subscriptions page. Saves several seconds of load time with 1k streams. - Fixed Jinja2 migration bug when trying to register an email that already has an account. diff --git a/docs/overview/directory-structure.md b/docs/overview/directory-structure.md index d04f1706fb..611293ff28 100644 --- a/docs/overview/directory-structure.md +++ b/docs/overview/directory-structure.md @@ -19,12 +19,12 @@ paths will be familiar to Django developers. - `zerver/models.py` Main [Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/) - file. Defines Zulip's database tables. + file. Defines Zulip's database tables. - `zerver/lib/*.py` Most library code. - `zerver/lib/actions.py` Most code doing writes to user-facing - database tables lives here. In particular, we have a policy that + database tables lives here. In particular, we have a policy that all code calling `send_event` to trigger [pushing data to clients](../subsystems/events-system.md) must live here. @@ -121,7 +121,7 @@ Django context (i.e. with database access). the development environment setup process. - `tools/ci/` Subdirectory of `tools/` for things only used to - set up and run our tests in CI. Actual test suites should + set up and run our tests in CI. Actual test suites should go in `tools/`. --------------------------------------------------------- @@ -155,11 +155,11 @@ This is used to deploy essentially all configuration in production. - `analytics` Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). -- `corporate` The old Zulip.com website. Not included in production +- `corporate` The old Zulip.com website. Not included in production distribution. - `zilencer` Primarily used to hold management commands that aren't - used in production. Not included in production distribution. + used in production. Not included in production distribution. ----------------------------------------------------------------------- @@ -177,7 +177,7 @@ This is used to deploy essentially all configuration in production. ### Documentation -- `docs/` Source for this documentation. +- `docs/` Source for this documentation. -------------------------------------------------------------- diff --git a/docs/overview/release-lifecycle.md b/docs/overview/release-lifecycle.md index 811b7ae003..b9dbb12266 100644 --- a/docs/overview/release-lifecycle.md +++ b/docs/overview/release-lifecycle.md @@ -31,7 +31,7 @@ server repository][zulip-server]. - Zulip Server **stable releases**, such as Zulip 4.5. Organizations self-hosting Zulip primarily use stable releases. - The numbering scheme is simple: the first digit indicates the major - release series (which we'll refer to as "4.x"). (Before Zulip 3.0, + release series (which we'll refer to as "4.x"). (Before Zulip 3.0, Zulip versions had another digit, e.g. 1.9.2 was a bug fix release in the Zulip 1.9.x major release series). - [New major releases][blog-major-releases], like Zulip 4.0, are @@ -45,7 +45,7 @@ server repository][zulip-server]. you use the latest version of the upgrade code. Starting with Zulip 4.0, the Zulip webapp displays the current server -version in the gear menu. With older releases, the server version is +version in the gear menu. With older releases, the server version is available [via the API](https://zulip.com/api/get-server-settings). This ReadTheDocs documentation has a widget in the lower-left corner @@ -68,11 +68,11 @@ in a stable release. userbase. - [chat.zulip.org][chat-zulip-org], the bleeding-edge server for the Zulip development community, is upgraded to `main` several times - every week. We also often "test deploy" changes not yet in `main` + every week. We also often "test deploy" changes not yet in `main` to chat.zulip.org to facilitate design feedback. - We maintain Git branches with names like `4.x` containing backported commits from `main` that we plan to include in the next maintenance - release. Self hosters can [upgrade][upgrade-from-git] to these + release. Self hosters can [upgrade][upgrade-from-git] to these stable release branches to get bug fixes staged for the next stable release (which is very useful when you reported a bug whose fix we choose to backport). We support these branches as though they were a @@ -89,10 +89,10 @@ for self-hosters, has no regressions, and that the [Zulip upgrade process](../production/upgrade-or-modify.md) Just Works. The Zulip server and clients apps are all carefully engineered to -ensure compatibility with old versions. In particular: +ensure compatibility with old versions. In particular: - The Zulip mobile and desktop apps maintain backwards-compatibility - code to support any Zulip server since 2.1.0. (They may also work + code to support any Zulip server since 2.1.0. (They may also work with older versions, with a degraded experience). - Zulip maintains an [API changelog](https://zulip.com/api/changelog) detailing all changes to the API to make it easy for client @@ -147,7 +147,7 @@ You can adjust the deadline for your installation by setting e.g. For platforms we support, like Debian and Ubuntu, Zulip aims to support all versions of the upstream operating systems that are fully -supported by the vendor. We document how to correctly [upgrade the +supported by the vendor. We document how to correctly [upgrade the operating system][os-upgrade] for a Zulip server, including how to correctly chain upgrades when the latest Zulip release no longer supports your OS. @@ -177,7 +177,7 @@ aggregate, just as important as the big things. Most resolved issues do not have any of these priority labels. We welcome participation from our user community in influencing the -Zulip roadmap. If a bug or missing feature is causing significant +Zulip roadmap. If a bug or missing feature is causing significant pain for you, we'd love to hear from you, either in [chat.zulip.org](../contributing/chat-zulip-org.md) or on the relevant GitHub issue. Please an include an explanation of your use case: such diff --git a/docs/production/authentication-methods.md b/docs/production/authentication-methods.md index 1c72a2e1cf..382a37de69 100644 --- a/docs/production/authentication-methods.md +++ b/docs/production/authentication-methods.md @@ -1,6 +1,6 @@ # Authentication methods -Zulip supports a wide variety of authentication methods. Some of them +Zulip supports a wide variety of authentication methods. Some of them require configuration to set up. To configure or disable authentication methods on your Zulip server, @@ -19,7 +19,7 @@ Users set a password with the Zulip server, and log in with their email and password. When first setting up your Zulip server, this method must be used for -creating the initial realm and user. You can disable it after that. +creating the initial realm and user. You can disable it after that. ## Plug-and-play SSO (Google, GitHub, GitLab) @@ -32,7 +32,7 @@ authentication providers: - Microsoft Azure Active Directory, with `AzureADAuthBackend` Each of these requires one to a handful of lines of configuration in -`settings.py`, as well as a secret in `zulip-secrets.conf`. Details +`settings.py`, as well as a secret in `zulip-secrets.conf`. Details are documented in your `settings.py`. ```eval_rst @@ -46,7 +46,7 @@ optionally using LDAP as an authentication mechanism. In either configuration, you will need to do the following: 1. These instructions assume you have an installed Zulip server and - are logged into a shell there. You can have created an + are logged into a shell there. You can have created an organization already using EmailAuthBackend, or plan to create the organization using LDAP authentication. @@ -55,17 +55,17 @@ In either configuration, you will need to do the following: integration, part 1: Connecting to the LDAP server". - If a password is required, put it in `/etc/zulip/zulip-secrets.conf` by setting - `auth_ldap_bind_password`. For example: + `auth_ldap_bind_password`. For example: `auth_ldap_bind_password = abcd1234`. 1. Decide how you want to map the information in your LDAP database to - users' account data in Zulip. For each Zulip user, two closely + users' account data in Zulip. For each Zulip user, two closely related concepts are: - - their **email address**. Zulip needs this in order to send, for + - their **email address**. Zulip needs this in order to send, for example, a notification when they're offline and another user sends a PM. - - their **Zulip username**. This means the name the user types into the - Zulip login form. You might choose for this to be the user's + - their **Zulip username**. This means the name the user types into the + Zulip login form. You might choose for this to be the user's email address (`sam@example.com`), or look like a traditional "username" (`sam`), or be something else entirely, depending on your environment. @@ -74,7 +74,7 @@ In either configuration, you will need to do the following: in your LDAP database. 1. Tell Zulip how to map the user information in your LDAP database to - the form it needs for authentication. There are three supported + the form it needs for authentication. There are three supported ways to set up the username and/or email mapping: (A) Using email addresses as Zulip usernames, if LDAP has each @@ -111,7 +111,7 @@ You can quickly test whether your configuration works by running: /home/zulip/deployments/current/manage.py query_ldap username ``` -from the root of your Zulip installation. If your configuration is +from the root of your Zulip installation. If your configuration is working, that will output the full name for your user (and that user's email address, if it isn't the same as the "Zulip username"). @@ -138,14 +138,14 @@ of the following configurations: **If you are using LDAP for authentication**: you will need to enable the `zproject.backends.ZulipLDAPAuthBackend` auth backend, in -`AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`. After doing so +`AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`. After doing so (and as always [restarting the Zulip server](settings.md) to ensure your settings changes take effect), you should be able to log in to Zulip by entering your email address and LDAP password on the Zulip login form. You may also want to configure Zulip's settings for [inviting new -users](https://zulip.com/help/invite-new-users). If LDAP is the +users](https://zulip.com/help/invite-new-users). If LDAP is the only enabled authentication method, the main use case for Zulip's invitation feature is selecting the initial streams for invited users (invited users will still need to use their LDAP password to create an @@ -196,10 +196,10 @@ or `jpegPhoto` attribute in LDAP) by configuring the `avatar` key in Starting with Zulip 2.0, Zulip supports syncing [custom profile fields][custom-profile-fields] from LDAP / Active -Directory. To configure this, you first need to +Directory. To configure this, you first need to [configure some custom profile fields][custom-profile-fields] for your -Zulip organization. Then, define a mapping from the fields you'd like -to sync from LDAP to the corresponding LDAP attributes. For example, +Zulip organization. Then, define a mapping from the fields you'd like +to sync from LDAP to the corresponding LDAP attributes. For example, if you have a custom profile field `LinkedIn Profile` and the corresponding LDAP attribute is `linkedinProfile` then you just need to add `'custom_profile_field__linkedin_profile': 'linkedinProfile'` @@ -210,22 +210,22 @@ to the `AUTH_LDAP_USER_ATTR_MAP`. #### Automatically deactivating users with Active Directory Starting with Zulip 2.0, Zulip supports synchronizing the -disabled/deactivated status of users from Active Directory. You can +disabled/deactivated status of users from Active Directory. You can configure this by uncommenting the sample line `"userAccountControl": "userAccountControl",` in -`AUTH_LDAP_USER_ATTR_MAP` (and restarting the Zulip server). Zulip +`AUTH_LDAP_USER_ATTR_MAP` (and restarting the Zulip server). Zulip will then treat users that are disabled via the "Disable Account" feature in Active Directory as deactivated in Zulip. Users disabled in active directory will be immediately unable to log in to Zulip, since Zulip queries the LDAP/Active Directory server on -every login attempt. The user will be fully deactivated the next time +every login attempt. The user will be fully deactivated the next time your `manage.py sync_ldap_user_data` cron job runs (at which point they will be forcefully logged out from all active browser sessions, appear as deactivated in the Zulip UI, etc.). This feature works by checking for the `ACCOUNTDISABLE` flag on the -`userAccountControl` field in Active Directory. See +`userAccountControl` field in Active Directory. See [this handy resource](https://jackstromberg.com/2013/01/useraccountcontrol-attributeflag-values/) for details on the various `userAccountControl` flags. @@ -234,11 +234,11 @@ for details on the various `userAccountControl` flags. Starting with Zulip 2.0, Zulip supports automatically deactivating users if they are not found by the `AUTH_LDAP_USER_SEARCH` query (either because the user is no longer in LDAP/Active Directory, or -because the user no longer matches the query). This feature is +because the user no longer matches the query). This feature is enabled by default if LDAP is the only authentication backend -configured on the Zulip server. Otherwise, you can enable this +configured on the Zulip server. Otherwise, you can enable this feature by setting `LDAP_DEACTIVATE_NON_MATCHING_USERS` to `True` in -`/etc/zulip/settings.py`. Nonmatching users will be fully deactivated +`/etc/zulip/settings.py`. Nonmatching users will be fully deactivated the next time your `manage.py sync_ldap_user_data` cron job runs. #### Other fields @@ -246,7 +246,7 @@ the next time your `manage.py sync_ldap_user_data` cron job runs. Other fields you may want to sync from LDAP include: - Boolean flags; `is_realm_admin` (the organization's administrator - permission) is the main one. You can use the + permission) is the main one. You can use the [AUTH_LDAP_USER_FLAGS_BY_GROUP][django-auth-booleans] feature of `django-auth-ldap` to configure a group to get this permissions. (We don't recommend using this flags feature for managing @@ -269,7 +269,7 @@ the fields that would be useful to sync from your LDAP databases. ### Multiple LDAP searches -To do the union of multiple LDAP searches, use `LDAPSearchUnion`. For example: +To do the union of multiple LDAP searches, use `LDAPSearchUnion`. For example: ```python AUTH_LDAP_USER_SEARCH = LDAPSearchUnion( LDAPSearch("ou=users,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(uid=%(user)s)"), @@ -281,7 +281,7 @@ AUTH_LDAP_USER_SEARCH = LDAPSearchUnion( You can restrict access to your Zulip server to a set of LDAP groups using the `AUTH_LDAP_REQUIRE_GROUP` and `AUTH_LDAP_DENY_GROUP` -settings in `/etc/zulip/settings.py`. See the +settings in `/etc/zulip/settings.py`. See the [upstream django-auth-ldap documentation][upstream-ldap-groups] for details. @@ -292,7 +292,7 @@ details. If you're hosting multiple Zulip organizations, you can restrict which users have access to which organizations. This is done by setting `org_membership` in `AUTH_LDAP_USER_ATTR_MAP` to the name of -the LDAP attribute which will contain a list of subdomains that the +the LDAP attribute which will contain a list of subdomains that the user should be allowed to access. For the root subdomain, `www` in the list will work, or any other of @@ -308,7 +308,7 @@ department: www ``` More complex access control rules are possible via the -`AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL` setting. Note that +`AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL` setting. Note that `org_membership` takes precedence over `AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL`: @@ -316,7 +316,7 @@ More complex access control rules are possible via the 2. If `org_membership` is not set or does not allow access, `AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL` will control access. -This contains a map keyed by the organization's subdomain. The +This contains a map keyed by the organization's subdomain. The organization list with multiple maps, that contain a map with an attribute, and a required value for that attribute. If for any of the attribute maps, all user's LDAP attributes match what is configured, access is granted. @@ -331,7 +331,7 @@ LDAP attributes match what is configured, access is granted. ### Troubleshooting Most issues with LDAP authentication are caused by misconfigurations of -the user and email search settings. Some things you can try to get to +the user and email search settings. Some things you can try to get to the bottom of the problem: - Review the instructions for the LDAP configuration type you're @@ -340,7 +340,7 @@ the bottom of the problem: instructions for that configuration type. - Use the `manage.py query_ldap` tool to verify your configuration. The output of the command will usually indicate the cause of any - configuration problem. For the LDAP integration to work, this + configuration problem. For the LDAP integration to work, this command should be able to successfully fetch a complete, correct set of data for the queried user. - You can find LDAP-specific logs in `/var/log/zulip/ldap.log`. If @@ -351,7 +351,7 @@ the bottom of the problem: ## SAML Zulip 2.1 and later supports SAML authentication, used by Okta, -OneLogin, and many other IdPs (identity providers). You can configure +OneLogin, and many other IdPs (identity providers). You can configure it as follows: 1. These instructions assume you have an installed Zulip server; if @@ -372,12 +372,12 @@ it as follows: `/home/zulip/deployments/current/scripts/get-django-setting SOCIAL_AUTH_SAML_SP_ENTITY_ID`. - **SSO URL**: - `https://yourzulipdomain.example.com/complete/saml/`. This is + `https://yourzulipdomain.example.com/complete/saml/`. This is the "SAML ACS url" in SAML terminology. If you're [hosting multiple organizations](../production/multiple-organizations.html#authentication), - you need to use `SOCIAL_AUTH_SUBDOMAIN`. For example, + you need to use `SOCIAL_AUTH_SUBDOMAIN`. For example, if `SOCIAL_AUTH_SUBDOMAIN="auth"` and `EXTERNAL_HOST=zulip.example.com`, this should be `https://auth.zulip.example.com/complete/saml/`. @@ -388,14 +388,14 @@ it as follows: organization name (`displayname` may appear in the IdP's authentication flow; `name` won't be displayed to humans). - Fill out `SOCIAL_AUTH_SAML_ENABLED_IDPS` with data provided by - your identity provider. You may find [the python-social-auth + your identity provider. You may find [the python-social-auth SAML docs](https://python-social-auth.readthedocs.io/en/latest/backends/saml.html) - helpful. You'll need to obtain several values from your IdP's + helpful. You'll need to obtain several values from your IdP's metadata and enter them on the right-hand side of this Python dictionary: 1. Set the outer `idp_name` key to be an identifier for your IdP, - e.g. `testshib` or `okta`. This field appears in URLs for + e.g. `testshib` or `okta`. This field appears in URLs for parts of your Zulip server's SAML authentication flow. 2. The IdP should provide the `url` and `entity_id` values. 3. Save the `x509cert` value to a file; you'll use it in the @@ -403,7 +403,7 @@ it as follows: 4. The values needed in the `attr_` fields are often configurable in your IdP's interface when setting up SAML authentication (referred to as "Attribute Statements" with Okta, or - "Attribute Mapping" with GSuite). You'll want to connect + "Attribute Mapping" with GSuite). You'll want to connect these so that Zulip gets the email address (used as a unique user ID) and name for the user. 5. The `display_name` and `display_icon` fields are used to @@ -411,13 +411,13 @@ it as follows: 6. The `auto_signup` field determines how Zulip should handle login attempts by users who don't have an account yet. -3. Install the certificate(s) required for SAML authentication. You - will definitely need the public certificate of your IdP. Some IdP +3. Install the certificate(s) required for SAML authentication. You + will definitely need the public certificate of your IdP. Some IdP providers also support the Zulip server (Service Provider) having - a certificate used for encryption and signing. We detail these + a certificate used for encryption and signing. We detail these steps as optional below, because they aren't required for basic setup, and some IdPs like Okta don't fully support Service - Provider certificates. You should install them as follows: + Provider certificates. You should install them as follows: 1. On your Zulip server, `mkdir -p /etc/zulip/saml/idps/` 2. Put the IDP public certificate in `/etc/zulip/saml/idps/{idp_name}.crt` @@ -437,7 +437,7 @@ it as follows: certificates above, you can enable the additional setting `"authnRequestsSigned": True` in `SOCIAL_AUTH_SAML_SECURITY_CONFIG` to have the SAMLRequests the server will be issuing to the IdP - signed using those certificates. Additionally, if the IdP supports + signed using those certificates. Additionally, if the IdP supports it, you can upload the public certificate to enable encryption of assertions in the SAMLResponses the IdP will send about authenticated users. @@ -446,7 +446,7 @@ it as follows: `AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`. 6. [Restart the Zulip server](../production/settings.md) to ensure -your settings changes take effect. The Zulip login page should now +your settings changes take effect. The Zulip login page should now have a button for SAML authentication that you can use to log in or create an account (including when creating a new organization). @@ -461,7 +461,7 @@ IdP. The above configuration is sufficient for Service Provider initialized SSO, i.e. you can visit the Zulip webapp and click "Sign in with -{IdP}" and it'll correctly start the authentication flow. If you are +{IdP}" and it'll correctly start the authentication flow. If you are not hosting multiple organizations, with Zulip 3.0+, the above configuration is also sufficient for Identity Provider initiated SSO, i.e. clicking a "Sign in to Zulip" button on the IdP's website can @@ -472,7 +472,7 @@ If you're hosting multiple organizations and thus using the `RelayState` in your IdP of the form `{"subdomain": "yourzuliporganization"}` to let Zulip know which organization to authenticate the user to when they visit your SSO URL -from the IdP. (If the organization is on the root domain, use the +from the IdP. (If the organization is on the root domain, use the empty string: `{"subdomain": ""}`.). ### Restricting access to specific organizations @@ -533,7 +533,7 @@ straightforward way to deploy that SSO solution with Zulip. 4. To configure our SSO integration, edit a copy of `/etc/apache2/sites-available/zulip-sso.example`, saving the result - as `/etc/apache2/sites-available/zulip-sso.conf`. The example sets + as `/etc/apache2/sites-available/zulip-sso.conf`. The example sets up HTTP basic auth, with an `htpasswd` file; you'll want to replace that with configuration for your SSO solution to authenticate the user and set `REMOTE_USER`. @@ -541,7 +541,7 @@ straightforward way to deploy that SSO solution with Zulip. For testing, you may want to move ahead with the rest of the setup using the `htpasswd` example configuration and demonstrate that working end-to-end, before returning later to configure your SSO - solution. You can do that with the following steps: + solution. You can do that with the following steps: ```bash /home/zulip/deployments/current/scripts/restart-server cd /etc/apache2/sites-available/ @@ -551,7 +551,7 @@ straightforward way to deploy that SSO solution with Zulip. 5. Run `a2ensite zulip-sso` to enable the SSO integration within Apache. -6. Run `service apache2 reload` to use your new configuration. If +6. Run `service apache2 reload` to use your new configuration. If Apache isn't already running, you may need to run `service apache2 start` instead. @@ -561,7 +561,7 @@ at `https://zulip.example.com/`) and log in via the SSO solution. ### Troubleshooting Apache-based SSO Most issues with this setup tend to be subtle issues with the -hostname/DNS side of the configuration. Suggestions for how to +hostname/DNS side of the configuration. Suggestions for how to improve this SSO setup documentation are very welcome! - For example, common issues have to do with `/etc/hosts` not mapping @@ -606,14 +606,14 @@ to debug. - The Apache `zulip-sso` site which you've enabled listens on `localhost:8888` and (in the example config) presents the `htpasswd` - dialogue. (In a real configuration, it takes the user through - whatever more complex interaction your SSO solution performs.) The + dialogue. (In a real configuration, it takes the user through + whatever more complex interaction your SSO solution performs.) The user provides correct login information, and the request reaches a second Zulip Django app instance, running behind Apache, with - `REMOTE_USER` set. That request is served by + `REMOTE_USER` set. That request is served by `zerver.views.remote_user_sso`, which just checks the `REMOTE_USER` variable and either logs the user in or, if they don't have an - account already, registers them. The login sets a cookie. + account already, registers them. The login sets a cookie. - After succeeding, that redirects the user back to `/` on port 443. This request is sent by nginx to the main Zulip Django app, which @@ -623,7 +623,7 @@ to debug. ## Sign in with Apple Zulip supports using the web flow for Sign in with Apple on -self-hosted servers. To do so, you'll need to do the following: +self-hosted servers. To do so, you'll need to do the following: 1. Visit [the Apple Developer site][apple-developer] and [Create a Services ID.][apple-create-services-id]. When prompted for a "Return @@ -633,7 +633,7 @@ domain for your server). 1. Create a [Sign in with Apple private key][apple-create-private-key]. 1. Store the resulting private key at - `/etc/zulip/apple-auth-key.p8`. Be sure to set + `/etc/zulip/apple-auth-key.p8`. Be sure to set permissions correctly: ```bash @@ -650,16 +650,16 @@ domain for your server). app that you used in step 1 to configure your Services ID. This might look like "com.example.app". - `SOCIAL_AUTH_APPLE_KEY`: Despite the name this is not a key, but - rather the Key ID of the key you created in step 2. This looks + rather the Key ID of the key you created in step 2. This looks like "F6G7H8I9J0". - `AUTHENTICATION_BACKENDS`: Uncomment (or add) a line like `'zproject.backends.AppleAuthBackend',` to enable Apple auth using the created configuration. 1. Register with Apple the email addresses or domains your Zulip - server sends email to users from. For instructions and background, + server sends email to users from. For instructions and background, see the "Email Relay Service" subsection of - [this page][apple-get-started]. For details on what email + [this page][apple-get-started]. For details on what email addresses Zulip sends from, see our [outgoing email documentation][outgoing-email]. @@ -679,7 +679,7 @@ bit of code, and pull requests to add new backends are welcome. For example, the [Azure Active Directory integration](https://github.com/zulip/zulip/commit/49dbd85a8985b12666087f9ea36acb6f7da0aa4f) was about 30 lines of code, plus some documentation and an -[automatically generated migration][schema-migrations]. We also have +[automatically generated migration][schema-migrations]. We also have helpful developer documentation on [testing auth backends](../development/authentication.md). @@ -689,5 +689,5 @@ helpful developer documentation on ## Development only The `DevAuthBackend` method is used only in development, to allow -passwordless login as any user in a development environment. It's +passwordless login as any user in a development environment. It's mentioned on this page only for completeness. diff --git a/docs/production/deployment.md b/docs/production/deployment.md index 22ff2b4b41..f7ea66b51f 100644 --- a/docs/production/deployment.md +++ b/docs/production/deployment.md @@ -4,7 +4,7 @@ The default Zulip installation instructions will install a complete Zulip server, with all of the services it needs, on a single machine. For production deployment, however, it's common to want to do -something more complicated. This page documents the options for doing so. +something more complicated. This page documents the options for doing so. ## Installing Zulip from Git @@ -23,7 +23,7 @@ You can also [upgrade Zulip from Git](../production/upgrade-or-modify.html#upgra The most common use case for this is upgrading to `main` to get a feature that hasn't made it into an official release yet (often -support for a new base OS release). See [upgrading to +support for a new base OS release). See [upgrading to main][upgrade-to-main] for notes on how `main` works and the support story for it, and [upgrading to future releases][upgrade-to-future-release] for notes on upgrading Zulip @@ -39,7 +39,7 @@ our next major release has a reliable install experience. ## Zulip in Docker Zulip has an officially supported, experimental -[docker image](https://github.com/zulip/docker-zulip). Please note +[docker image](https://github.com/zulip/docker-zulip). Please note that Zulip's [normal installer](../production/install.md) has been extremely reliable for years, whereas the Docker image is new and has rough edges, so we recommend the normal installer unless you have a @@ -52,7 +52,7 @@ as well as those mentioned in the [install](../production/install.html#installer-options) documentation: - `--postgresql-version`: Sets the version of PostgreSQL that will be - installed. We currently support PostgreSQL 10, 11, 12, and 13. + installed. We currently support PostgreSQL 10, 11, 12, and 13. - `--postgresql-missing-dictionaries`: Set `postgresql.missing_dictionaries` ([docs][doc-settings]) in the @@ -91,11 +91,11 @@ env PUPPET_CLASSES=zulip::profile::redis ./scripts/setup/install ``` All puppet modules under `zulip::profile` are allowed to be configured -stand-alone on a host. You can see most likely manifests you might +stand-alone on a host. You can see most likely manifests you might want to choose in the list of includes in [the main manifest for the default all-in-one Zulip server][standalone.pp], though it's also possible to subclass some of the lower-level manifests defined in that -directory if you want to customize. A good example of doing this is +directory if you want to customize. A good example of doing this is in the [zulip_ops Puppet configuration][zulipchat-puppet] that we use as part of managing chat.zulip.org and zulip.com. @@ -116,7 +116,7 @@ below. #### Step 1: Set up Zulip Follow the [standard instructions](../production/install.md), with one -change. When running the installer, pass the `--no-init-db` +change. When running the installer, pass the `--no-init-db` flag, e.g.: ```bash @@ -157,7 +157,7 @@ database provider for the available options. In `/etc/zulip/settings.py` on your Zulip server, configure the following settings with details for how to connect to your PostgreSQL -server. Your database provider should provide these details. +server. Your database provider should provide these details. - `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server. - `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server. @@ -197,7 +197,7 @@ configure that as follows: ``` 1. As root, run - `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This + `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will convert Zulip's main `nginx` configuration file to use your new port. @@ -211,14 +211,14 @@ behind reverse proxies. Zulip supports routing all of its outgoing HTTP and HTTPS traffic through an HTTP `CONNECT` proxy, such as [Smokescreen][smokescreen]; this includes outgoing webhooks, image and website previews, and -mobile push notifications. You may wish to enable this feature to +mobile push notifications. You may wish to enable this feature to provide a consistent egress point, or enforce access control on URLs to prevent [SSRF][ssrf] against internal resources. To use Smokescreen: 1. Add `, zulip::profile::smokescreen` to the list of `puppet_classes` - in `/etc/zulip/zulip.conf`. A typical value after this change is: + in `/etc/zulip/zulip.conf`. A typical value after this change is: ```ini puppet_classes = zulip::profile::standalone, zulip::profile::smokescreen ``` @@ -238,12 +238,12 @@ To use Smokescreen: ``` 1. If you intend to also make the Smokescreen install available to - other hosts, set `listen_address` in the same block. Note that you + other hosts, set `listen_address` in the same block. Note that you must control access to the Smokescreen port if you do this, as failing to do so opens a public HTTP proxy! 1. As root, run - `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This + `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will compile and install Smokescreen, reconfigure services to use it, and restart Zulip. @@ -286,7 +286,7 @@ HTTP as follows: ``` 1. As root, run -`/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This +`/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will convert Zulip's main `nginx` configuration file to allow HTTP instead of HTTPS. @@ -296,12 +296,12 @@ instead of HTTPS. ### nginx configuration For `nginx` configuration, there's two things you need to set up: -- The root `nginx.conf` file. We recommend using +- The root `nginx.conf` file. We recommend using `/etc/nginx/nginx.conf` from your Zulip server for our recommended - settings. E.g. if you don't set `client_max_body_size`, it won't be + settings. E.g. if you don't set `client_max_body_size`, it won't be possible to upload large files to your Zulip server. - The `nginx` site-specific configuration (in - `/etc/nginx/sites-available`) for the Zulip app. The following + `/etc/nginx/sites-available`) for the Zulip app. The following example is a good starting point: ```nginx @@ -351,7 +351,7 @@ make the following changes in two configuration files. 3. Restart your Zulip server with `/home/zulip/deployments/current/scripts/restart-server`. 4. Create an Apache2 virtual host configuration file, similar to the - following. Place it the appropriate path for your Apache2 + following. Place it the appropriate path for your Apache2 installation and enable it (E.g. if you use Debian or Ubuntu, then place it in `/etc/apache2/sites-available/zulip.example.com.conf` and then run @@ -417,28 +417,28 @@ things you need to be careful about when configuring it: 1. Configure your reverse proxy (or proxies) to correctly maintain the `X-Forwarded-For` HTTP header, which is supposed to contain the series -of IP addresses the request was forwarded through. You can verify +of IP addresses the request was forwarded through. You can verify your work by looking at `/var/log/zulip/server.log` and checking it has the actual IP addresses of clients, not the IP address of the proxy server. 2. Ensure your proxy doesn't interfere with Zulip's use of long-polling for real-time push from the server to your users' -browsers. This [nginx code snippet][nginx-proxy-longpolling-config] +browsers. This [nginx code snippet][nginx-proxy-longpolling-config] does this. The key configuration options are, for the `/json/events` and `/api/1/events` endpoints: -- `proxy_read_timeout 1200;`. It's critical that this be +- `proxy_read_timeout 1200;`. It's critical that this be significantly above 60s, but the precise value isn't important. -- `proxy_buffering off`. If you don't do this, your `nginx` proxy may +- `proxy_buffering off`. If you don't do this, your `nginx` proxy may return occasional 502 errors to clients using Zulip's events API. 3. The other tricky failure mode we've seen with `nginx` reverse proxies is that they can load-balance between the IPv4 and IPv6 -addresses for a given hostname. This can result in mysterious errors -that can be quite difficult to debug. Be sure to declare your +addresses for a given hostname. This can result in mysterious errors +that can be quite difficult to debug. Be sure to declare your `upstreams` equivalent in a way that won't do load-balancing unexpectedly (e.g. pointing to a DNS name that you haven't configured with multiple IPs for your Zulip machine; sometimes this happens with @@ -449,7 +449,7 @@ IPv6 configuration). The file `/etc/zulip/zulip.conf` is used to configure properties of the system and deployment; `/etc/zulip/settings.py` is used to -configure the application itself. The `zulip.conf` sections and +configure the application itself. The `zulip.conf` sections and settings are described below. ### `[machine]` @@ -484,11 +484,11 @@ extension](../subsystems/full-text-search.html#multi-language-full-text-search). #### `deploy_options` Options passed by `upgrade-zulip` and `upgrade-zulip-from-git` into -`upgrade-zulip-stage-2`. These might be any of: +`upgrade-zulip-stage-2`. These might be any of: - - **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need + - **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need to run `zulip-puppet-apply` manually after the upgrade. - - **`--skip-migrations`** skips running database migrations. The + - **`--skip-migrations`** skips running database migrations. The user will need to run `./manage.py migrate` manually after the upgrade. - **`--skip-purge-old-deployments`** skips purging old deployments; without it, only deployments with the last two weeks are kept. @@ -535,7 +535,7 @@ mode). The calculation is based on whether the system has enough memory (currently 3.5GiB) to run a single-server Zulip installation in the multiprocess mode. -Set to `true` or `false` to override the automatic calculation. This +Set to `true` or `false` to override the automatic calculation. This override is useful both Docker systems (where the above algorithm might see the host's memory, not the container's) and/or when using remote servers for postgres, memcached, redis, and RabbitMQ. @@ -560,7 +560,7 @@ more than 3.5GiB of RAM, 4 on hosts with less. #### `auto_renew` If set to the string `yes`, [Certbot will attempt to automatically -renew its certificate](../production/ssl-certificates.html#certbot-recommended). Do +renew its certificate](../production/ssl-certificates.html#certbot-recommended). Do no set by hand; use `scripts/setup/setup-certbot` to configure this. @@ -612,7 +612,7 @@ connections. #### `version` -The version of PostgreSQL that is in use. Do not set by hand; use the +The version of PostgreSQL that is in use. Do not set by hand; use the [PostgreSQL upgrade tool](../production/upgrade-or-modify.html#upgrading-postgresql). diff --git a/docs/production/email-gateway.md b/docs/production/email-gateway.md index 02959f375e..3dcb9cd670 100644 --- a/docs/production/email-gateway.md +++ b/docs/production/email-gateway.md @@ -1,19 +1,19 @@ # Incoming email integration Zulip's incoming email gateway integration makes it possible to send -messages into Zulip by sending an email. It's highly recommended +messages into Zulip by sending an email. It's highly recommended because it enables: - When users reply to one of Zulip's message notification emails from their email client, the reply can go directly into Zulip. - Integrating third-party services that can send email notifications - into Zulip. See the [integration + into Zulip. See the [integration documentation](https://zulip.com/integrations/doc/email) for details. Once this integration is configured, each stream will have a special -email address displayed on the stream settings page. Emails sent to +email address displayed on the stream settings page. Emails sent to that address will be delivered into the stream. There are two ways to configure Zulip's email gateway: @@ -24,7 +24,7 @@ There are two ways to configure Zulip's email gateway: inbox (`username@example.com`) every minute for new emails. The local delivery configuration is preferred for production because -it supports nicer looking email addresses and has no cron delay. The +it supports nicer looking email addresses and has no cron delay. The polling option is convenient for testing/developing this feature because it doesn't require a public IP address or setting up MX records in DNS. @@ -45,7 +45,7 @@ integration; you just need to enable and configure it as follows. The main decision you need to make is what email domain you want to use for the gateway; for this discussion we'll use -`emaildomain.example.com`. The email addresses used by the gateway +`emaildomain.example.com`. The email addresses used by the gateway will look like `foo@emaildomain.example.com`, so we recommend using `EXTERNAL_HOST` here. @@ -55,7 +55,7 @@ using an [HTTP reverse proxy][reverse-proxy]). 1. Using your DNS provider, create a DNS MX (mail exchange) record configuring email for `emaildomain.example.com` to be processed by - `hostname.example.com`. You can check your work using this command: + `hostname.example.com`. You can check your work using this command: ```console $ dig +short emaildomain.example.com -t MX @@ -65,7 +65,7 @@ using an [HTTP reverse proxy][reverse-proxy]). 1. Log in to your Zulip server; the remaining steps all happen there. 1. Add `, zulip::postfix_localmail` to `puppet_classes` in - `/etc/zulip/zulip.conf`. A typical value after this change is: + `/etc/zulip/zulip.conf`. A typical value after this change is: ```ini puppet_classes = zulip::profile::standalone, zulip::postfix_localmail ``` @@ -93,15 +93,15 @@ using an [HTTP reverse proxy][reverse-proxy]). 1. Restart your Zulip server with `/home/zulip/deployments/current/scripts/restart-server`. -Congratulations! The integration should be fully operational. +Congratulations! The integration should be fully operational. [reverse-proxy]: ../production/deployment.html#putting-the-zulip-application-behind-a-reverse-proxy ## Polling setup 1. Create an email account dedicated to Zulip's email gateway - messages. We assume the address is of the form - `username@example.com`. The email provider needs to support the + messages. We assume the address is of the form + `username@example.com`. The email provider needs to support the standard model of delivering emails sent to `username+stuff@example.com` to the `username@example.com` inbox. @@ -122,4 +122,4 @@ Congratulations! The integration should be fully operational. sudo cp puppet/zulip/files/cron.d/email-mirror /etc/cron.d/ ``` -Congratulations! The integration should be fully operational. +Congratulations! The integration should be fully operational. diff --git a/docs/production/email.md b/docs/production/email.md index a56a0da1d4..26b08d03ec 100644 --- a/docs/production/email.md +++ b/docs/production/email.md @@ -6,13 +6,13 @@ email addresses and send notifications. ## How to configure 1. Identify an outgoing email (SMTP) account where you can have Zulip - send mail. If you don't already have one you want to use, see + send mail. If you don't already have one you want to use, see [Email services](#email-services) below. 1. Fill out the section of `/etc/zulip/settings.py` headed "Outgoing - email (SMTP) settings". This includes the hostname and typically + email (SMTP) settings". This includes the hostname and typically the port to reach your SMTP provider, and the username to log in to - it. You'll also want to fill out the noreply email section. + it. You'll also want to fill out the noreply email section. 1. Put the password for the SMTP user account in `/etc/zulip/zulip-secrets.conf` by setting `email_password`. For @@ -57,7 +57,7 @@ the best documentation). If you don't have an existing outgoing SMTP provider, don't worry! Each of the options we recommend above (as well as dozens of other -services) have free options. Once you've signed up, you'll want to +services) have free options. Once you've signed up, you'll want to find the service's provided "SMTP credentials", and configure Zulip as follows: @@ -88,7 +88,7 @@ EMAIL_HOST_USER = "" We should emphasize that because modern spam filtering is very aggressive, you should make sure your downstream email system is configured to properly sign outgoing email sent by your Zulip server -(or check your spam folder) when using this configuration. See +(or check your spam folder) when using this configuration. See [documentation on using Django with a local postfix server][postfix-email] for additional advice. @@ -97,7 +97,7 @@ for additional advice. ### Using Gmail for outgoing email We don't recommend using an inbox product like Gmail for outgoing -email, because Gmail's anti-spam measures make this annoying. But if +email, because Gmail's anti-spam measures make this annoying. But if you want to use a Gmail account to send outgoing email anyway, here's how to make it work: - Create a totally new Gmail account for your Zulip server; you don't @@ -111,13 +111,13 @@ how to make it work: Gmail doesn't allow servers to send outgoing email by default. - Note also that the rate limits for Gmail are also quite low (e.g. 100 / day), so it's easy to get rate-limited if your server - has significant traffic. For more active servers, we recommend + has significant traffic. For more active servers, we recommend moving to a free account on a transactional email service. ### Logging outgoing email to a file for prototyping For prototyping, you might want to proceed without setting up an email -provider. If you want to see the emails Zulip would have sent, you +provider. If you want to see the emails Zulip would have sent, you can log them to a file instead. To do so, add these lines to `/etc/zulip/settings.py`: @@ -142,14 +142,14 @@ su zulip -c '/home/zulip/deployments/current/manage.py send_test_email user@exam ``` If it doesn't throw an error, it probably worked; you can confirm by -checking your email. You should get two emails: One sent by the +checking your email. You should get two emails: One sent by the default From address for your Zulip server, and one sent by the "noreply" From address. If it doesn't work, check these common failure causes: - Your hosting provider may block outgoing SMTP traffic in its default - firewall rules. Check whether the port `EMAIL_PORT` is blocked in + firewall rules. Check whether the port `EMAIL_PORT` is blocked in your hosting provider's firewall. - Your SMTP server's permissions might not allow the email account @@ -162,11 +162,11 @@ If it doesn't work, check these common failure causes: If necessary, you can set `ADD_TOKENS_TO_NOREPLY_ADDRESS` to `False` in `/etc/zulip/settings.py` (which will cause these confirmation - emails to be sent from a consistent `noreply@` address). Disabling + emails to be sent from a consistent `noreply@` address). Disabling `ADD_TOKENS_TO_NOREPLY_ADDRESS` is generally safe if you are not using Zulip's feature that allows anyone to create an account in your Zulip organization if they have access to an email address in a - certain domain. See [this article][helpdesk-attack] for details on + certain domain. See [this article][helpdesk-attack] for details on the security issue with helpdesk software that `ADD_TOKENS_TO_NOREPLY_ADDRESS` helps protect against. @@ -192,7 +192,7 @@ aren't receiving emails from Zulip: - Starting with Zulip 1.7, Zulip logs an entry in `/var/log/zulip/send_email.log` whenever it attempts to send an - email. The log entry includes whether the request succeeded or failed. + email. The log entry includes whether the request succeeded or failed. - If attempting to send an email throws an exception, a traceback should be in `/var/log/zulip/errors.log`, along with any other @@ -205,14 +205,14 @@ aren't receiving emails from Zulip: - Zulip's email sending configuration is based on the standard Django [SMTP backend](https://docs.djangoproject.com/en/2.0/topics/email/#smtp-backend) - configuration. So if you're having trouble getting your email + configuration. So if you're having trouble getting your email provider working, you may want to search for documentation related to using your email provider with Django. The one thing we've changed from the Django defaults is that we read the email password from the `email_password` entry in the Zulip secrets file, as part of our policy of not having any secret - information in the `/etc/zulip/settings.py` file. In other words, + information in the `/etc/zulip/settings.py` file. In other words, if Django documentation references setting `EMAIL_HOST_PASSWORD`, you should instead set `email_password` in `/etc/zulip/zulip-secrets.conf`. diff --git a/docs/production/expensive-migrations.md b/docs/production/expensive-migrations.md index 542099d1ee..985c0b1352 100644 --- a/docs/production/expensive-migrations.md +++ b/docs/production/expensive-migrations.md @@ -8,14 +8,14 @@ Zulip 1.7 and 1.9 each contain some significant database migrations that can take several minutes to run. The upgrade process automatically minimizes disruption by running -these first, before beginning the user-facing downtime. However, if +these first, before beginning the user-facing downtime. However, if you'd like to watch the downtime phase of the upgrade closely, you can run them manually before starting the upgrade: 1. Log in to your Zulip server as the `zulip` user (or as `root` and then run `su zulip` to drop privileges), and `cd /home/zulip/deployments/current` -2. Run `./manage.py dbshell`. This will open a shell connected to the +2. Run `./manage.py dbshell`. This will open a shell connected to the PostgreSQL database. 3. In the PostgreSQL shell, run the following commands: @@ -61,12 +61,12 @@ can run them manually before starting the upgrade: ``` These will take some time to run, during which the server will -continue to serve user traffic as usual with no disruption. Once they +continue to serve user traffic as usual with no disruption. Once they finish, you can proceed with installing Zulip 1.7. To help you estimate how long these will take on your server: count the number of UserMessage rows, with `select COUNT(*) from zerver_usermessage;` -at the `./manage.py dbshell` prompt. At the time these migrations +at the `./manage.py dbshell` prompt. At the time these migrations were run on chat.zulip.org, it had 75M UserMessage rows; the first 5 indexes took about 1 minute each to create, and the final, "unread_message" index took more like 10 minutes. diff --git a/docs/production/export-and-import.md b/docs/production/export-and-import.md index 72f0685322..d78aae7ef6 100644 --- a/docs/production/export-and-import.md +++ b/docs/production/export-and-import.md @@ -7,7 +7,7 @@ service (or back): - The [Backup](#backups) tool is designed for exact restoration of a Zulip server's state, for disaster recovery, testing with production - data, or hardware migration. This tool has a few limitations: + data, or hardware migration. This tool has a few limitations: - Backups must be restored on a server running the same Zulip version (most precisely, one where `manage.py showmigrations` has @@ -20,13 +20,13 @@ service (or back): We highly recommend this tool in situations where it is applicable, because it is highly optimized and highly stable, since the hard - work is done by the built-in backup feature of PostgreSQL. We also + work is done by the built-in backup feature of PostgreSQL. We also document [backup details](#backup-details) for users managing backups manually. - The logical [Data export](#data-export) tool is designed for migrating data between Zulip Cloud and other Zulip servers, as well - as various auditing purposes. The logical export tool produces a + as various auditing purposes. The logical export tool produces a `.tar.gz` archive with most of the Zulip database data encoded in JSON files–a format shared by our [data import](#import-into-a-new-zulip-server) tools for third-party @@ -34,10 +34,10 @@ service (or back): [Slack](https://zulip.com/help/import-from-slack). Like the backup tool, logical data exports must be imported on a - Zulip server running the same version. However, logical data + Zulip server running the same version. However, logical data exports can be imported on Zulip servers running a different PostgreSQL version or hosting a different set of Zulip - organizations. We recommend this tool in cases where the backup + organizations. We recommend this tool in cases where the backup tool isn't applicable, including situations where an easily machine-parsable export format is desired. @@ -66,9 +66,9 @@ su zulip -c '/home/zulip/deployments/current/manage.py backup' The backup tool provides the following options: - `--output=/tmp/backup.tar.gz`: Filename to write the backup tarball - to (default: write to a file in `/tmp`). On success, the + to (default: write to a file in `/tmp`). On success, the console output will show the path to the output tarball. -- `--skip-db`: Skip backup of the database. Useful if you're using a +- `--skip-db`: Skip backup of the database. Useful if you're using a remote PostgreSQL host with its own backup system and just need to back up non-database state. - `--skip-uploads`: If `LOCAL_UPLOADS_DIR` is set, user-uploaded files @@ -82,7 +82,7 @@ server's state on another machine perfectly. First, [install a new Zulip server through Step 3][install-server] with the same version of both the base OS and Zulip from your previous -installation. Then, run as root: +installation. Then, run as root: ```bash /home/zulip/deployments/current/scripts/setup/restore-backup /path/to/backup @@ -109,7 +109,7 @@ errors when trying to access it via `zuliptest.example.com`. If you're not sure what versions were in use when a given backup was created, you can get that information via the files in the backup -tarball: `postgres-version`, `os-version`, and `zulip-version`. The +tarball: `postgres-version`, `os-version`, and `zulip-version`. The following command may be useful for viewing these files without extracting the entire archive. @@ -128,27 +128,27 @@ server, including the database, settings, secrets from The following data is not included in these backup archives, and you may want to back up separately: -- The server access/error logs from `/var/log/zulip`. The Zulip +- The server access/error logs from `/var/log/zulip`. The Zulip server only appends to logs, and they can be very large compared to the rest of the data for a Zulip server. - Files uploaded with the Zulip - [S3 file upload backend](../production/upload-backends.md). We + [S3 file upload backend](../production/upload-backends.md). We don't include these for two reasons. First, the uploaded file data in S3 can easily be many times larger than the rest of the backup, and downloading it all to a server doing a backup could easily - exceed its disk capacity. Additionally, S3 is a reliable persistent + exceed its disk capacity. Additionally, S3 is a reliable persistent storage system with its own high-quality tools for doing backups. -- SSL certificates. These are not included because they are +- SSL certificates. These are not included because they are particularly security-sensitive and are either trivially replaced (if generated via Certbot) or provided by the system administrator. For completeness, Zulip's backups do not include certain highly -transient state that Zulip doesn't store in a database. For example, +transient state that Zulip doesn't store in a database. For example, typing status data, API rate-limiting counters, and RabbitMQ queues that are essentially always empty in a healthy server (like outgoing -emails to send). You can check whether these queues are empty using +emails to send). You can check whether these queues are empty using `rabbitmqctl list_queues`. #### Backup details @@ -156,14 +156,14 @@ emails to send). You can check whether these queues are empty using This section is primarily for users managing backups themselves (E.g. if they're using a remote PostgreSQL database with an existing backup strategy), and also serves as documentation for what is -included in the backups generated by Zulip's standard tools. The +included in the backups generated by Zulip's standard tools. The data includes: -- The PostgreSQL database. You can back this up with any standard -database export or backup tool. Zulip has built-in support for taking +- The PostgreSQL database. You can back this up with any standard +database export or backup tool. Zulip has built-in support for taking daily incremental backups using [wal-g](https://github.com/wal-g/wal-g); these backups are stored for -30 days in S3. If you have an Amazon S3 bucket you wish to store for +30 days in S3. If you have an Amazon S3 bucket you wish to store for storing the backups, edit `/etc/zulip/zulip-secrets.conf` on the PostgreSQL server to add: @@ -174,12 +174,12 @@ PostgreSQL server to add: ``` After adding the secrets, run - `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. You + `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. You can (and should) monitor that backups are running regularly via the Nagios plugin installed into `/usr/lib/nagios/plugins/zulip_postgresql_backups/check_postgresql_backup`. -- Any user-uploaded files. If you're using S3 as storage for file +- Any user-uploaded files. If you're using S3 as storage for file uploads, this is backed up in S3. But if you have instead set `LOCAL_UPLOADS_DIR`, any files uploaded by users (including avatars) will be stored in that directory and you'll want to back it up. @@ -237,7 +237,7 @@ replication. You can see the configuration in these files: - `puppet/zulip_ops/files/postgresql/*` We use this configuration for Zulip Cloud, and it works well in -production, but it's not fully generic. Contributions to make it a +production, but it's not fully generic. Contributions to make it a supported and documented option for other installations are appreciated. @@ -254,16 +254,16 @@ backups. ### Preventing changes during the export For best results, you'll want to shut down access to the organization -before exporting; so that nobody can send new messages (etc.) while -you're exporting data. There are two ways to do this: +before exporting; so that nobody can send new messages (etc.) while +you're exporting data. There are two ways to do this: -1. `./scripts/stop-server`, which stops the whole server. This is +1. `./scripts/stop-server`, which stops the whole server. This is preferred if you're not hosting multiple organizations, because it has no side effects other than disabling the Zulip server for the duration. 1. Pass `--deactivate` to `./manage export`, which first deactivates the target organization, logging out all active login sessions and -preventing all accounts from logging in or accessing the API. This is +preventing all accounts from logging in or accessing the API. This is preferred for environments like Zulip Cloud where you might want to export a single organization without disrupting any other users, and the intent is to move hosting of the organization (and forcing users @@ -271,7 +271,7 @@ to re-log in would be required as part of the hosting migration anyway). We include both options in the instructions below, commented out so -that neither runs (using the `# ` at the start of the lines). If +that neither runs (using the `# ` at the start of the lines). If you'd like to use one of these options, remove the `# ` at the start of the lines for the appropriate option. @@ -291,7 +291,7 @@ cd /home/zulip/deployments/current the default organization hosted at the Zulip server's root domain.) This will generate a tarred archive with a name like -`/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several +`/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several JSON files (containing the Zulip organization's data) as well as an archive of all the organization's uploaded files. @@ -323,25 +323,25 @@ archive of all the organization's uploaded files. 2. If your new Zulip server is meant to fully replace a previous Zulip server, you may want to copy some settings from `/etc/zulip` to your new server to reuse the server-level configuration and secret keys -from your old server. There are a few important details to understand +from your old server. There are a few important details to understand about doing so: - Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is - safe and recommended. Care is required when copying secrets from + safe and recommended. Care is required when copying secrets from `/etc/zulip/zulip-secrets.conf` (details below). - If you copy `zulip_org_id` and `zulip_org_key` (the credentials for the [mobile push notifications service](../production/mobile-push-notifications.md)), you should be very careful to make sure the no users had their IDs renumbered during the import process (this can be checked using - `manage.py shell` with some care). The push notifications + `manage.py shell` with some care). The push notifications service has a mapping of which `user_id` values are associated with which devices for a given Zulip server (represented by the - `zulip_org_id` registration). This means that if any `user_id` + `zulip_org_id` registration). This means that if any `user_id` values were renumbered during the import and you don't register a new `zulip_org_id`, push notifications meant for the user who now has ID 15 may be sent to devices registered by the user who had - user ID 15 before the data export (yikes!). The solution is + user ID 15 before the data export (yikes!). The solution is simply to not copy these settings and re-register your server for mobile push notifications if any users had their IDs renumbered during the logical export/import process. @@ -354,7 +354,7 @@ about doing so: authentication and email delivery so that those work on your new server. - Copying `avatar_salt` is not recommended, due to similar issues - to the mobile push notifications service. Zulip will + to the mobile push notifications service. Zulip will automatically rewrite avatars at URLs appropriate for the new user IDs, and using the same avatar salt (and same server URL) post import could result in issues with browsers caching the @@ -400,7 +400,7 @@ Your users will need to either authenticate using something like Google auth or start by resetting their passwords. You can use the `./manage.py send_password_reset_email` command to -send password reset emails to your users. We +send password reset emails to your users. We recommend starting with sending one to yourself for testing: ```bash @@ -418,7 +418,7 @@ and then once you're ready, you can email them to everyone using e.g. If you did a test import of a Zulip organization, you may want to delete the test import data from your Zulip server before doing a -final import. You can **permanently delete** all data from a Zulip +final import. You can **permanently delete** all data from a Zulip organization using the following procedure: - Start a [Zulip management shell](../production/management-commands.html#manage-py-shell) diff --git a/docs/production/giphy-gif-integration.md b/docs/production/giphy-gif-integration.md index c19736105f..c3e120ba03 100644 --- a/docs/production/giphy-gif-integration.md +++ b/docs/production/giphy-gif-integration.md @@ -17,7 +17,7 @@ To enable this integration, you need to get a production API key from 1. Choose **SDK** as product type and click **Next Step**. 1. Enter a name and a description for your app and click on **Create - New App**. The hostname for your Zulip server is a fine name. + New App**. The hostname for your Zulip server is a fine name. 1. You will receive a beta API key. Apply for a production API key by following the steps mentioned by GIPHY on the same page. @@ -36,9 +36,9 @@ follows: 1. Restart the Zulip server with `/home/zulip/deployments/current/scripts/restart-server`. -Congratulations! You've configured the GIPHY integration for your -Zulip server. Your users can now use the integration as described in -[the help center article][help-center-giphy]. (A browser reload may +Congratulations! You've configured the GIPHY integration for your +Zulip server. Your users can now use the integration as described in +[the help center article][help-center-giphy]. (A browser reload may be required). diff --git a/docs/production/install-existing-server.md b/docs/production/install-existing-server.md index 1f2e3f4e1c..cb064a9281 100644 --- a/docs/production/install-existing-server.md +++ b/docs/production/install-existing-server.md @@ -16,7 +16,7 @@ recommended, and you may break your server. Make sure you have backups and a provisioning script ready to go to wipe and restore your existing services if (when) your server goes down. -These instructions are only for experts. If you're not an experienced +These instructions are only for experts. If you're not an experienced Linux sysadmin, you will have a much better experience if you get a dedicated VM to install Zulip on instead (or [use zulip.com](https://zulip.com)). @@ -36,7 +36,7 @@ sudo meld /etc/nginx/nginx.conf /etc/nginx/nginx.conf.zulip # be sure to merge Since the file in Zulip is an [ERB Puppet template](https://puppet.com/docs/puppet/7/lang_template_erb.html), you will also need to replace any `<%= ... %>` sections with -appropriate content. For instance `<%= @ca_crt %>` should be replaced +appropriate content. For instance `<%= @ca_crt %>` should be replaced with `/etc/ssl/certs/ca-certificates.crt` on Debian and Ubuntu installs. @@ -49,7 +49,7 @@ $ sudo service nginx restart ``` Zulip's Puppet configuration will change the ownership of -`/var/log/nginx` so that the `zulip` user can access it. Depending on +`/var/log/nginx` so that the `zulip` user can access it. Depending on your configuration, this may or may not cause problems. ### Puppet @@ -73,9 +73,9 @@ $ sudo service puppet stop Zulip expects to install PostgreSQL 12, and find that listening on port 5432; any other version of PostgreSQL that is detected at install -time will cause the install to abort. If you already have PostgreSQL +time will cause the install to abort. If you already have PostgreSQL installed, you can pass `--postgresql-version=` to the installer to -have it use that version. It will replace the package with the latest +have it use that version. It will replace the package with the latest from the PostgreSQL apt repository, but existing data will be retained. @@ -85,7 +85,7 @@ that. ### Memcached, Redis, and RabbitMQ -Zulip will, by default, configure these services for its use. The +Zulip will, by default, configure these services for its use. The configuration we use is pretty basic, but if you're using them for something else, you'll want to make sure the configurations are compatible. @@ -97,6 +97,6 @@ We don't provide a convenient way to uninstall a Zulip server. ## No support, but contributions welcome! Most of the limitations are things we'd accept a pull request to fix; -we welcome contributions to shrink this list of gotchas. Chat with us +we welcome contributions to shrink this list of gotchas. Chat with us in the [chat.zulip.org community](../contributing/chat-zulip-org.md) if you're interested in helping! diff --git a/docs/production/install.md b/docs/production/install.md index 3d47602eb4..f5a9587714 100644 --- a/docs/production/install.md +++ b/docs/production/install.md @@ -59,7 +59,7 @@ If the script gives an error, consult [Troubleshooting](#troubleshooting) below. settings. - `--self-signed-cert`: With this option, the Zulip installer - generates a self-signed SSL certificate for the server. This isn't + generates a self-signed SSL certificate for the server. This isn't suitable for production use, but may be convenient for testing. - `--certbot`: With this option, the Zulip installer automatically @@ -77,7 +77,7 @@ documentation. ## Step 3: Create a Zulip organization, and log in -On success, the install script prints a link. If you're [restoring a +On success, the install script prints a link. If you're [restoring a backup][zulip-backups] or importing your data from [Slack][slack-import], or another Zulip server, you should stop here and return to the import instructions. @@ -85,13 +85,13 @@ and return to the import instructions. [slack-import]: https://zulip.com/help/import-from-slack [zulip-backups]: ../production/export-and-import.html#backups -Otherwise, open the link in a browser. Follow the prompts to set up +Otherwise, open the link in a browser. Follow the prompts to set up your organization, and your own user account as an administrator. Then, log in! -The link is a secure one-time-use link. If you need another +The link is a secure one-time-use link. If you need another later, you can generate a new one by running -`manage.py generate_realm_creation_link` on the server. See also our +`manage.py generate_realm_creation_link` on the server. See also our doc on running [multiple organizations on the same server](multiple-organizations.md) if that's what you're planning to do. @@ -109,7 +109,7 @@ Learning more: - Subscribe to the [Zulip announcements email list](https://groups.google.com/forum/#!forum/zulip-announce) for -server administrators. This extremely low-traffic list is for +server administrators. This extremely low-traffic list is for important announcements, including [new releases](../overview/release-lifecycle.md) and security issues. You can also use the [RSS @@ -130,7 +130,7 @@ server. The install script does several things: - Creates the `zulip` user, which the various Zulip servers will run as. - Creates `/home/zulip/deployments/`, which the Zulip code for this -deployment (and future deployments when you upgrade) goes into. At the +deployment (and future deployments when you upgrade) goes into. At the very end of the install process, the script moves the Zulip code tree it's running from (which you unpacked from a tarball above) to a directory there, and makes `/home/zulip/deployments/current` as a @@ -146,20 +146,20 @@ machines, check out our [deployment options documentation](deployment.md). ## Troubleshooting **Install script.** -The Zulip install script is designed to be idempotent. This means +The Zulip install script is designed to be idempotent. This means that if it fails, then once you've corrected the cause of the failure, you can just rerun the script. The install script automatically logs a transcript to -`/var/log/zulip/install.log`. In case of failure, you might find the -log handy for resolving the issue. Please include a copy of this log +`/var/log/zulip/install.log`. In case of failure, you might find the +log handy for resolving the issue. Please include a copy of this log file in any bug reports. **The `zulip` user's password.** By default, the `zulip` user doesn't have a password, and is intended to be accessed by `su zulip` from the `root` user (or via SSH keys or a password, if you want to set those -up, but that's up to you as the system administrator). Most people +up, but that's up to you as the system administrator). Most people who are prompted for a password when running `su zulip` turn out to already have switched to the `zulip` user earlier in their session, and can just skip that step. @@ -173,7 +173,7 @@ how to debug. **Community.** If the tips above don't help, please visit [#production help][production-help] in the [Zulip development community server][chat-zulip-org] for realtime help, and we'll try to help you -out! Please provide details like the full traceback from the bottom +out! Please provide details like the full traceback from the bottom of `/var/log/zulip/errors.log` in your report (ideally in a [code block][code-block]). diff --git a/docs/production/management-commands.md b/docs/production/management-commands.md index e464ef29b9..af1a800bad 100644 --- a/docs/production/management-commands.md +++ b/docs/production/management-commands.md @@ -1,13 +1,13 @@ # Management commands Sometimes, you need to modify or inspect Zulip data from the command -line. To help with this, Zulip ships with over 100 command-line tools +line. To help with this, Zulip ships with over 100 command-line tools implemented using the [Django management commands framework][django-management]. ## Running management commands -Start by logging in as the `zulip` user on the Zulip server. Then run +Start by logging in as the `zulip` user on the Zulip server. Then run them as follows: ```bash @@ -25,7 +25,7 @@ primarily want to use those in the `[zerver]` section as those are the ones specifically built for Zulip. As a warning, some of them are designed for specific use cases and may -cause problems if run in other situations. If you're not sure, it's +cause problems if run in other situations. If you're not sure, it's worth reading the documentation (or the code, usually available at `zerver/management/commands/`; they're generally very simple programs). @@ -54,14 +54,14 @@ unlikely to ever need to interact with that realm.) Unless you are [hosting multiple organizations on your Zulip server](../production/multiple-organizations.md), your single Zulip organization on the root domain will have the empty -string (`''`) as its `string_id`. So you can run e.g.: +string (`''`) as its `string_id`. So you can run e.g.: ```console zulip@zulip:~$ /home/zulip/deployments/current/manage.py show_admins -r '' ``` Otherwise, the `string_id` will correspond to the organization's -subdomain. E.g. on `it.zulip.example.com`, use +subdomain. E.g. on `it.zulip.example.com`, use `/home/zulip/deployments/current/manage.py show_admins -r it`. ## manage.py shell @@ -86,11 +86,11 @@ formatting data from Zulip's tables for inspection; Zulip's own you understand how the codebase is organized. We recommend against directly editing objects and saving them using -Django's `object.save()`. While this will save your changes to the +Django's `object.save()`. While this will save your changes to the database, for most objects, in addition to saving the changes to the database, one may also need to flush caches, notify the apps and open browser windows, and record the change in Zulip's `RealmAuditLog` -audit history table. For almost any data change you want to do, there +audit history table. For almost any data change you want to do, there is already a function in `zerver.lib.actions.py` with a name like `do_change_full_name` that updates that field and notifies clients correctly. @@ -102,16 +102,16 @@ access other functions, you'll need to import them yourself. ## Other useful manage.py commands There are dozens of useful management commands under -`zerver/management/commands/`. We detail a few here: +`zerver/management/commands/`. We detail a few here: - `./manage.py help`: Lists all available management commands. - `./manage.py dbshell`: If you're more comfortable with raw SQL than Python, this will open a PostgreSQL SQL shell connected to the Zulip - server's database. Beware of changing data; editing data directly + server's database. Beware of changing data; editing data directly with SQL will often not behave correctly because PostgreSQL doesn't know to flush Zulip's caches or notify browsers of changes. - `./manage.py send_custom_email`: Can be used to send an email to a set - of users. The `--help` documents how to run it from a + of users. The `--help` documents how to run it from a `manage.py shell` for use with more complex programmatically computed sets of users. - `./manage.py send_password_reset_email`: Sends password reset email(s) @@ -154,7 +154,7 @@ self-hosted Zulip server: Custom management commands are Python 3 programs that run inside Zulip's context, so that they can access its libraries, database, and -code freely. They can be the best choice when you want to run custom +code freely. They can be the best choice when you want to run custom code that is not permitted by Zulip's security model (and thus can't be done more easily using the [REST API][zulip-api]) and that you might want to run often (and so the interactive `manage.py shell` is diff --git a/docs/production/mobile-push-notifications.md b/docs/production/mobile-push-notifications.md index 4075af1ab4..58326e2412 100644 --- a/docs/production/mobile-push-notifications.md +++ b/docs/production/mobile-push-notifications.md @@ -2,13 +2,13 @@ Zulip's iOS and Android mobile apps support receiving push notifications from Zulip servers to let users know when new messages -have arrived. This is an important feature to having a great +have arrived. This is an important feature to having a great experience using the Zulip mobile apps. For technical reasons (explained below), in order to deliver mobile push notifications in the app store versions of our mobile apps, you will need to register your Zulip server with the Zulip mobile push -notification service. This service will forward push notifications +notification service. This service will forward push notifications generated by your server to the Zulip mobile app automatically. @@ -17,7 +17,7 @@ generated by your server to the Zulip mobile app automatically. Starting with Zulip 1.6 for both Android and iOS, Zulip servers support forwarding push notifications to a central push notification -forwarding service. Accessing this service requires outgoing HTTPS +forwarding service. Accessing this service requires outgoing HTTPS access to the public Internet; if that is restricted by a proxy, you will need to [configure Zulip to use your outgoing HTTP proxy](../production/deployment.html#using-an-outgoing-http-proxy) @@ -29,7 +29,7 @@ You can enable this for your Zulip server as follows: `PUSH_NOTIFICATION_BOUNCER_URL = 'https://push.zulipchat.com'` line in your `/etc/zulip/settings.py` file (i.e. remove the `#` at the start of the line), and [restart your Zulip - server](../production/settings.html#making-changes). If you + server](../production/settings.html#making-changes). If you installed your Zulip server with a version older than 1.6, you'll need to add the line (it won't be there to uncomment). @@ -53,10 +53,10 @@ You can enable this for your Zulip server as follows: you'll each need to log out and log back in again in order to start getting push notifications. -Congratulations! You've successfully set up the service. +Congratulations! You've successfully set up the service. If you'd like to verify that everything is working, you can do the -following. Please follow the instructions carefully: +following. Please follow the instructions carefully: - [Configure mobile push notifications to always be sent][mobile-notifications-always] (normally they're only sent if you're idle, which isn't ideal for @@ -69,7 +69,7 @@ notifications on login. - Hit the home button, so Zulip is running in the background, and then have **another user** send you a **private message** (By default, Zulip only sends push notifications for private messages sent by other -users and messages mentioning you). A push notification should appear +users and messages mentioning you). A push notification should appear in the Android notification area. [mobile-notifications-always]: https://zulip.com/help/test-mobile-notifications @@ -79,7 +79,7 @@ in the Android notification area. Your server's registration includes the server's hostname and contact email address (from `EXTERNAL_HOST` and `ZULIP_ADMINISTRATOR` in `/etc/zulip/settings.py`, aka the `--hostname` and `--email` options -in the installer). You can update your server's registration data by +in the installer). You can update your server's registration data by running `manage.py register_server` again. If you'd like to rotate your server's API key for this service @@ -92,10 +92,10 @@ generate a new `zulip_org_key` and store that new key in Both Google's and Apple's push notification services have a security model that does not support mutually untrusted self-hosted servers -sending push notifications to the same app. In particular, when an +sending push notifications to the same app. In particular, when an app is published to their respective app stores, one must compile into the app a secret corresponding to the server that will be able to -publish push notifications for the app. This means that it is +publish push notifications for the app. This means that it is impossible for a single app in their stores to receive push notifications from multiple, mutually untrusted, servers. @@ -183,18 +183,18 @@ Zulip mobile apps. We don't recommend this path -- patching and shipping a production mobile app can take dozens of hours to set up even for an experienced -developer, and even more time to maintain. And it doesn't provide +developer, and even more time to maintain. And it doesn't provide material privacy benefits -- your organization's push notification data would still go through Apple/Google's servers, just not Kandra -Labs'. Our view is the correct way to optimize for privacy is -end-to-end encryption of push notifications. But in the interest of +Labs'. Our view is the correct way to optimize for privacy is +end-to-end encryption of push notifications. But in the interest of transparency, we document in this section roughly what's involved in doing so. As we discussed above, it is impossible for a single app in their stores to receive push notifications from multiple, mutually -untrusted, servers. The Mobile Push Notification Service is one of -the possible solutions to this problem. The other possible solution +untrusted, servers. The Mobile Push Notification Service is one of +the possible solutions to this problem. The other possible solution is for an individual Zulip server's administrators to build and distribute their own copy of the Zulip mobile apps, hardcoding a key that they possess. @@ -205,8 +205,8 @@ the Zulip mobile apps (and there's nothing the Zulip team can do to eliminate this onerous requirement). The main work is distributing your own copies of the Zulip mobile apps -configured to use APNS/FCM keys that you generate. This is not for -the faint of heart! If you haven't done this before, be warned that +configured to use APNS/FCM keys that you generate. This is not for +the faint of heart! If you haven't done this before, be warned that one can easily spend hundreds of dollars (on things like a DUNS number registration) and a week struggling through the hoops Apple requires to build and distribute an app through the Apple app store, even if @@ -223,7 +223,7 @@ push notifications through the new app is quite straightforward: `/etc/zulip/zulip-secrets.conf` to that key. - Register for a [mobile push notification certificate][apple-docs] - from Apple's developer console. Set `APNS_SANDBOX=False` and + from Apple's developer console. Set `APNS_SANDBOX=False` and `APNS_CERT_FILE` to be the path of your APNS certificate file in `/etc/zulip/settings.py`. - Set the `APNS_TOPIC` and `ZULIP_IOS_APP_ID` settings to the ID for diff --git a/docs/production/multiple-organizations.md b/docs/production/multiple-organizations.md index d1acf7f1fa..87948d9153 100644 --- a/docs/production/multiple-organizations.md +++ b/docs/production/multiple-organizations.md @@ -5,12 +5,12 @@ # Hosting multiple organizations The vast majority of Zulip servers host just a single organization (or -"realm", as the Zulip code calls organizations). This article +"realm", as the Zulip code calls organizations). This article documents what's involved in hosting multiple Zulip organizations on a single server. Throughout this article, we'll assume you're working on a Zulip server -with hostname `zulip.example.com`. You may also find the more +with hostname `zulip.example.com`. You may also find the more [technically focused article on realms](../subsystems/realms.md) to be useful reading. @@ -18,7 +18,7 @@ reading. Zulip's approach for supporting multiple organizations on a single Zulip server is for each organization to be hosted on its own -subdomain. E.g. you'd have `org1.zulip.example.com` and +subdomain. E.g. you'd have `org1.zulip.example.com` and `org2.zulip.example.com`. Web security standards mean that one subdomain per organization is @@ -30,15 +30,15 @@ things: - If you're using Zulip older than 1.7, you'll need to set `REALMS_HAVE_SUBDOMAINS=True` in your `/etc/zulip/settings.py` - file. That setting is the default in 1.7 and later. + file. That setting is the default in 1.7 and later. - Make sure you have SSL certificates for all of the subdomains you're - going to use. If you're using + going to use. If you're using [our Let's Encrypt instructions](ssl-certificates.md), it's easy to just specify multiple subdomains in your certificate request. - If necessary, modify your `nginx` configuration to use your new certificates. - Use `./manage.py generate_realm_creation_link` again to create your - new organization. Review + new organization. Review [the install instructions](install.md) if you need a refresher on how this works. - If you're planning on using GitHub auth or another social @@ -53,11 +53,11 @@ the homepage for the server is a copy of the Zulip homepage. ### SSL certificates You'll need to install an SSL certificate valid for all the -(sub)domains you're using your Zulip server with. You can get an SSL +(sub)domains you're using your Zulip server with. You can get an SSL certificate covering several domains for free by using [our Certbot wrapper tool](../production/ssl-certificates.html#after-zulip-is-already-installed), though if you're going to host a large number of organizations, you -may want to get a wildcard certificate. You can also get a wildcard +may want to get a wildcard certificate. You can also get a wildcard certificate for [free using Certbot](https://community.letsencrypt.org/t/getting-wildcard-certificates-with-certbot/56285), but because of the stricter security checks for acquiring a wildcard @@ -86,7 +86,7 @@ into the database. ### The root domain Most Zulip servers host a single Zulip organization on the root domain -(e.g. `zulip.example.com`). The way this is implemented internally +(e.g. `zulip.example.com`). The way this is implemented internally involves the organization having the empty string (`''`) as its "subdomain". @@ -95,7 +95,7 @@ on subdomains (e.g. `subdivision.zulip.example.com`), but this only works well if there are no users in common between the two organizations, because the auth cookies for the root domain are visible to the subdomain (so it's not possible for a single -browser/client to be logged into both). So we don't recommend that +browser/client to be logged into both). So we don't recommend that configuration. ### Authentication @@ -103,7 +103,7 @@ configuration. Many of Zulip's supported authentication methods (Google, GitHub, SAML, etc.) can require providing the third-party authentication provider with a whitelist of callback URLs to your Zulip server (or -even a single URL). For those vendors that support a whitelist, you +even a single URL). For those vendors that support a whitelist, you can provide the callback URLs for each of your Zulip organizations. The cleaner solution is to register a special subdomain, e.g. @@ -118,10 +118,10 @@ avoid confusion as to why there's an extra realm when inspecting the Zulip database. Every Zulip server comes with 1 realm that isn't created by users: the -`zulipinternal` realm. By default, this realm only contains the Zulip "system -bots". You can get a list of these on your system via +`zulipinternal` realm. By default, this realm only contains the Zulip "system +bots". You can get a list of these on your system via `./scripts/get-django-setting INTERNAL_BOTS`, but this is where bots -like "Notification Bot", "Welcome Bot", etc. exist. In the future, +like "Notification Bot", "Welcome Bot", etc. exist. In the future, we're considering moving these bots to exist in every realm, so that we wouldn't need the system realm anymore. diff --git a/docs/production/password-strength.md b/docs/production/password-strength.md index 508e4d5a22..431b57301c 100644 --- a/docs/production/password-strength.md +++ b/docs/production/password-strength.md @@ -8,10 +8,10 @@ When a user tries to set a password, we use [zxcvbn][zxcvbn] to check that it isn't a weak one. See discussion in [our main docs for server -admins](../production/security-model.html#passwords). This doc explains in more +admins](../production/security-model.html#passwords). This doc explains in more detail how we set the default threshold (`PASSWORD_MIN_GUESSES`) we use. -First, read the doc section there. (It's short.) +First, read the doc section there. (It's short.) Then, the CACM article ["Passwords and the Evolution of Imperfect Authentication"][BHOS15] is comprehensive, educational, and readable, @@ -19,7 +19,7 @@ and is especially recommended. The CACM article is convincing that password requirements should be set to make passwords withstand an online attack, but not an offline -one. Offline attacks are much less common, and there is a wide gap in +one. Offline attacks are much less common, and there is a wide gap in the level of password strength required to beat them vs that for online attacks -- and therefore in the level of user frustration that such a requirement would cause. @@ -38,7 +38,7 @@ just after 10k guesses, and grows steadily thereafter. Moreover, the [Yahoo study][Bon12] shows that resistance to even 1M guesses is more than nearly half of users accomplish with a freely -chosen password, and 100k is too much for about 20%. (See Figure 6.) +chosen password, and 100k is too much for about 20%. (See Figure 6.) It doesn't make sense for a Zulip server to try to educate or push so many users far beyond the security practices they're accustomed to; in the few environments where users can be expected to work much harder @@ -49,7 +49,7 @@ auth in Zulip entirely in favor of using that. Our threshold of 10k guesses provides significant protection against online attacks, and quite strong protection with appropriate -rate-limiting. On the other hand it stays within the range where +rate-limiting. On the other hand it stays within the range where zxcvbn rarely underestimates the strength of a password too severely, and only about 10% of users do worse than this without prompting. diff --git a/docs/production/postgresql.md b/docs/production/postgresql.md index 3fab9893bf..7e4d984d62 100644 --- a/docs/production/postgresql.md +++ b/docs/production/postgresql.md @@ -2,12 +2,12 @@ PostgreSQL database details ========================= Starting with Zulip 3.0, Zulip supports a range of PostgreSQL -versions. PostgreSQL 13 is the current default for new installations; +versions. PostgreSQL 13 is the current default for new installations; PostgreSQL 10, 11, and 12 are all supported. Previous versions of Zulip used whatever version of PostgreSQL was included with the base operating system (E.g. PostgreSQL 12 on Ubuntu -Focal, 10 on Ubuntu Bionic, and 9.6 on Ubuntu Xenial). We recommend +Focal, 10 on Ubuntu Bionic, and 9.6 on Ubuntu Xenial). We recommend that installations currently using older PostgreSQL releases [upgrade to PostgreSQL 13][upgrade-postgresql], as we may drop support for older PostgreSQL in a future release. @@ -89,7 +89,7 @@ To stop a runaway query, you can run `SELECT pg_terminate_backend(pid int)` as the 'postgres' user. The former cancels the backend's current query and the latter terminates the backend process. They are implemented by sending SIGINT and -SIGTERM to the processes, respectively. We recommend against sending +SIGTERM to the processes, respectively. We recommend against sending a PostgreSQL process SIGKILL. Doing so will cause the database to kill all current connections, roll back any pending transactions, and enter recovery mode. @@ -144,9 +144,9 @@ pg_ctlcluster does. #### PostgreSQL vacuuming alerts The `autovac_freeze` PostgreSQL alert from `check_postgres` is -particularly important. This alert indicates that the age (in terms +particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is -getting close to the `autovacuum_freeze_max_age` setting. When the +getting close to the `autovacuum_freeze_max_age` setting. When the oldest XID hits that age, PostgreSQL will force a VACUUM operation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of the oldest XID reached 2 billion, diff --git a/docs/production/requirements.md b/docs/production/requirements.md index aa8aa3ca40..846669af7f 100644 --- a/docs/production/requirements.md +++ b/docs/production/requirements.md @@ -19,9 +19,9 @@ For details on each of these requirements, see below. The installer expects Zulip to be the **only thing** running on the system; it will install system packages with `apt` (like Nginx, -PostgreSQL, and Redis) and configure them for its own use. We +PostgreSQL, and Redis) and configure them for its own use. We strongly recommend using either a fresh machine instance in a cloud -provider, a fresh VM, or a dedicated machine. If you decide to +provider, a fresh VM, or a dedicated machine. If you decide to disregard our advice and use a server that hosts other services, we can't support you, but [we do have some notes on issues you'll encounter](install-existing-server.md). @@ -29,7 +29,7 @@ can't support you, but #### Operating system Ubuntu 20.04 Focal, 18.04 Bionic, and Debian 10 Buster are supported -for running Zulip in production. 64-bit is recommended. We recommend +for running Zulip in production. 64-bit is recommended. We recommend installing on the newest supported OS release you're comfortable with, to save a bit of future work [upgrading the operating system][upgrade-os]. @@ -54,12 +54,12 @@ https://help.ubuntu.com/community/Repositories/Ubuntu minimum of **2 CPUs** and **4GB RAM**. For installations with fewer users, 1 CPU and 2GB RAM is sufficient. We strongly recommend against installing with less than 2GB of RAM, as you will likely experience - out of memory issues installing dependencies. We recommend against + out of memory issues installing dependencies. We recommend against using highly CPU-limited servers like the AWS `t2` style instances for organizations with hundreds of users (active or no). - Disk space: You'll need at least 10GB of free disk space for a - server with dozens of users. We recommend using an SSD and avoiding + server with dozens of users. We recommend using an SSD and avoiding cloud storage backends that limit the IOPS per second, since the disk is primarily used for the Zulip database. @@ -72,19 +72,19 @@ on hardware requirements for larger organizations. [configurable](../production/deployment.html#using-an-alternate-port)) from the networks where your users are (usually, the public Internet). -- Incoming port 80 access (optional). Zulip only serves content over +- Incoming port 80 access (optional). Zulip only serves content over HTTPS, and will redirect HTTP requests to HTTPS. - Incoming port 25 if you plan to enable Zulip's [incoming email integration](../production/email-gateway.md). - Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage image and website previews and mobile - push notifications. Outgoing Internet access is not required if you + push notifications. Outgoing Internet access is not required if you [disable those features](https://zulip.com/help/allow-image-link-previews). - Outgoing SMTP access (usually port 587) to your [SMTP server](../production/email.md) so that Zulip can send emails. - A domain name (e.g. `zulip.example.com`) that your users will use to - access the Zulip server. In order to generate valid SSL + access the Zulip server. In order to generate valid SSL certificates [with Certbot][doc-certbot], and to enable other services such as Google authentication, public DNS name is simpler, but Zulip can be configured to use a non-public domain or even an IP @@ -106,7 +106,7 @@ on hardware requirements for larger organizations. #### SSL certificate Your Zulip server will need an SSL certificate for the domain name it -uses. For most Zulip servers, the recommended (and simplest) way to +uses. For most Zulip servers, the recommended (and simplest) way to get this is to just [use the `--certbot` option][doc-certbot] in the Zulip installer, which will automatically get a certificate for you and keep it renewed. @@ -126,7 +126,7 @@ certificate documentation](ssl-certificates.md). - Outgoing email (SMTP) credentials that Zulip can use to send outgoing emails to users (e.g. email address confirmation emails during the signup process, message notification emails, password - reset, etc.). If you don't have an existing outgoing SMTP solution, + reset, etc.). If you don't have an existing outgoing SMTP solution, read about [free outgoing SMTP options and options for prototyping](email.html#free-outgoing-email-services). @@ -139,7 +139,7 @@ Zulip in production](../production/install.md). This section details some basic guidelines for running a Zulip server for larger organizations (especially >1000 users or 500+ daily active -users). Zulip's resource needs depend mainly on 3 parameters: +users). Zulip's resource needs depend mainly on 3 parameters: - daily active users (e.g. number of employees if everyone's an employee) - total user accounts (can be much larger) @@ -147,15 +147,15 @@ employee) In the following, we discuss a configuration with at most two types of servers: application servers (running Django, Tornado, RabbitMQ, -Redis, Memcached, etc.) and database servers. Of the application -server services, Django dominates the resource requirements. One can +Redis, Memcached, etc.) and database servers. Of the application +server services, Django dominates the resource requirements. One can run every service on its own system (as [docker-zulip](https://github.com/zulip/docker-zulip) does) but for -most use cases, there's little scalability benefit to doing so. See +most use cases, there's little scalability benefit to doing so. See [deployment options](../production/deployment.md) for details on installing Zulip with a dedicated database server. -- **Dedicated database**. For installations with hundreds of daily +- **Dedicated database**. For installations with hundreds of daily active users, we recommend using a [remote PostgreSQL database](postgresql.md), but it's not required. @@ -170,23 +170,23 @@ installing Zulip with a dedicated database server. - **CPU:** The Zulip application server's CPU usage is heavily optimized due to extensive work on optimizing the performance of - requests for latency reasons. Because most servers with sufficient + requests for latency reasons. Because most servers with sufficient RAM have sufficient CPU resources, CPU requirements are rarely an - issue. For larger installations with a dedicated database, we + issue. For larger installations with a dedicated database, we recommend high-CPU instances for the application server and a database-optimized (usually low CPU, high memory) instance for the database. - **Disk for application server:** We recommend using [the S3 file - uploads backend][s3-uploads] to store uploaded files at scale. With + uploads backend][s3-uploads] to store uploaded files at scale. With the S3 backend configuration, we recommend 50GB of disk for the OS, - Zulip software, logs and scratch/free space. Disk needs when + Zulip software, logs and scratch/free space. Disk needs when storing uploads locally -- **Disk for database:** SSD disk is highly recommended. For +- **Disk for database:** SSD disk is highly recommended. For installations where most messages have <100 recipients, 10GB per 1M messages of history is sufficient plus 1GB per 1000 users is - sufficient. If most messages are to public streams with 10K+ users + sufficient. If most messages are to public streams with 10K+ users subscribed (like on chat.zulip.org), add 20GB per (1000 user accounts) per (1M messages to public streams). @@ -199,7 +199,7 @@ installing Zulip with a dedicated database server. - **Disaster recovery:** One can easily run a hot spare application server and a hot spare database (using [PostgreSQL streaming - replication][streaming-replication]). Make sure the hot spare + replication][streaming-replication]). Make sure the hot spare application server has copies of `/etc/zulip` and you're either syncing `LOCAL_UPLOADS_DIR` or using the [S3 file uploads backend][s3-uploads]. @@ -207,7 +207,7 @@ installing Zulip with a dedicated database server. - **Sharding:** Zulip releases do not fully support dividing Tornado traffic for a single Zulip realm/organization between multiple application servers, which is why we recommend a hot spare over - load-balancing. We don't have an easily deployed configuration for + load-balancing. We don't have an easily deployed configuration for load-balancing Tornado within a single organization, and as a result can't currently offer this model outside of enterprise support contracts. diff --git a/docs/production/security-model.md b/docs/production/security-model.md index 7411e5d388..152d043ab0 100644 --- a/docs/production/security-model.md +++ b/docs/production/security-model.md @@ -1,6 +1,6 @@ # Security model -This section attempts to document the Zulip security model. It likely +This section attempts to document the Zulip security model. It likely does not cover every issue; if there are details you're curious about, please feel free to ask questions in [#production help](https://chat.zulip.org/#narrow/stream/31-production-help) on the @@ -19,7 +19,7 @@ announcement). or Zulip database server, or with access to the `zulip` user on a Zulip application server, has complete control over the Zulip installation and all of its data (so they can read messages, modify - history, etc.). It would be difficult or impossible to avoid this, + history, etc.). It would be difficult or impossible to avoid this, because the server needs access to the data to support features expected of a group chat system like the ability to search the entire message history, and thus someone with control over the @@ -28,7 +28,7 @@ announcement). ## Encryption and authentication - Traffic between clients (web, desktop and mobile) and the Zulip - server is encrypted using HTTPS. By default, all Zulip services + server is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. @@ -37,7 +37,7 @@ announcement). - The preferred way to log in to Zulip is using an SSO solution like Google auth, LDAP, or similar, but Zulip also supports password - authentication. See + authentication. See [the authentication methods documentation](../production/authentication-methods.md) for details on Zulip's available authentication methods. @@ -46,8 +46,8 @@ announcement). Zulip stores user passwords using the standard PBKDF2 algorithm. When the user is choosing a password, Zulip checks the password's -strength using the popular [zxcvbn][zxcvbn] library. Weak passwords -are rejected, and strong passwords encouraged. The minimum password +strength using the popular [zxcvbn][zxcvbn] library. Weak passwords +are rejected, and strong passwords encouraged. The minimum password strength allowed is controlled by two settings in `/etc/zulip/settings.py`: @@ -71,7 +71,7 @@ strength allowed is controlled by two settings in impossible to efficiently do perfectly. For background or when considering an alternate value for this setting, the article ["Passwords and the Evolution of Imperfect Authentication"][BHOS15] - is recommended. The [2016 zxcvbn paper][zxcvbn-paper] adds useful + is recommended. The [2016 zxcvbn paper][zxcvbn-paper] adds useful information about the performance of zxcvbn, and [a large 2012 study of Yahoo users][Bon12] is informative about the strength of the passwords users choose. @@ -99,7 +99,7 @@ strength allowed is controlled by two settings in - Zulip supports both public streams and private streams. - Any non-guest user can join any public stream in the organization, and can view the complete message history of any public stream - without joining the stream. Guests can only access streams that + without joining the stream. Guests can only access streams that another user adds them to. - Organization owners and administrators can see and modify most @@ -159,13 +159,13 @@ strength allowed is controlled by two settings in - Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can - do; for that reason, users should keep their API key safe. Users + do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. - To properly remove a user's access to a Zulip team, it does not suffice to change their password or deactivate their account in a SSO system, since neither of those prevents authenticating with the - user's API key or those of bots the user has created. Instead, you + user's API key or those of bots the user has created. Instead, you should [deactivate the user's account](https://zulip.com/help/deactivate-or-reactivate-a-user) via Zulip's "Organization settings" interface. @@ -188,7 +188,7 @@ strength allowed is controlled by two settings in notifications, or create other bots). - Bots with the `can_forge_sender` permission can send messages that appear to have been sent by another user. They also have the ability to see the names of all - streams, including private streams. This is important for implementing + streams, including private streams. This is important for implementing integrations like the Jabber, IRC, and Zephyr mirrors. These bots cannot be created by Zulip users, including @@ -197,14 +197,14 @@ strength allowed is controlled by two settings in ## User-uploaded content and user-generated requests -- Zulip supports user-uploaded files. Ideally they should be hosted +- Zulip supports user-uploaded files. Ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com). We support two ways of hosting them: the basic `LOCAL_UPLOADS_DIR` file storage backend, where they are stored in a directory on the Zulip server's filesystem, and the S3 backend, where the files are - stored in Amazon S3. It would not be difficult to add additional + stored in Amazon S3. It would not be difficult to add additional supported backends should there be a need; see `zerver/lib/upload.py` for the full interface. @@ -221,11 +221,11 @@ strength allowed is controlled by two settings in provide additional layers of protection in both backends as well. In the Zulip S3 backend, the random URLs to access files that are - presented to users don't actually host the content. Instead, the S3 + presented to users don't actually host the content. Instead, the S3 backend verifies that the user has a valid Zulip session in the relevant organization (and that has access to a Zulip message linking to the file), and if so, then redirects the browser to a temporary S3 - URL for the file that expires a short time later. In this way, + URL for the file that expires a short time later. In this way, possessing a URL to a secret file in Zulip does not provide unauthorized users with access to that file. @@ -241,7 +241,7 @@ strength allowed is controlled by two settings in servers to fetch images, improving privacy. - By default, Zulip will provide image previews inline in the body of - messages when a message contains a link to an image. You can + messages when a message contains a link to an image. You can control this using the `INLINE_IMAGE_PREVIEW` setting. - Zulip may make outgoing HTTP connections to other servers in a @@ -255,11 +255,11 @@ strength allowed is controlled by two settings in - Mobile push notifications (must be configured to be enabled) - Notably, these first 3 features give end users (limited) control to cause - the Zulip server to make HTTP requests on their behalf. As a result, + the Zulip server to make HTTP requests on their behalf. As a result, Zulip supports routing all outgoing outgoing HTTP requests [through Smokescreen][smokescreen-setup] to ensure that Zulip cannot be used to execute [SSRF attacks][SSRF] against other systems on an - internal corporate network. The default Smokescreen configuration + internal corporate network. The default Smokescreen configuration denies access to all non-public IP addresses, including 127.0.0.1. [SSRF]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery diff --git a/docs/production/settings.md b/docs/production/settings.md index 6a046b7d3d..79db332e8c 100644 --- a/docs/production/settings.md +++ b/docs/production/settings.md @@ -12,7 +12,7 @@ administrators][realm-admin-docs]. [realm-admin-docs]: https://zulip.com/help/getting-your-organization-started-with-zulip This page discusses additional configuration that a system -administrator can do. To change any of the following settings, edit +administrator can do. To change any of the following settings, edit the `/etc/zulip/settings.py` file on your Zulip server, and then restart the server with the following command: ```bash @@ -42,12 +42,12 @@ if there's something you'd like to do but can't figure out how to. `EXTERNAL_HOST`: the user-accessible domain name for your Zulip installation (i.e., what users will type in their web browser). This should of course match the DNS name you configured to point to your -server and for which you configured SSL certificates. If you passed +server and for which you configured SSL certificates. If you passed `--hostname` to the installer, this will be prefilled with that value. `ZULIP_ADMINISTRATOR`: the email address of the person or team maintaining this installation and who will get support and error -emails. If you passed `--email` to the installer, this will be +emails. If you passed `--email` to the installer, this will be prefilled with that value. ### Authentication backends @@ -68,14 +68,14 @@ them. The Zulip apps expect to be talking to servers with a properly signed SSL certificate, in most cases and will not accept a -self-signed certificate. You should get a proper SSL certificate +self-signed certificate. You should get a proper SSL certificate before testing the apps. Because of how Google and Apple have architected the security model of their push notification protocols, the Zulip mobile apps for [iOS](https://itunes.apple.com/us/app/zulip/id1203036395) and [Android](https://play.google.com/store/apps/details?id=com.zulipmobile) -can only receive push notifications from a single Zulip server. We +can only receive push notifications from a single Zulip server. We have configured that server to be `push.zulipchat.com`, and offer a [push notification forwarding service](mobile-push-notifications.md) that forwards push notifications through our servers to mobile devices. @@ -85,10 +85,10 @@ and configure this service. ### Terms of Service and Privacy policy Zulip allows you to configure your server's Terms of Service and -Privacy Policy pages (`/terms` and `/privacy`, respectively). You can +Privacy Policy pages (`/terms` and `/privacy`, respectively). You can use the `TERMS_OF_SERVICE` and `PRIVACY_POLICY` settings to configure -the path to your server's policies. The syntax is Markdown (with -support for included HTML). A good approach is to use paths like +the path to your server's policies. The syntax is Markdown (with +support for included HTML). A good approach is to use paths like `/etc/zulip/terms.md`, so that it's easy to back up your policy configuration along with your other Zulip server configuration. diff --git a/docs/production/ssl-certificates.md b/docs/production/ssl-certificates.md index e02a4472e4..347cb8f715 100644 --- a/docs/production/ssl-certificates.md +++ b/docs/production/ssl-certificates.md @@ -16,8 +16,8 @@ files into place at the following paths: Your certificate file should contain not only your own certificate but its **full chain, including any intermediate certificates** used by -your certificate authority (CA). See the [nginx -documentation][nginx-chains] for details on what this means. If +your certificate authority (CA). See the [nginx +documentation][nginx-chains] for details on what this means. If you're missing part of the chain, your server may work with some browsers, but not others and not the Zulip mobile and desktop apps. The desktop apps support [configuring a custom CA][desktop-certs] to @@ -33,7 +33,7 @@ browsers ignore errors that others don't. Two good tests include: - If your server is accessible from the public Internet, use the [SSL - Labs tester][ssllabs-tester]. Be sure to check for "Chain issues"; + Labs tester][ssllabs-tester]. Be sure to check for "Chain issues"; if any, your certificate file is missing intermediate certificates. - Alternatively, run a command like `curl -SsI https://zulip.example.com` @@ -49,11 +49,11 @@ Two good tests include: [Let's Encrypt](https://letsencrypt.org/) is a free, completely automated CA launched in 2016 to help make HTTPS routine for the -entire Web. Zulip offers a simple automation for +entire Web. Zulip offers a simple automation for [Certbot](https://certbot.eff.org/), a Let's Encrypt client, to get SSL certificates from Let's Encrypt and renew them automatically. -We recommend most Zulip servers use Certbot. You'll want something +We recommend most Zulip servers use Certbot. You'll want something else if: - you have an existing workflow for managing SSL certificates that you prefer; @@ -72,7 +72,7 @@ To enable the Certbot automation when first installing Zulip, just pass the `--certbot` flag when [running the install script][doc-install-script]. The `--hostname` and `--email` options are required when using -`--certbot`. You'll need the hostname to be a real DNS name, and the +`--certbot`. You'll need the hostname to be a real DNS name, and the Zulip server machine to be reachable by that name from the public Internet. @@ -100,11 +100,11 @@ When the Certbot automation in Zulip is first enabled, by either method, it creates an account for the server at the Let's Encrypt CA; requests a certificate for the given hostname; proves to the CA that the server controls the website at that hostname; and is then given a -certificate. (For details, refer to +certificate. (For details, refer to [Let's Encrypt](https://letsencrypt.org/how-it-works/).) Then it records a flag in `/etc/zulip/zulip.conf` saying Certbot is in -use and should be auto-renewed. A cron job checks that flag, then +use and should be auto-renewed. A cron job checks that flag, then checks if any certificates are due for renewal, and if they are (so approximately once every 60 days), repeats the process of request, prove, get a fresh certificate. @@ -113,10 +113,10 @@ prove, get a fresh certificate. ## Self-signed certificate If you aren't able to use Certbot, you can generate a self-signed SSL -certificate. This can be convenient for testing, but isn't -recommended for production, as it is insecure. The Zulip desktop and +certificate. This can be convenient for testing, but isn't +recommended for production, as it is insecure. The Zulip desktop and mobile apps will not connect to a server if they cannot validate its -SSL certificate. The desktop apps support [configuring a custom +SSL certificate. The desktop apps support [configuring a custom certificate authority][desktop-certs] to allow validation of an internal certificate. @@ -146,7 +146,7 @@ service nginx reload ### The Android app can't connect to the server -This is most often caused by an incomplete certificate chain. See +This is most often caused by an incomplete certificate chain. See discussion in the [Manual install](#manual-install) section above. @@ -156,9 +156,9 @@ This can be caused by a server set up to support only TLS 1.1 or older (including TLS 1.0, SSL 3, or SSL 2.) TLS 1.2 has been a standard for over 10 years, and all modern web -server software supports it. Starting in early 2020, all major +server software supports it. Starting in early 2020, all major browsers [will *require* TLS 1.2 or later][tls12-required-news], and -will refuse to connect over TLS 1.1 or older. And on iOS, Apple [has +will refuse to connect over TLS 1.1 or older. And on iOS, Apple [has since iOS 9][apple-ats] required TLS 1.2 for all connections made by apps, unless the app specifically opts into lower security. @@ -170,7 +170,7 @@ to check what TLS versions it supports is the [SSL Labs tester][ssllabs-tester]. To resolve this issue, update your server to support TLS 1.2, -and preferably also TLS 1.3. For nginx, see [the `ssl_protocols` +and preferably also TLS 1.3. For nginx, see [the `ssl_protocols` directive][nginx-doc-protocols] in your configuration. [nginx-doc-protocols]: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols @@ -188,14 +188,14 @@ configuration. The issue is that Android 7.0 supports only the curve `secp256r1` when doing elliptic-curve cryptography for TLS, and not other curves like -`secp384r1` or `secp512r1`. If your server's TLS/SSL configuration +`secp384r1` or `secp512r1`. If your server's TLS/SSL configuration offers only other curves, then Android 7.0 clients will be unable to connect. By default `nginx` (and therefore a Zulip server) offers the -`secp256r1` curve among others, and so everything works. You can +`secp256r1` curve among others, and so everything works. You can control the offered curves with `ssl_ecdh_curve` in the `nginx` -configuration on your server. See [nginx docs][nginx-doc-curve] for +configuration on your server. See [nginx docs][nginx-doc-curve] for details. [nginx-doc-curve]: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve @@ -207,9 +207,9 @@ cause: later. - If your server is reachable from the public Internet, use the [SSL - Labs tester][ssllabs-tester]. Under "Cipher Suites" you may see + Labs tester][ssllabs-tester]. Under "Cipher Suites" you may see lines beginning with `TLS_ECDHE`, for cipher suites which use - elliptic-curve cryptography. These lines will have further text + elliptic-curve cryptography. These lines will have further text like `ECDH secp256r1` or `ECDH secp384r1`, which identifies specific - elliptic curves your server offers to use. This issue applies if + elliptic curves your server offers to use. This issue applies if your server does not offer `secp256r1`. diff --git a/docs/production/troubleshooting.md b/docs/production/troubleshooting.md index 6bfc271b5a..6bb19ba027 100644 --- a/docs/production/troubleshooting.md +++ b/docs/production/troubleshooting.md @@ -11,7 +11,7 @@ overview](../overview/architecture-overview.md), particularly the understand the many services Zulip uses. If you encounter issues while running Zulip, take a look at Zulip's logs, which -are located in `/var/log/zulip/`. That directory contains one log file for +are located in `/var/log/zulip/`. That directory contains one log file for each service, plus `errors.log` (has all errors), `server.log` (has logs from the Django and Tornado servers), and `workers.log` (has combined logs from the queue workers). @@ -21,7 +21,7 @@ on this page includes details about how to fix common issues with Zulip services If you run into additional problems, [please report them](https://github.com/zulip/zulip/issues) so that we can update -this page! The Zulip installation scripts logs its full output to +this page! The Zulip installation scripts logs its full output to `/var/log/zulip/install.log`, so please include the context for any tracebacks from that log. @@ -63,10 +63,10 @@ zulip-workers:zulip-events-user-presence RUNNING pid 21 If you see any services showing a status other than `RUNNING`, or you see an uptime under 5 seconds (which indicates it's crashing immediately after startup and repeatedly restarting), that service -isn't running. If you don't see relevant logs in +isn't running. If you don't see relevant logs in `/var/log/zulip/errors.log`, check the log file declared via `stdout_logfile` for that service's entry in -`/etc/supervisor/conf.d/zulip.conf` for details. Logs only make it to +`/etc/supervisor/conf.d/zulip.conf` for details. Logs only make it to `/var/log/zulip/errors.log` once a service has started fully. ### Restarting services with `supervisorctl restart all` @@ -103,13 +103,13 @@ application: - memcached If one of these services is not installed or functioning correctly, -Zulip will not work. Below we detail some common configuration +Zulip will not work. Below we detail some common configuration problems and how to resolve them: - If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did - not provide SSL certificates. To fix this, configure them properly + not provide SSL certificates. To fix this, configure them properly and then run: ```bash service nginx restart @@ -154,7 +154,7 @@ those connections throwing errors. Zulip is designed to recover from system service downtime by creating new connections once the system service is back up, so the Zulip -outage will end once the system service finishes restarting. But +outage will end once the system service finishes restarting. But you'll get a bunch of error emails during the system service outage whenever one of the Zulip server's ~20 workers attempts to access the system service. @@ -191,11 +191,11 @@ Unattended-Upgrade::Package-Blacklist { ## Monitoring -Chat is mission-critical to many organizations. This section contains +Chat is mission-critical to many organizations. This section contains advice on monitoring your Zulip server to minimize downtime. First, we should highlight that Zulip sends Django error emails to -`ZULIP_ADMINISTRATOR` for any backend exceptions. A properly +`ZULIP_ADMINISTRATOR` for any backend exceptions. A properly functioning Zulip server shouldn't send any such emails, so it's worth reporting/investigating any that you do see. @@ -205,16 +205,16 @@ standard stuff: - Basic host health monitoring for issues running out of disk space, especially for the database and where uploads are stored. - Service uptime and standard monitoring for the [services Zulip - depends on](#troubleshooting-services). Most monitoring software + depends on](#troubleshooting-services). Most monitoring software has standard plugins for Nginx, PostgreSQL, Redis, RabbitMQ, and memcached, and those will work well with Zulip. - `supervisorctl status` showing all services `RUNNING`. - Checking for processes being OOM killed. Beyond that, Zulip ships a few application-specific end-to-end health -checks. The Nagios plugins `check_send_receive_time`, +checks. The Nagios plugins `check_send_receive_time`, `check_rabbitmq_queues`, and `check_rabbitmq_consumers` are generally -sufficient to point to the cause of any Zulip production issue. See +sufficient to point to the cause of any Zulip production issue. See the next section for details. ### Nagios configuration @@ -226,14 +226,14 @@ tarballs). The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirectories -under `/usr/lib/nagios/plugins/`. The following is a summary of the +under `/usr/lib/nagios/plugins/`. The following is a summary of the useful Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: - `check_send_receive_time`: Sends a test message through the system between two bot users to check that end-to-end message sending - works. An effective end-to-end check for Zulip's Django and Tornado + works. An effective end-to-end check for Zulip's Django and Tornado systems being healthy. - `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective checks for Zulip's RabbitMQ-based queuing systems being healthy. @@ -265,5 +265,5 @@ encouraged! As a measure to mitigate the potential impact of any future memory leak bugs in one of the Zulip daemons, Zulip service automatically -restarts itself every Sunday early morning. See +restarts itself every Sunday early morning. See `/etc/cron.d/restart-zulip` for the precise configuration. diff --git a/docs/production/upgrade-or-modify.md b/docs/production/upgrade-or-modify.md index d95885e729..8599b178da 100644 --- a/docs/production/upgrade-or-modify.md +++ b/docs/production/upgrade-or-modify.md @@ -17,7 +17,7 @@ This page explains how to upgrade, patch, or modify Zulip, including: Note that there are additional instructions if you're [using docker-zulip][docker-upgrade], have [patched Zulip](#modifying-zulip), or have [modified Zulip-managed configuration -files](#preserving-local-changes-to-service-configuration-files). To upgrade +files](#preserving-local-changes-to-service-configuration-files). To upgrade to a new Zulip release: 1. Read the [upgrade notes](../overview/changelog.html#upgrade-notes) @@ -56,7 +56,7 @@ Upgrading will result in brief downtime for the service, which should be under 30 seconds unless there is an expensive database migration involved (these will be documented in the [release notes](../overview/changelog.md), and usually can be avoided with -some care). If downtime is problematic for your organization, +some care). If downtime is problematic for your organization, consider testing the upgrade on a [backup](../production/export-and-import.html#backups) in advance, doing the final upgrade at off hours, or buying a support contract. @@ -69,7 +69,7 @@ run into any issues or need to roll back the upgrade. Zulip supports upgrading a production installation to any commit in a Git repository, which is great for [running pre-release changes from `main`](#applying-changes-from-main) or [maintaining a -fork](#making-changes). The process is simple: +fork](#making-changes). The process is simple: ```bash # Upgrade to an official release @@ -163,15 +163,15 @@ See also the general Zulip server [troubleshooting guide](../production/troubleshooting.md). The upgrade scripts are idempotent, so there's no harm in trying again -after resolving an issue. The most common causes of errors are: +after resolving an issue. The most common causes of errors are: - Networking issues (e.g. your Zulip server doesn't have reliable - Internet access or needs a proxy set up). Fix the networking issue + Internet access or needs a proxy set up). Fix the networking issue and try again. - Especially when using `upgrade-zulip-from-git`, systems with the minimal RAM for running Zulip can run into out-of-memory issues during the upgrade process (generally `tools/webpack` is the step - that fails). You can get past this by shutting down the Zulip + that fails). You can get past this by shutting down the Zulip server with `supervisorctl stop all` to free up RAM before running the upgrade process. @@ -206,7 +206,7 @@ This means that if the new version isn't working, you can quickly downgrade to the old version by running `/home/zulip/deployments/last/scripts/restart-server`, or to an earlier previous version by running -`/home/zulip/deployments/DATE/scripts/restart-server`. The +`/home/zulip/deployments/DATE/scripts/restart-server`. The `restart-server` script stops any running Zulip server, and starts the version corresponding to the `restart-server` path you call. @@ -228,9 +228,9 @@ configuration. That said, Zulip's configuration files are designed to be flexible enough for a wide range of installations, from a small self-hosted -system to Zulip Cloud. Before making local changes to a configuration +system to Zulip Cloud. Before making local changes to a configuration file, first check whether there's an option supported by -`/etc/zulip/zulip.conf` for the customization you need. And if you +`/etc/zulip/zulip.conf` for the customization you need. And if you need to make local modifications, please report the issue so that we can make the Zulip Puppet configuration flexible enough to handle your setup. @@ -267,7 +267,7 @@ instructions for other supported platforms. ``` 3. Switch to the root user and upgrade the operating system using the - OS's standard tooling. E.g. for Ubuntu, this means running + OS's standard tooling. E.g. for Ubuntu, this means running `do-release-upgrade` and following the prompts until it completes successfully: @@ -283,7 +283,7 @@ instructions for other supported platforms. When `do-release-upgrade` asks you how to upgrade configuration files for services that Zulip manages like Redis, PostgreSQL, Nginx, and memcached, the best choice is `N` to keep the - currently installed version. But it's not important; the next + currently installed version. But it's not important; the next step will re-install Zulip's configuration in any case. 4. As root, upgrade the database to the latest version of PostgreSQL: @@ -295,7 +295,7 @@ instructions for other supported platforms. 5. Ubuntu 20.04 has a different version of the low-level glibc library, which affects how PostgreSQL orders text data (known as "collations"); this corrupts database indexes that rely on - collations. Regenerate the affected indexes by running: + collations. Regenerate the affected indexes by running: ```bash /home/zulip/deployments/current/scripts/setup/reindex-textual-data --force @@ -319,7 +319,7 @@ instructions for other supported platforms. ### Upgrading from Ubuntu 16.04 Xenial to 18.04 Bionic -1. Upgrade your server to the latest Zulip `2.1.x` release. You can +1. Upgrade your server to the latest Zulip `2.1.x` release. You can only upgrade to Zulip 3.0 and newer after completing this process, since newer releases don't support Ubuntu 16.04 Xenial. @@ -367,7 +367,7 @@ instructions for other supported platforms. ### Upgrading from Ubuntu 14.04 Trusty to 16.04 Xenial -1. Upgrade your server to the latest Zulip `2.0.x` release. You can +1. Upgrade your server to the latest Zulip `2.0.x` release. You can only upgrade to Zulip `2.1.x` and newer after completing this process, since newer releases don't support Ubuntu 14.04 Trusty. @@ -410,7 +410,7 @@ instructions for other supported platforms. ### Upgrading from Debian Stretch to Debian Buster -1. Upgrade your server to the latest Zulip `2.1.x` release. You can +1. Upgrade your server to the latest Zulip `2.1.x` release. You can only upgrade to Zulip 3.0 and newer after completing this process, since newer releases don't support Ubuntu Debian Stretch. @@ -423,7 +423,7 @@ instructions for other supported platforms. When prompted for you how to upgrade configuration files for services that Zulip manages like Redis, PostgreSQL, Nginx, and memcached, the best choice is `N` to keep the - currently installed version. But it's not important; the next + currently installed version. But it's not important; the next step will re-install Zulip's configuration in any case. 4. As root, upgrade the database installation and OS configuration to @@ -461,7 +461,7 @@ instructions for other supported platforms. 7. Debian Buster has a different version of the low-level glibc library, which affects how PostgreSQL orders text data (known as "collations"); this corrupts database indexes that rely on - collations. Regenerate the affected indexes by running: + collations. Regenerate the affected indexes by running: ```bash /home/zulip/deployments/current/scripts/setup/reindex-textual-data --force @@ -476,7 +476,7 @@ instructions for other supported platforms. ## Upgrading PostgreSQL Starting with Zulip 3.0, we use the latest available version of -PostgreSQL at installation time (currently version 13). Upgrades to +PostgreSQL at installation time (currently version 13). Upgrades to the version of PostgreSQL are no longer linked to upgrades of the distribution; that is, you may opt to upgrade to PostgreSQL 13 while running Ubuntu 18.04 Bionic. @@ -518,7 +518,7 @@ confirm everything is working correctly. ## Modifying Zulip Zulip is 100% free and open source software, and you're welcome to -modify it! This section explains how to make and maintain +modify it! This section explains how to make and maintain modifications in a safe and convenient fashion. If you do modify Zulip and then report an issue you see in your @@ -538,7 +538,7 @@ section](#applying-changes-from-main). ## Making changes One way to modify Zulip is to just edit files under -`/home/zulip/deployments/current` and then restart the server. This +`/home/zulip/deployments/current` and then restart the server. This can work OK for testing small changes to Python code or shell scripts. But we don't recommend this approach for maintaining changes because: @@ -555,16 +555,16 @@ But we don't recommend this approach for maintaining changes because: Instead, we recommend the following GitHub-based workflow (see [our Git guide][git-guide] if you need a primer): -- Decide where you're going to edit Zulip's code. We recommend [using +- Decide where you're going to edit Zulip's code. We recommend [using the Zulip development environment](../development/overview.md) on a desktop or laptop as it will make it extremely convenient for you - to test your changes without deploying them in production. But if + to test your changes without deploying them in production. But if your changes are small or you're OK with risking downtime, you don't strictly need it; you just need an environment with Git installed. -- **Important**. Determine what Zulip version you're running on your - server. You can check by inspecting `ZULIP_VERSION` in +- **Important**. Determine what Zulip version you're running on your + server. You can check by inspecting `ZULIP_VERSION` in `/home/zulip/deployments/current/version.py` (we'll use `2.0.4` - below). If you apply your changes to the wrong version of Zulip, + below). If you apply your changes to the wrong version of Zulip, it's likely to fail and potentially cause downtime. - [Fork and clone][fork-clone] the [zulip/zulip][] repository on [GitHub](https://github.com). @@ -600,10 +600,10 @@ across future Zulip releases. ### Upgrading to future releases -Eventually, you'll want to upgrade to a new Zulip release. If your +Eventually, you'll want to upgrade to a new Zulip release. If your changes were integrated into that Zulip release or are otherwise no longer needed, you can just [upgrade as -usual](#upgrading-to-a-release). If you [upgraded to +usual](#upgrading-to-a-release). If you [upgraded to `main`](#upgrading-to-main); review that section again; new maintenance releases are likely "older" than your current installation and you might need to upgrade to `main` again rather than to the @@ -611,7 +611,7 @@ new maintenance release. Otherwise, you'll need to update your branch by rebasing your changes (starting from a [clone][fork-clone] of the [zulip/zulip][] -repository). The example below assumes you have a branch off of 2.0.4 +repository). The example below assumes you have a branch off of 2.0.4 and want to upgrade to 2.1.0. ```bash @@ -649,12 +649,12 @@ different from the above: If you are experiencing an issue that has already been fixed by the Zulip development community, and you'd like to get the fix now, you -have a few options. There are two possible ways you might get those +have a few options. There are two possible ways you might get those fixes on your local Zulip server without waiting for an official release. ### Applying a small change -Many bugs have small/simple fixes. In this case, you can use the Git +Many bugs have small/simple fixes. In this case, you can use the Git workflow [described above](#making-changes), using: ```bash @@ -671,12 +671,12 @@ cherry-picking arbitrary commits if the issues don't also affect The exception to this rule is when we ask or encourage a user to apply a change to their production system to help verify the fix resolves -the issue for them. You can expect the Zulip community to be +the issue for them. You can expect the Zulip community to be responsive in debugging any problems caused by a patch we asked you to apply. Also, consider asking whether a small fix that is important to you can -be added to the current stable release branch (E.g. `2.1.x`). In +be added to the current stable release branch (E.g. `2.1.x`). In addition to scheduling that change for Zulip's next bug fix release, we support changes in stable release branches as though they were released. @@ -684,15 +684,15 @@ released. ### Upgrading to `main` Many Zulip servers (including chat.zulip.org and zulip.com) upgrade to -`main` on a regular basis to get the latest features. Before doing +`main` on a regular basis to get the latest features. Before doing so, it's important to understand how to happily run a server based on `main`. For background, it's backporting arbitrary patches from `main` to an -older version requires some care. Common issues include: +older version requires some care. Common issues include: - Changes containing database migrations (new files under - `*/migrations/`), which includes most new features. We + `*/migrations/`), which includes most new features. We don't support applying database migrations out of order. - Changes that are stacked on top of other changes to the same system. - Essentially any patch with hundreds of lines of changes will have @@ -703,29 +703,29 @@ unlikely to succeed without help from the core team via a support contract. If you need an unreleased feature, the best path is usually to -upgrade to Zulip `main` using [upgrade-zulip-from-git][]. Before +upgrade to Zulip `main` using [upgrade-zulip-from-git][]. Before upgrading to `main`, make sure you understand: - In Zulip's version numbering scheme, `main` will always be "newer" than the latest maintenance release (E.g. `3.1` or `2.1.6`) and "older" than the next major release (E.g. `3.0` or `4.0`). - The `main` branch is under very active development; dozens of new - changes are integrated into it on most days. The `main` branch + changes are integrated into it on most days. The `main` branch can have thousands of changes not present in the latest release (all - of which will be included in our next major release). On average + of which will be included in our next major release). On average `main` usually has fewer total bugs than the latest release (because we fix hundreds of bugs in every major release) but it might have some bugs that are more severe than we would consider acceptable for a release. - We deploy `main` to chat.zulip.org and zulip.com on a regular basis (often daily), so it's very important to the project that it - be stable. Most regressions will be minor UX issues or be fixed + be stable. Most regressions will be minor UX issues or be fixed quickly, because we need them to be fixed for Zulip Cloud. - The development community is very interested in helping debug issues that arise when upgrading from the latest release to `main`, since they provide us an opportunity to fix that category of issue before - our next major release. (Much more so than we are in helping folks - debug other custom changes). That said, we cannot make any + our next major release. (Much more so than we are in helping folks + debug other custom changes). That said, we cannot make any guarantees about how quickly we'll resolve an issue to folks without a formal support contract. - We do not support downgrading from `main` to earlier versions, so @@ -735,20 +735,20 @@ upgrading to `main`, make sure you understand: upgrade fails. - Our changelog contains [draft release notes](../overview/changelog.md) available listing major changes - since the last release. The **Upgrade notes** section will always + since the last release. The **Upgrade notes** section will always be current, even if some new features aren't documented. - Whenever we push a security or maintenance release, the changes in that release will always be merged to `main`; so you can get the security fixes by upgrading to `main`. - You can always upgrade from `main` to the next major release when it comes out, using either [upgrade-zulip-from-git][] or the release - tarball. So there's no risk of upgrading to `main` resulting in + tarball. So there's no risk of upgrading to `main` resulting in a system that's not upgradeable back to a normal release. ## Contributing patches Zulip contains thousands of changes submitted by volunteer -contributors like you. If your changes are likely to be of useful to +contributors like you. If your changes are likely to be of useful to other organizations, consider [contributing them](../overview/contributing.md). diff --git a/docs/production/upload-backends.md b/docs/production/upload-backends.md index 2916589205..65f15cac59 100644 --- a/docs/production/upload-backends.md +++ b/docs/production/upload-backends.md @@ -18,11 +18,11 @@ provider supported by the `boto` library). ## S3 backend configuration Here, we document the process for configuring Zulip's S3 file upload -backend. To enable this backend, you need to do the following: +backend. To enable this backend, you need to do the following: 1. In the AWS management console, create a new IAM account (aka API user) for your Zulip server, and two buckets in S3, one for uploaded -files included in messages, and another for user avatars. You need +files included in messages, and another for user avatars. You need two buckets because the "user avatars" bucket is generally configured as world-readable, whereas the "uploaded files" one is not. @@ -68,7 +68,7 @@ as world-readable, whereas the "uploaded files" one is not. (`/home/zulip/deployments/current/scripts/restart-server`). It's simplest to just do this configuration when setting up your Zulip -server for production usage. Note that if you had any existing +server for production usage. Note that if you had any existing uploading files, this process does not upload them to Amazon S3; see [migration instructions](#migrating-from-local-uploads-to-amazon-s3-backend) below for those steps. @@ -78,7 +78,7 @@ below for those steps. ## S3 bucket policy The best way to do the S3 integration with Amazon is to create a new -IAM user just for your Zulip server with limited permissions. For +IAM user just for your Zulip server with limited permissions. For each of the two buckets, you'll want to [add an S3 bucket policy](https://awspolicygen.s3.amazonaws.com/policygen.html) entry that looks something like this: @@ -129,14 +129,14 @@ need a block like this: } ``` -The file-uploads bucket should not be world-readable. See the +The file-uploads bucket should not be world-readable. See the [documentation on the Zulip security model](security-model.md) for details on the security model for uploaded files. ## Migrating from local uploads to Amazon S3 backend As you scale your server, you might want to migrate the uploads from -your local backend to Amazon S3. Follow these instructions, step by +your local backend to Amazon S3. Follow these instructions, step by step, to do the migration. 1. First, [set up the S3 backend](#s3-backend-configuration) in the settings @@ -145,13 +145,13 @@ step, to do the migration. 2. Run `./manage.py transfer_uploads_to_s3`. This will upload all the files from the local uploads directory to Amazon S3. By default, this command runs on 6 parallel processes, since uploading is a - latency-sensitive operation. You can control this parameter using + latency-sensitive operation. You can control this parameter using the `--processes` option. 3. Once the transfer script completes, disable `LOCAL_UPLOADS_DIR`, and restart your server (continuing the last few steps of the S3 backend setup instructions). -Congratulations! Your uploaded files are now migrated to S3. +Congratulations! Your uploaded files are now migrated to S3. **Caveat**: The current version of this tool does not migrate an uploaded organization avatar or logo. diff --git a/docs/production/video-calls.md b/docs/production/video-calls.md index 00c25c293c..178f3246c4 100644 --- a/docs/production/video-calls.md +++ b/docs/production/video-calls.md @@ -40,7 +40,7 @@ follows: 1. Restart the Zulip server with `/home/zulip/deployments/current/scripts/restart-server`. -This enables Zoom support in your Zulip server. Finally, [configure +This enables Zoom support in your Zulip server. Finally, [configure Zoom as the video call provider](https://zulip.com/help/start-a-call) in the Zulip organization(s) where you want to use it. @@ -71,7 +71,7 @@ Server as follows: 3. Restart the Zulip server with `/home/zulip/deployments/current/scripts/restart-server`. -This enables Big Blue Button support in your Zulip server. Finally, [configure +This enables Big Blue Button support in your Zulip server. Finally, [configure Big Blue Button as the video call provider](https://zulip.com/help/start-a-call) in the Zulip organization(s) where you want to use it. diff --git a/docs/subsystems/analytics.md b/docs/subsystems/analytics.md index 1fa3749c6a..067d04e5a8 100644 --- a/docs/subsystems/analytics.md +++ b/docs/subsystems/analytics.md @@ -109,12 +109,12 @@ efficient: then the endpoints to server data to users can be fast. - Doing expensive operations inside the database, rather than fetching data to Python and then sending it back to the database (which can be far - slower if there's a lot of data involved). The Django ORM currently + slower if there's a lot of data involved). The Django ORM currently doesn't support the "insert into .. select" type SQL query that's needed for this, which is why we use raw database queries (which we usually avoid in Zulip) rather than the ORM. - Aggregating where possible to avoid unnecessary queries against the - Message and UserMessage tables. E.g. rather than querying the Message + Message and UserMessage tables. E.g. rather than querying the Message table both to generate sent message counts for each realm and again for each user, we just query for each user, and then add up the numbers for the users to get the totals for the realm. @@ -129,10 +129,10 @@ There are a few types of automated tests that are important for this sort of system: - Most important: Tests for the code path that actually populates data into - the analytics tables. These are most important, because it can be very + the analytics tables. These are most important, because it can be very expensive to fix bugs in the logic that generates these tables (one basically needs to regenerate all of history for those tables), and these - bugs are hard to discover. It's worth taking the time to think about + bugs are hard to discover. It's worth taking the time to think about interesting corner cases and add them to the test suite. - Tests for the backend views code logic for extracting data from the database and serving it to clients. @@ -147,7 +147,7 @@ analytics tests, to make sure it stays that way as we refactor. The system discussed above is designed primarily around the technical problem of showing useful analytics about things where the raw data is -already stored in the database (e.g. Message, UserMessage). This is great +already stored in the database (e.g. Message, UserMessage). This is great because we can always backfill that data to the beginning of time, but of course sometimes one wants to do analytics on things that aren't worth storing every data point for (e.g. activity data, request performance @@ -161,10 +161,10 @@ statistics, etc.). There is currently a reference implementation of a The main testing approach for the /stats page UI is manual testing. For most UI testing, you can visit `/stats/realm/analytics` while logged in as Iago (this is the server administrator view of stats for -a given realm). The only piece that you can't test here is the "Me" -buttons, which won't have any data. For those, you can instead log in +a given realm). The only piece that you can't test here is the "Me" +buttons, which won't have any data. For those, you can instead log in as the `shylock@analytics.ds` in the `analytics` realm and visit -`/stats` there (which is only a bit more work). Note that the +`/stats` there (which is only a bit more work). Note that the `analytics` realm is a shell with no streams, so you'll only want to use it for testing the graphs. @@ -221,9 +221,9 @@ Tips and tricks: ### /activity page - There's a somewhat less developed /activity page, for server - administrators, showing data on all the realms on a server. To + administrators, showing data on all the realms on a server. To access it, you need to have the `is_staff` bit set on your - UserProfile object. You can set it using `manage.py shell` and - editing the UserProfile object directly. A great future project is + UserProfile object. You can set it using `manage.py shell` and + editing the UserProfile object directly. A great future project is to clean up that page's data sources, and make this a documented interface. diff --git a/docs/subsystems/caching.md b/docs/subsystems/caching.md index 4ea951bf72..497a57ec3a 100644 --- a/docs/subsystems/caching.md +++ b/docs/subsystems/caching.md @@ -1,7 +1,7 @@ # Caching in Zulip Like any product with good performance characteristics, Zulip makes -extensive use of caching. This article talks about our caching +extensive use of caching. This article talks about our caching strategy, focusing on how we use `memcached` (since it's the thing people generally think about when they ask about how a server does caching). @@ -9,7 +9,7 @@ caching). ## Backend caching with memcached On the backend, Zulip uses `memcached`, a popular key-value store, for -caching. Our `memcached` caching helps let us optimize Zulip's +caching. Our `memcached` caching helps let us optimize Zulip's performance and scalability, since most requests don't need to talk to the database (which, even for a trivial query with everything on the same machine, usually takes 3-10x as long as a memcached fetch). @@ -37,11 +37,11 @@ Zulip's Django codebase, all one needs to do is call the standard accessor functions for data (like `get_user` or `get_stream` to fetch user and stream objects, or for view code, functions like `access_stream_by_id`, which checks permissions), and everything will -work great. The data fetches automatically benefit from `memcached` +work great. The data fetches automatically benefit from `memcached` caching, since those accessor methods have already been written to transparently use Zulip's memcached caching system, and the developer doesn't need to worry about whether the data returned is up-to-date: -it is. In the following sections, we'll talk about how we make this +it is. In the following sections, we'll talk about how we make this work. As a sidenote, the policy of using these accessor functions wherever @@ -51,7 +51,7 @@ also generally take care of details you might not think about It's amazing how slightly tricky logic that's duplicated in several places invariably ends up buggy in some of those places, and in aggregate we call these accessor functions hundreds of times in -Zulip. But the caching is certainly a nice bonus. +Zulip. But the caching is certainly a nice bonus. ### The core implementation @@ -74,17 +74,17 @@ def get_user(email: str, realm: Realm) -> UserProfile: This decorator implements a pretty classic caching paradigm: - The `user_profile_cache_key` function defines a unique map from a - canonical form of its arguments to a string. These strings are + canonical form of its arguments to a string. These strings are namespaced (the `user_profile:` part) so that they won't overlap with other caches, and encode the arguments so that two uses of this - cache won't overlap. In this case, a hash of the email address and - realm ID are those canonicalized arguments. (The `make_safe_digest` + cache won't overlap. In this case, a hash of the email address and + realm ID are those canonicalized arguments. (The `make_safe_digest` is important to ensure we don't send special characters to - memcached). And we have two versions, depending whether the caller + memcached). And we have two versions, depending whether the caller has access to a `Realm` or just a `realm_id`. - When `get_user` is called, `cache_with_key` will compute the key, and do a Django `cache_get` query for the key (which goes to - memcached). If the key is in the cache, it just returns the value. + memcached). If the key is in the cache, it just returns the value. Otherwise, it fetches the value from the database (using the actual code in the body of `get_user`), and then stores the value back to that memcached key before returning the result to the caller. @@ -101,12 +101,12 @@ huge amount of otherwise very self-similar caching code. The one thing to be really careful with in using `cache_with_key` is that if an item is in the cache, the body of `get_user` (above) is -never called. This means some things that might seem like clever code -reuse are actually a really bad idea. For example: +never called. This means some things that might seem like clever code +reuse are actually a really bad idea. For example: - Don't add a `get_active_user` function that uses the same cache key function as `get_user` (but with a different query that filters our - deactivated users). If one called `get_active_user` to access a + deactivated users). If one called `get_active_user` to access a deactivated user, the right thing would happen, but if you called `get_user` to access that user first, then the `get_active_user` function would happily return the user from the cache, without ever @@ -118,13 +118,13 @@ even if they feature the same objects. ### Cache invalidation after writes The caching strategy described above works pretty well for anything -where the state it's storing is immutable (i.e. never changes). With +where the state it's storing is immutable (i.e. never changes). With mutable state, one needs to do something to ensure that the Python processes don't end up fetching stale data from the cache after a write to the database. We handle this using Django's longstanding -[post_save signals][post-save-signals] feature. Django signals let +[post_save signals][post-save-signals] feature. Django signals let you configure some code to run every time Django does something (for `post_save`, right after any write to the database using Django's `.save()`). @@ -139,11 +139,11 @@ post_save.connect(flush_user_profile, sender=UserProfile) Once this `post_save` hook is registered, whenever one calls `user_profile.save(...)` with a UserProfile object in our Django -project, Django will call the `flush_user_profile` function. Zulip is +project, Django will call the `flush_user_profile` function. Zulip is systematic about using the standard Django `.save()` function for modifying `user_profile` objects (and passing the `update_fields` argument to `.save()` consistently, which encodes which fields on an -object changed). This means that all we have to do is write those +object changed). This means that all we have to do is write those cache-flushing functions correctly, and people writing Zulip code won't need to think about (or even know about!) the caching. @@ -156,7 +156,7 @@ those keys from the cache (if present). Maintaining these flush functions requires some care (every time we add a new cache, we need to look through them), but overall it's a pretty simple algorithm: If the changed data appears in any form in a -given cache key, that cache key needs to be cleared. E.g. the +given cache key, that cache key needs to be cleared. E.g. the `active_user_ids_cache_key` cache for a realm needs to be flushed whenever a new user is created in that realm, or user is deactivated/reactivated, even though it's just a list of IDs and thus @@ -175,8 +175,8 @@ code in Zulip just needs to modify Django model objects and call When upgrading a Zulip server, it's important to avoid having one version of the code interact with cached objects from another version -that has a different data layout. In Zulip, we avoid this through -some clever caching strategies. Each "deployment directory" for Zulip +that has a different data layout. In Zulip, we avoid this through +some clever caching strategies. Each "deployment directory" for Zulip in production has inside it a `var/remote_cache_prefix` file, containing a cache prefix (`KEY_PREFIX` in the code) that is automatically appended to the start of any cache keys accessed by that @@ -188,19 +188,19 @@ from inconsistent versions of the source code / data formats in the cache. ### Automated testing and memcached -For Zulip's `test-backend` unit tests, we use the same strategy. In +For Zulip's `test-backend` unit tests, we use the same strategy. In particular, we just edit `KEY_PREFIX` before each unit test; this means each of the thousands of test cases in Zulip has its own -independent memcached key namespace on each run of the unit tests. As +independent memcached key namespace on each run of the unit tests. As a result, we never have to worry about memcached caching causing problems across multiple tests. -This is a really important detail. It makes it possible for us to do +This is a really important detail. It makes it possible for us to do assertions in our tests on the number of database queries or memcached queries that are done as part of a particular function/route, and have those checks consistently get the same result (those tests are great for catching bugs where we accidentally do database queries in a -loop). And it means one can debug failures in the test suite without +loop). And it means one can debug failures in the test suite without having to consider the possibility that memcached is somehow confusing the situation. @@ -225,8 +225,8 @@ You can run the server with that behavior disabled using ### Performance One thing be careful about with memcached queries is to avoid doing -them in loops (the same applies for database queries!). Instead, one -should use a bulk query. We have a fancy function, +them in loops (the same applies for database queries!). Instead, one +should use a bulk query. We have a fancy function, `generate_bulk_cached_fetch`, which is super magical and handles this for us, with support for a bunch of fancy features like marshalling data before/after going into the cache (e.g. to compress `message` @@ -236,7 +236,7 @@ objects to minimize data transfer between Django and memcached). We generally try to avoid in-process backend caching in Zulip's Django codebase, because every Zulip production installation involves -multiple servers. We do have a few, however: +multiple servers. We do have a few, however: - `per_request_display_recipient_cache`: A cache flushed at the start of every request; this simplifies correctly implementing our goal of not repeatedly fetching the "display recipient" (e.g. stream name) @@ -252,11 +252,11 @@ apps; details like which users exist, with metadata like names and avatars, similar details for streams, recent message history, etc. This data is fetched in the `/register` endpoint (or `page_params` -for the webapp), and kept correct over time. The key to keeping these +for the webapp), and kept correct over time. The key to keeping these state up to date is Zulip's [real-time events system](../subsystems/events-system.md), which allows the server to notify clients whenever state that might be -cached by clients is changed. Clients are responsible for handling +cached by clients is changed. Clients are responsible for handling the events, updating their state, and rerendering any UI components that might display the modified state. diff --git a/docs/subsystems/client.md b/docs/subsystems/client.md index f466ff52c3..53e9440818 100644 --- a/docs/subsystems/client.md +++ b/docs/subsystems/client.md @@ -1,7 +1,7 @@ # Clients in Zulip `zerver.models.Client` is Zulip's analogue of the HTTP User-Agent -header (and is populated from User-Agent). It exists for use in +header (and is populated from User-Agent). It exists for use in analytics and other places to provide human-readable summary data about "which Zulip client" was used for an operation (e.g. was it the Android app, the desktop app, or a bot?). @@ -19,7 +19,7 @@ A `Client` is used to sort messages into client categories such as Generally, integrations in Zulip should declare a unique User-Agent, so that it's easy to figure out which integration is involved when -debugging an issue. For incoming webhook integrations, we do that +debugging an issue. For incoming webhook integrations, we do that convenentialy via the auth decorators (as we will describe shortly); other integrations generally should set the first User-Agent element on their HTTP requests to something of the form diff --git a/docs/subsystems/dependencies.md b/docs/subsystems/dependencies.md index 8b0c37f05e..7599133e43 100644 --- a/docs/subsystems/dependencies.md +++ b/docs/subsystems/dependencies.md @@ -1,22 +1,22 @@ # Provisioning and third-party dependencies Zulip is a large project, with well over 100 third-party dependencies, -and managing them well is essential to the quality of the project. In +and managing them well is essential to the quality of the project. In this document, we discuss the various classes of dependencies that -Zulip has, and how we manage them. Zulip's dependency management has +Zulip has, and how we manage them. Zulip's dependency management has some really nice properties: -- **Fast provisioning**. When switching to a different commit in the +- **Fast provisioning**. When switching to a different commit in the Zulip project with the same dependencies, it takes under 5 seconds to re-provision a working Zulip development environment after - switching. If there are new dependencies, one only needs to wait to + switching. If there are new dependencies, one only needs to wait to download the new ones, not all the pre-existing dependencies. -- **Consistent provisioning**. Every time a Zulip development or +- **Consistent provisioning**. Every time a Zulip development or production environment is provisioned/installed, it should end up using the exactly correct versions of all major dependencies. -- **Low maintenance burden**. To the extent possible, we want to +- **Low maintenance burden**. To the extent possible, we want to avoid manual work and keeping track of things that could be - automated. This makes it easy to keep running the latest versions + automated. This makes it easy to keep running the latest versions of our various dependencies. The purpose of this document is to detail all of Zulip's third-party @@ -25,11 +25,11 @@ dependencies and how we manage their versions. ## Provisioning We refer to "provisioning" as the process of installing and -configuring the dependencies of a Zulip development environment. It's +configuring the dependencies of a Zulip development environment. It's done using `tools/provision`, and the output is conveniently logged by -`var/log/provision.log` to help with debugging. Provisioning makes -use of a lot of caching. Some of those caches are not immune to being -corrupted if you mess around with files in your repository a lot. We +`var/log/provision.log` to help with debugging. Provisioning makes +use of a lot of caching. Some of those caches are not immune to being +corrupted if you mess around with files in your repository a lot. We have `tools/provision --force` to (still fairly quickly) rerun most steps that would otherwise have been skipped due to caching. @@ -42,13 +42,13 @@ also run an initial provision the first time only. In `version.py`, we have a special parameter, `PROVISION_VERSION`, which is used to help ensure developers don't spend time debugging test/linter/etc. failures that actually were caused by the developer -rebasing and forgetting to provision". `PROVISION_VERSION` has a +rebasing and forgetting to provision". `PROVISION_VERSION` has a format of `x.y`; when `x` doesn't match the value from the last time the user provisioned, or `y` is higher than than the value from last time, most Zulip tools will crash early and ask the user to provision. This has empirically made a huge impact on how often developers spend time debugging a "weird failure" after rebasing that had an easy -solution. (Of course, the other key part of achieving this is all the +solution. (Of course, the other key part of achieving this is all the work that goes into making sure that `provision` reliably leaves the development environment in a good state.) @@ -58,29 +58,29 @@ require re-running provision, so don't forget about it! ## Philosophy on adding third-party dependencies In the Zulip project, we take a pragmatic approach to third-party -dependencies. Overall, if a third-party project does something well +dependencies. Overall, if a third-party project does something well that Zulip needs to do (and has an appropriate license), we'd love to -use it rather than reinventing the wheel. If the third-party project +use it rather than reinventing the wheel. If the third-party project needs some small changes to work, we prefer to make those changes and -contribute them upstream. When the upstream maintainer is slow to +contribute them upstream. When the upstream maintainer is slow to respond, we may use a fork of the dependency until the code is merged upstream; as a result, we usually have a few packages in `requirements.txt` that are installed from a GitHub URL. What we look for in choosing dependencies is whether the project is -well-maintained. Usually one can tell fairly quickly from looking at +well-maintained. Usually one can tell fairly quickly from looking at a project's issue tracker how well-managed it is: a quick look at how the issue tracker is managed (or not) and the test suite is usually enough to decide if a project is going to be a high-maintenance -dependency or not. That said, we do still take on some smaller +dependency or not. That said, we do still take on some smaller dependencies that don't have a well-managed project, if we feel that using the project will still be a better investment than writing our -own implementation of that project's functionality. We've adopted a +own implementation of that project's functionality. We've adopted a few projects in the past that had a good codebase but whose maintainer no longer had time for them. One case where we apply added scrutiny to third-party dependencies is -JS libraries. They are a particularly important concern because we +JS libraries. They are a particularly important concern because we want to keep the Zulip webapp's JS bundle small, so that Zulip continues to load quickly on systems with low network bandwidth. We'll look at large JS libraries with much greater scrutiny for @@ -94,10 +94,10 @@ For the third-party services like PostgreSQL, Redis, Nginx, and RabbitMQ that are documented in the [architecture overview](../overview/architecture-overview.md), we rely on the versions of those packages provided alongside the Linux distribution -on which Zulip is deployed. Because Zulip +on which Zulip is deployed. Because Zulip [only supports Ubuntu in production](../production/requirements.md), this usually means `apt`, though we do support -[other platforms in development](../development/setup-advanced.md). Since +[other platforms in development](../development/setup-advanced.md). Since we don't control the versions of these dependencies, we avoid relying on specific versions of these packages wherever possible. @@ -107,7 +107,7 @@ few places: the `Package` and `SafePackage` directives. - For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`. - The packages needed to build a Zulip virtualenv, in - `VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are + `VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are separate from the rest because (1) we may need to install a virtualenv before running the more complex scripts that, in turn, install other dependencies, and (2) because that list is shared @@ -121,79 +121,79 @@ extension, used by our [full-text search](full-text-search.md). We manage Python packages via the Python-standard `requirements.txt` system and virtualenvs, but there’s a number of interesting details about how Zulip makes this system work well for us that are worth -highlighting. The system is largely managed by the code in +highlighting. The system is largely managed by the code in `scripts/lib/setup_venv.py` -- **Using `pip` to manage dependencies**. This is standard in the +- **Using `pip` to manage dependencies**. This is standard in the Python ecosystem, and means we only need to record a list of versions in a `requirements.txt` file to declare what we're using. Since we have a few different installation targets, we maintain several `requirements.txt` format files in the `requirements/` directory (e.g. `dev.in` for development, `prod.in` for production, `docs.in` for ReadTheDocs, `common.in` for the vast - majority of packages common to prod and development, etc.). We use + majority of packages common to prod and development, etc.). We use `pip install --no-deps` to ensure we only install the packages we explicitly declare as dependencies. -- **virtualenv with pinned versions**. For a large application like +- **virtualenv with pinned versions**. For a large application like Zulip, it is important to ensure that we're always using consistent, - predictable versions of all of our Python dependencies. To ensure + predictable versions of all of our Python dependencies. To ensure this, we install our dependencies in a [virtualenv][] that contains only the packages and versions that Zulip needs, and we always pin exact versions of our dependencies in our `requirements.txt` files. We pin exact versions, not minimum versions, so that installing - Zulip won't break if a dependency makes a buggy release. A side + Zulip won't break if a dependency makes a buggy release. A side effect is that it's easy to debug problems caused by dependency upgrades, since we're always doing those upgrades with an explicit commit updating the `requirements/` directory. -- **Pinning versions of indirect dependencies**. We "pin" or "lock" +- **Pinning versions of indirect dependencies**. We "pin" or "lock" the versions of our indirect dependencies files with - `tools/update-locked-requirements` (powered by `pip-compile`). What + `tools/update-locked-requirements` (powered by `pip-compile`). What this means is that we have some "source" requirements files, like `requirements/common.in`, that declare the packages that Zulip - depends on directly. Those packages have their own recursive - dependencies. When adding or removing a dependency from Zulip, one + depends on directly. Those packages have their own recursive + dependencies. When adding or removing a dependency from Zulip, one simply edits the appropriate "source" requirements files, and then - runs `tools/update-locked-requirements`. That tool will use + runs `tools/update-locked-requirements`. That tool will use `pip-compile` to generate the locked requirements files like `prod.txt`, `dev.txt` etc files that explicitly declare versions of - all of Zulip's recursive dependencies. For indirect dependencies + all of Zulip's recursive dependencies. For indirect dependencies (i.e. dependencies not explicitly declared in the source requirements files), it provides helpful comments explaining which direct dependency (or dependencies) needed that indirect dependency. The process for using this system is documented in more detail in `requirements/README.md`. -- **Caching of virtualenvs and packages**. To make updating the +- **Caching of virtualenvs and packages**. To make updating the dependencies of a Zulip installation efficient, we maintain a cache of virtualenvs named by the hash of the relevant `requirements.txt` - file (`scripts/lib/hash_reqs.py`). These caches live under - `/srv/zulip-venv-cache/`. That way, when re-provisioning a + file (`scripts/lib/hash_reqs.py`). These caches live under + `/srv/zulip-venv-cache/`. That way, when re-provisioning a development environment or deploying a new production version with the same Python dependencies, no downloading or installation is - required: we just use the same virtualenv. When the only changes + required: we just use the same virtualenv. When the only changes are upgraded versions, we'll use [virtualenv-clone][] to clone the most similar existing virtualenv and then just upgrade the packages - needed, making small version upgrades extremely efficient. And + needed, making small version upgrades extremely efficient. And finally, we use `pip`'s built-in caching to ensure that a specific version of a specific package is only downloaded once. -- **Garbage-collecting caches**. We have a tool, +- **Garbage-collecting caches**. We have a tool, `scripts/lib/clean_venv_cache.py`, which will clean old cached - virtualenvs that are no longer in use. In production, the algorithm + virtualenvs that are no longer in use. In production, the algorithm preserves recent virtualenvs as well as those in use by any current production deployment directory under `/home/zulip/deployments/`. This helps ensure that a Zulip installation doesn't leak large amounts of disk over time. -- **Scripts**. Often, we want a script running in production to use - the Zulip virtualenv. To make that work without a lot of duplicated +- **Scripts**. Often, we want a script running in production to use + the Zulip virtualenv. To make that work without a lot of duplicated code, we have a helpful function, `scripts.lib.setup_path.setup_path`, which on import will put the - currently running Python script into the Zulip virtualenv. This is + currently running Python script into the Zulip virtualenv. This is called by `./manage.py` to ensure that our Django code always uses the correct virtualenv as well. -- **Mypy type checker**. Because we're using mypy in a strict mode, +- **Mypy type checker**. Because we're using mypy in a strict mode, when you add use of a new Python dependency, you usually need to either adds stubs to the `stubs/` directory for the library, or edit `mypy.ini` in the root of the Zulip project to configure - `ignore_missing_imports` for the new library. See + `ignore_missing_imports` for the new library. See [our mypy docs][mypy-docs] for more details. ### Upgrading packages @@ -214,29 +214,29 @@ reasoning here. - In a fashion very analogous to the Python codebase, `scripts/lib/node_cache.py` manages cached `node_modules` - directories in `/srv/zulip-npm-cache`. Each is named by its hash, + directories in `/srv/zulip-npm-cache`. Each is named by its hash, computed by the `generate_sha1sum_node_modules` function. `scripts/lib/clean_node_cache.py` handles garbage-collection. - We use [yarn][], a `pip`-like tool for JavaScript, to download most - JavaScript dependencies. Yarn talks to standard the [npm][] - repository. We use the standard `package.json` file to declare our + JavaScript dependencies. Yarn talks to standard the [npm][] + repository. We use the standard `package.json` file to declare our direct dependencies, with sections for development and - production. Yarn takes care of pinning the versions of indirect + production. Yarn takes care of pinning the versions of indirect dependencies in the `yarn.lock` file; `yarn install` updates the `yarn.lock` files. -- `tools/update-prod-static`. This process is discussed in detail in +- `tools/update-prod-static`. This process is discussed in detail in the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline) article, but we don't use the `node_modules` directories directly in - production. Instead, static assets are compiled using our static + production. Instead, static assets are compiled using our static asset pipeline and it is the compiled assets that are served - directly to users. As a result, we don't ship the `node_modules` + directly to users. As a result, we don't ship the `node_modules` directory in a Zulip production release tarball, which is a good thing, because doing so would more than double the size of a Zulip release tarball. -- **Checked-in packages**. In contrast with Python, we have a few +- **Checked-in packages**. In contrast with Python, we have a few JavaScript dependencies that we have copied into the main Zulip - repository under `static/third`, often with patches. These date - from an era before `npm` existed. It is a project goal to eliminate + repository under `static/third`, often with patches. These date + from an era before `npm` existed. It is a project goal to eliminate these checked-in versions of dependencies and instead use versions managed by the npm repositories. @@ -253,7 +253,7 @@ its own path, which we use, though we install a `/usr/local/bin/node` wrapper to access the desired version conveniently and efficiently (`nvm` has a lot of startup overhead). - `install-yarn.sh` is configured to install `yarn` at -`/srv/zulip-yarn`. We don't do anything special to try to manage +`/srv/zulip-yarn`. We don't do anything special to try to manage multiple versions of `yarn`. ## Other third-party and generated files @@ -266,9 +266,9 @@ maintain them. ### Emoji Zulip uses the [iamcal emoji data package][iamcal] for its emoji data -and sprite sheets. We download this dependency using `npm`, and then +and sprite sheets. We download this dependency using `npm`, and then have a tool, `tools/setup/build_emoji`, which reformats the emoji data -into the files under `static/generated/emoji`. Those files are in +into the files under `static/generated/emoji`. Those files are in turn used by our [Markdown processor](../subsystems/markdown.md) and `tools/update-prod-static` to make Zulip's emoji work in the various environments where they need to be displayed. @@ -277,9 +277,9 @@ Since processing emoji is a relatively expensive operation, as part of optimizing provisioning, we use the same caching strategy for the compiled emoji data as we use for virtualenvs and `node_modules` directories, with `scripts/lib/clean_emoji_cache.py` responsible for -garbage-collection. This caching and garbage-collection is required +garbage-collection. This caching and garbage-collection is required because a correct emoji implementation involves over 1000 small image -files and a few large ones. There is a more extended article on our +files and a few large ones. There is a more extended article on our [emoji infrastructure](emoji.md). ### Translations data @@ -287,7 +287,7 @@ files and a few large ones. There is a more extended article on our Zulip's [translations infrastructure](../translating/translating.md) generates several files from the source data, which we manage similar to our emoji, but without the caching (and thus without the -garbage-collection). New translations data is downloaded from +garbage-collection). New translations data is downloaded from Transifex and then compiled to generate both the production locale files and also language data in `locale/language*.json` using `manage.py compilemessages`, which extends the default Django @@ -296,7 +296,7 @@ implementation of that tool. ### Pygments data The list of languages supported by our Markdown syntax highlighting -comes from the [pygments][] package. `tools/setup/build_pygments_data` is +comes from the [pygments][] package. `tools/setup/build_pygments_data` is responsible for generating `static/generated/pygments_data.json` so that our JavaScript Markdown processor has access to the supported list. @@ -305,12 +305,12 @@ our JavaScript Markdown processor has access to the supported list. When making changes to Zulip's provisioning process or dependencies, usually one needs to think about making changes in 3 places: -- `tools/lib/provision.py`. This is the main provisioning script, +- `tools/lib/provision.py`. This is the main provisioning script, used by most developers to maintain their development environment. -- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation" - documentation. Strategically, we'd like to move the support for more +- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation" + documentation. Strategically, we'd like to move the support for more versions of Linux from here into `tools/lib/provision.py`. -- Production. Our tools for compiling/generating static assets need +- Production. Our tools for compiling/generating static assets need to be called from `tools/update-prod-static`, which is called by `tools/build-release-tarball` (for doing Zulip releases) as well as `tools/upgrade-zulip-from-git` (for deploying a Zulip server off of diff --git a/docs/subsystems/django-upgrades.md b/docs/subsystems/django-upgrades.md index 55fe806816..7a054e0e66 100644 --- a/docs/subsystems/django-upgrades.md +++ b/docs/subsystems/django-upgrades.md @@ -1,7 +1,7 @@ # Upgrading Django This article documents notes on the process for upgrading Zulip to -new major versions of Django. Here are the steps: +new major versions of Django. Here are the steps: - Carefully read the Django upstream changelog, and `git grep` to check if we're using anything deprecated or significantly modified @@ -10,19 +10,19 @@ new major versions of Django. Here are the steps: and open an issue listing them; [example](https://github.com/zulip/zulip/issues/2564). - Start submitting PRs to do any deprecation-type migrations that work - on both the old and new version of Django. The goal here is to have + on both the old and new version of Django. The goal here is to have the actual cutover commit be as small as possible, and to test as much of the changes for the migration as we can independently from the big cutover. - Check the version support of the third-party Django packages we use (`git grep django requirements/` to see a list), upgrade any as - needed and file bugs upstream for any that lack support. Look into + needed and file bugs upstream for any that lack support. Look into fixing said bugs. - Look at the pieces of Django code that we've copied and then adapted, and confirm whether Django has any updates to the modified - code we should apply. Partial list: + code we should apply. Partial list: - `CursorDebugWrapper`, which we have a modified version of in - `zerver/lib/db.py`. See + `zerver/lib/db.py`. See [the issue for contributing this upstream](https://github.com/zulip/zulip/issues/974) - `PasswordResetForm` and any other forms we import from `django.contrib.auth.forms` in `zerver/forms.py` (which has all of diff --git a/docs/subsystems/email.md b/docs/subsystems/email.md index 490e1b7b0a..b36e9e9c80 100644 --- a/docs/subsystems/email.md +++ b/docs/subsystems/email.md @@ -25,7 +25,7 @@ with only a few things you need to know to get started. `send_future_email`. The `ScheduledEmail` entries are eventually processed by a supervisor job that runs `zerver/management/commands/deliver_scheduled_emails.py`. - Always use `user_profile.delivery_email`, not `user_profile.email`, - when passing data into the `send_email` library. The + when passing data into the `send_email` library. The `user_profile.email` field may not always be valid. - A good way to find a bunch of example email pathways is to `git grep` for `zerver/emails` in the `zerver/` directory. @@ -48,10 +48,10 @@ from a queue. Documentation on our queueing system is available ## Development and testing All the emails sent in the development environment can be accessed by -visiting `/emails` in the browser. The way that this works is that +visiting `/emails` in the browser. The way that this works is that we've set the email backend (aka what happens when you call the email `.send()` method in Django) in the development environment to be our -custom backend, `EmailLogBackEnd`. It does the following: +custom backend, `EmailLogBackEnd`. It does the following: - Logs any sent emails to `var/log/email_content.log`. This log is displayed by the `/emails` endpoint @@ -65,7 +65,7 @@ You can also forward all the emails sent in the development environment to an email account of your choice by clicking on **Forward emails to an email account** on the `/emails` page. This feature can be used for testing how the emails gets rendered by -actual email clients. This is important because web email clients +actual email clients. This is important because web email clients have limited CSS functionality, autolinkify things, and otherwise mutate the HTML email one can see previewed on `/emails`. @@ -123,18 +123,18 @@ email_password = gmail_password ## Email templates -Zulip's email templates live under `templates/zerver/emails`. Email +Zulip's email templates live under `templates/zerver/emails`. Email templates are a messy problem, because on the one hand, you want nice, readable markup and styling, but on the other, email clients have very limited CSS support and generally require us to inject any CSS we're -using in the emails into the email as inline styles. And then you -also need both plain-text and HTML emails. We solve these problems +using in the emails into the email as inline styles. And then you +also need both plain-text and HTML emails. We solve these problems using a combination of the [premailer](https://github.com/peterbe/premailer) library and having two copies of each email (plain-text and HTML). So for each email, there are two source templates: the `.txt` version -(for plain-text format) as well as a `.source.html` template. The +(for plain-text format) as well as a `.source.html` template. The `.txt` version is used directly; while the `.source.html` template is processed by `scripts/setup/inline_email_css.py` (generating a `.html` template under `templates/zerver/emails/compiled`); that tool (powered by @@ -143,19 +143,19 @@ under `templates/zerver/emails/compiled`); that tool (powered by What this means is that when you're editing emails, **you need to run `scripts/setup/inline_email_css.py`** after making changes to see the changes -take effect. Our tooling automatically runs this as part of +take effect. Our tooling automatically runs this as part of `tools/provision` and production deployments; but you should bump `PROVISION_VERSION` when making changes to emails that change test behavior, or other developers will get test failures until they provision. While this model is great for the markup side, it isn't ideal for -[translations](../translating/translating.md). The Django +[translations](../translating/translating.md). The Django translation system works with exact strings, and having different new markup can require translators to re-translate strings, which can result in problems like needing 2 copies of each string (one for plain-text, one for HTML) and/or needing to re-translate a bunch of -strings after making a CSS tweak. Re-translating these strings is +strings after making a CSS tweak. Re-translating these strings is relatively easy in Transifex, but annoying. So when writing email templates, we try to translate individual diff --git a/docs/subsystems/emoji.md b/docs/subsystems/emoji.md index cbcf4c029e..30fc440882 100644 --- a/docs/subsystems/emoji.md +++ b/docs/subsystems/emoji.md @@ -1,7 +1,7 @@ # Emoji Emoji seem like a simple idea, but there's actually a ton of -complexity that goes into an effective emoji implementation. This +complexity that goes into an effective emoji implementation. This document discusses a number of these issues. Currently, Zulip supports these four display formats for emoji: @@ -14,15 +14,15 @@ Currently, Zulip supports these four display formats for emoji: ## Emoji codes The Unicode standard has various ranges of characters set aside for -emoji. So you can put emoji in your terminal using actual Unicode -characters like 😀 and 👍. If you paste those into Zulip, Zulip will +emoji. So you can put emoji in your terminal using actual Unicode +characters like 😀 and 👍. If you paste those into Zulip, Zulip will render them as the corresponding emoji image. However, the Unicode committee did not standardize on a set of -human-readable names for emoji. So, for example, when using the +human-readable names for emoji. So, for example, when using the popular `:` based style for entering emoji from the keyboard, we have to decide whether to use `:angry:` or `:angry_face:` to represent an -angry face. Different products use different approaches, but for +angry face. Different products use different approaches, but for purposes like emoji pickers or autocomplete, you definitely want to pick exactly one of these names, since otherwise users will always be seeing duplicates of a given emoji next to each other. @@ -32,9 +32,9 @@ section on [picking emoji names](#picking-emoji-names) below. ### Custom emoji -Zulip supports custom user-uploaded emoji. We manage those by having +Zulip supports custom user-uploaded emoji. We manage those by having the name of the emoji be its "emoji code", and using an emoji_type -field to keep track of it. We are in the progress of migrating Zulip +field to keep track of it. We are in the progress of migrating Zulip to refer to these emoji only by ID, which is a requirement for being able to support deprecating old realm emoji in a sensible way. @@ -42,17 +42,17 @@ able to support deprecating old realm emoji in a sensible way. We use the [iamcal emoji data package][iamcal] to provide sprite sheets and individual images for our emoji, as well as a data set of -emoji categories, code points, etc. The sprite sheets are used +emoji categories, code points, etc. The sprite sheets are used by the Zulip webapp to display emoji in messages, emoji reactions, -etc. However, we can't use the sprite sheets in some contexts, such +etc. However, we can't use the sprite sheets in some contexts, such as missed-message and digest emails, that need to have self-contained -assets. For those, we use individual emoji files under -`static/generated/emoji`. The structure of that repository contains +assets. For those, we use individual emoji files under +`static/generated/emoji`. The structure of that repository contains both files named after the Unicode representation of emoji (as actual image files) as well as symlinks pointing to those emoji. We need to maintain those both for the names used in the iamcal emoji -data set as well as our old emoji data set (`emoji_map.json`). Zulip +data set as well as our old emoji data set (`emoji_map.json`). Zulip has a tool, `tools/setup/emoji/build_emoji`, that combines the `emoji.json` file from iamcal with the old `emoji_map.json` data set to construct the various symlink farms and output files described @@ -64,30 +64,30 @@ The `build_emoji` tool generates the set of files under `static/generated/emoji` is a symlink to that tree; we do this in order to cache old versions to make provisioning and production deployments super fast in the common case that we haven't changed the -emoji tooling). See [our dependencies document](../subsystems/dependencies.md) +emoji tooling). See [our dependencies document](../subsystems/dependencies.md) for more details on this strategy. The emoji tree generated by this process contains several import elements: - `emoji_codes.json`: A set of mappings used by the Zulip frontend to understand what Unicode emoji exist and what their shortnames are, - used for autocomplete, emoji pickers, etc. This has been + used for autocomplete, emoji pickers, etc. This has been deduplicated using the logic in `tools/setup/emoji/emoji_setup_utils.py` to generally only have `:angry:` and not also `:angry_face:`, since having both is ugly and pointless for purposes like autocomplete and emoji pickers. - `images/emoji/unicode/*.png`: A farm of emoji - `images/emoji/*.png`: A farm of symlinks from emoji names to the - `images/emoji/unicode/` tree. This is used to serve individual emoji + `images/emoji/unicode/` tree. This is used to serve individual emoji images, as well as for the [backend Markdown processor](../subsystems/markdown.md) to know which emoji - names exist and what Unicode emoji / images they map to. In this + names exist and what Unicode emoji / images they map to. In this tree, we currently include all of the emoji in `emoji-map.json`; this means that if you send `:angry_face:`, it won't autocomplete, but will still work (but not in previews). - Some CSS and PNGs for the emoji spritesheets, used in Zulip for emoji pickers where we would otherwise need to download over 1000 of individual emoji images (which would cause a browser performance - problem). We have multiple spritesheets: one for each emoji + problem). We have multiple spritesheets: one for each emoji provider that we support (Google, Twitter, EmojiOne, and Apple.). [iamcal]: https://github.com/iamcal/emoji-data diff --git a/docs/subsystems/events-system.md b/docs/subsystems/events-system.md index f1b5f271fb..b7c58cf1a9 100644 --- a/docs/subsystems/events-system.md +++ b/docs/subsystems/events-system.md @@ -1,7 +1,7 @@ # Real-time push and events Zulip's "events system" is the server-to-client push system that -powers our real-time sync. This document explains how it works; to +powers our real-time sync. This document explains how it works; to read an example of how a complete feature using this system works, check out the [new application feature tutorial](../tutorials/new-feature-tutorial.md). @@ -9,23 +9,23 @@ check out the Any single-page web application like Zulip needs a story for how changes made by one client are synced to other clients, though having a good architecture for this is particularly important for a chat tool -like Zulip, since the state is constantly changing. When we talk +like Zulip, since the state is constantly changing. When we talk about clients, think a browser tab, mobile app, or API bot that needs -to receive updates to the Zulip data. The simplest example is a new +to receive updates to the Zulip data. The simplest example is a new message being sent by one client; other clients must be notified in -order to display the message. But a complete application like Zulip +order to display the message. But a complete application like Zulip has dozens of different types of data that need to be synced to other clients, whether it be new streams, changes in a user's name or -avatar, settings changes, etc. In Zulip, we call these updates that +avatar, settings changes, etc. In Zulip, we call these updates that need to be sent to other clients **events**. An important thing to understand when designing such a system is that events need to be synced to every client that has a copy of the old data if one wants to avoid clients displaying inaccurate data to -users. So if a user has two browser windows open and sends a message, +users. So if a user has two browser windows open and sends a message, every client controlled by that user as well as any recipients of the message, including both of those two browser windows, will receive -that event. (Technically, we don't need to send events to the client +that event. (Technically, we don't need to send events to the client that triggered the change, but this approach saves a bunch of unnecessary duplicate UI update code, since the client making the change can just use the same code as every other client, maybe plus a @@ -34,11 +34,11 @@ little notification that the operation succeeded). Architecturally, there are a few things needed to make a successful real-time sync system work: -- **Generation**. Generating events when changes happen to data, and +- **Generation**. Generating events when changes happen to data, and determining which users should receive each event. -- **Delivery**. Efficiently delivering those events to interested +- **Delivery**. Efficiently delivering those events to interested clients, ideally in an exactly-once fashion. -- **UI updates**. Updating the UI in the client once it has received +- **UI updates**. Updating the UI in the client once it has received events from the server. Reactive JavaScript libraries like React and Vue can help simplify the @@ -51,14 +51,14 @@ problems in a scalable, correct, and predictable way. ## Generation system Zulip's generation system is built around a Python function, -`send_event(realm, event, users)`. It accepts the realm (used for +`send_event(realm, event, users)`. It accepts the realm (used for sharding), the event data structure (just a Python dictionary with some keys and value; `type` is always one of the keys but the rest depends on the specific event) and a list of user IDs for the users -whose clients should receive the event. In special cases such as +whose clients should receive the event. In special cases such as message delivery, the list of users will instead be a list of dicts mapping user IDs to user-specific data like whether that user was -mentioned in that message. The data passed to `send_event` are simply +mentioned in that message. The data passed to `send_event` are simply marshalled as JSON and placed in the `notify_tornado` RabbitMQ queue to be consumed by the delivery system. @@ -72,9 +72,9 @@ Usually, this list of users is one of 3 things: or the people on a private message thread. It is the responsibility of the caller of `send_event` to choose the -list of user IDs correctly. There can be security problems if e.g. an +list of user IDs correctly. There can be security problems if e.g. an event containing private message content is sent to the entire -organization. However, if an event isn't sent to enough clients, +organization. However, if an event isn't sent to enough clients, there will likely be user-visible real-time sync bugs. Most of the hard work in event generation is about defining consistent @@ -84,7 +84,7 @@ wide range of possible clients, and make it easy for developers. ## Delivery system Zulip's event delivery (real-time push) system is based on Tornado, -which is ideal for handling a large number of open requests. Details +which is ideal for handling a large number of open requests. Details on Tornado are available in the [architecture overview](../overview/architecture-overview.md), but in short it is good at holding open a large number of connections for a long time. @@ -94,7 +94,7 @@ primarily `zerver/tornado/event_queue.py`. Zulip's event delivery system is based on "long-polling"; basically clients make `GET /json/events` calls to the server, and the server doesn't respond to the request until it has an event to deliver to the -client. This approach is reasonably efficient and works everywhere +client. This approach is reasonably efficient and works everywhere (unlike websockets, which have a decreasing but nonzero level of client compatibility problems). @@ -103,16 +103,16 @@ For each connected client, the **event queue server** maintains an that client which have not yet been acknowledged by that client. Ignoring the subtle details around error handling, the protocol is pretty simple; when a client does a `GET /json/events` call, the -server checks if there are any events in the queue. If there are, it -returns the events immediately. If there aren't, it records that +server checks if there are any events in the queue. If there are, it +returns the events immediately. If there aren't, it records that queue as having a waiting client (often called a `handler` in the code). When it pulls an event off the `notify_tornado` RabbitMQ queue, it simply delivers the event to each queue associated with one of the -target users. If the queue has a waiting client, it breaks the +target users. If the queue has a waiting client, it breaks the long-poll connection by returning an HTTP response to the waiting -client request. If there is no waiting client, it simply pushes the +client request. If there is no waiting client, it simply pushes the event onto the queue. When starting up, each client makes a `POST /json/register` to the @@ -120,19 +120,19 @@ server, which creates a new event queue for that client and returns the `queue_id` as well as an initial `last_event_id` to the client (it can also, optionally, fetch the initial data to save an RTT and avoid races; see the below section on initial data fetches for details on -why this is useful). Once the event queue is registered, the client +why this is useful). Once the event queue is registered, the client can just do an infinite loop calling `GET /json/events` with those parameters, updating `last_event_id` each time to acknowledge any events it has received (see `call_on_each_event` in the [Zulip Python API bindings][api-bindings-code] for a complete example -implementation). When handling each `GET /json/events` request, the +implementation). When handling each `GET /json/events` request, the queue server can safely delete any events that have an event ID less than or equal to the client's `last_event_id` (event IDs are just a counter for the events a given queue has received.) If network failures were impossible, the `last_event_id` parameter in the protocol would not be required, but it is important for enabling -exactly-once delivery in the presence of potential failures. (Without +exactly-once delivery in the presence of potential failures. (Without it, the queue server would have to delete events from the queue as soon as it attempted to send them to the client; if that specific HTTP response didn't reach the client due to a network TCP failure, then @@ -150,13 +150,13 @@ every 45s or so (if no other events have arrived in the meantime). To avoid a large memory and other resource leak, the queues are garbage collected after (by default) 10 minutes of inactivity from a client, under the theory that the client has likely gone off the -Internet (or no longer exists) access; this happens constantly. If +Internet (or no longer exists) access; this happens constantly. If the client returns, it will receive a "queue not found" error when requesting events; its handler for this case should just restart the client / reload the browser so that it refetches initial data the same -way it would on startup. Since clients have to implement their +way it would on startup. Since clients have to implement their startup process anyway, this approach adds minimal technical -complexity to clients. A nice side effect is that if the event queue +complexity to clients. A nice side effect is that if the event queue server (which stores queues in memory) were to crash and lose its data, clients would recover, just as if they had lost Internet access briefly (there is some DoS risk to manage, though). @@ -197,11 +197,11 @@ This is quite challenging to do technically, because fetching the initial state for a complex web application like Zulip might involve dozens of queries to the database, caches, etc. over the course of 100ms or more, and it is thus nearly impossible to do all of those -things together atomically. So instead, we use a more complicated +things together atomically. So instead, we use a more complicated algorithm that can produce the atomic result from non-atomic -subroutines. Here's how it works when you make a `register` API +subroutines. Here's how it works when you make a `register` API request; the logic is in `zerver/views/events_register.py` and -`zerver/lib/events.py`. The request is directly handled by Django: +`zerver/lib/events.py`. The request is directly handled by Django: - Django makes an HTTP request to Tornado, requesting that a new event queue be created, and records its queue ID. @@ -212,16 +212,16 @@ request; the logic is in `zerver/views/events_register.py` and that had been added to the Tornado event queue since it was created. - Finally, Django "applies" the events (see the `apply_events` - function) to the initial state that it fetched. E.g. for a name + function) to the initial state that it fetched. E.g. for a name change event, it finds the user data in the `realm_user` data structure, and updates it to have the new name. ### Testing The design above achieves everything we desire, at the cost that we need to -write a correct `apply_events` function. This is a difficult function to +write a correct `apply_events` function. This is a difficult function to implement correctly, because the situations that it handles almost never -happen (being race conditions) during manual testing. Fortunately, we have +happen (being race conditions) during manual testing. Fortunately, we have a protocol for testing `apply_events` in our automated backend tests. #### Overview @@ -257,7 +257,7 @@ within `test_events.py`. The `verify_action` function simulates the possible race condition in order to verify that the `apply_events` logic works correctly in the -context of some action function. To use our concrete example above, +context of some action function. To use our concrete example above, we are seeing that applying the events from the `do_remove_default_stream` action inside of `apply_events` to a stale copy of your state results in the same state dictionary as doing the @@ -290,7 +290,7 @@ behind the `apply_events` function. It may also be helpful to read the code for `verify_action` itself. Finally, you may want to ask for help on chat. Before we move on to the next step, it's worth noting that `verify_action` -only has one required parameter, which is the action function. We +only has one required parameter, which is the action function. We typically express the action function as a lambda, so that we can pass in arguments: @@ -303,7 +303,7 @@ There are some notable optional parameters for `verify_action`: - `state_change_expected` must be set to `False` if your action doesn't actually require state changes for some reason; otherwise, `verify_action` will complain that your test doesn't really - exercise any `apply_events` logic. Typing notifications (which + exercise any `apply_events` logic. Typing notifications (which are ephemereal) are a common place where we use this. - `num_events` will tell `verify_action` how many events the @@ -319,16 +319,16 @@ the code itself in `BaseAction` (in `test_events.py`). #### Schema checking -The `test_events.py` system has two forms of schema checking. The +The `test_events.py` system has two forms of schema checking. The first is verifying that you've updated the [GET /events API documentation](https://zulip.com/api/get-events) to document your new event's format for benefit of the developers of Zulip's mobile app, -terminal app, and other API clients. See the [API documentation +terminal app, and other API clients. See the [API documentation docs](../documentation/api.md) for details on the OpenAPI documentation. The second is higher-detail check inside `test_events` that this -specific test generated the expected series of events. Let's look at +specific test generated the expected series of events. Let's look at the last line of our example test snippet: ```python @@ -339,17 +339,17 @@ check_default_streams("events[0]", events[0]) We have discussed `verify_action` in some detail, and you will note that it returns the actual events generated by the action -function. It is part of our test discipline in `test_events` to +function. It is part of our test discipline in `test_events` to verify that the events are formatted in a predictable way. Ideally, we would test that events match the exact data that we expect, but it can be difficult to do this due to unpredictable -things like database ids. So instead, we just verify the "schema" -of the event(s). We use a schema checker like `check_default_streams` +things like database ids. So instead, we just verify the "schema" +of the event(s). We use a schema checker like `check_default_streams` to validate the types of the data. If you are creating a new event format, then you will have to -write your own schema checker in `event_schema.py`. Here is +write your own schema checker in `event_schema.py`. Here is the example relevant to our example: ```python @@ -362,9 +362,9 @@ default_streams_event = event_dict_type( check_default_streams = make_checker(default_streams_event) ``` -Note that `basic_stream_fields` is not shown in these docs. The +Note that `basic_stream_fields` is not shown in these docs. The best way to understand how to write schema checkers is to read -`event_schema.py`. There is a large block comment at the top of +`event_schema.py`. There is a large block comment at the top of the file, and then you can skim the rest of the file to see the patterns. @@ -386,9 +386,9 @@ against the two versions of the schema that you declared above using The final detail we need to ensure that `apply_events` always works correctly is to make sure that we have relevant tests for -every event type that can be generated by Zulip. This can be tested +every event type that can be generated by Zulip. This can be tested manually using `test-backend --coverage BaseAction` and then -checking that all the calls to `send_event` are covered. Someday +checking that all the calls to `send_event` are covered. Someday we'll add automation that verifies this directly by inspecting the coverage data. @@ -400,9 +400,9 @@ available via the `page_params` parameter. ### Messages One exception to the protocol described in the last section is the -actual messages. Because Zulip clients usually fetch them in a +actual messages. Because Zulip clients usually fetch them in a separate AJAX call after the rest of the site is loaded, we don't need -them to be included in the initial state data. To handle those +them to be included in the initial state data. To handle those correctly, clients are responsible for discarding events related to messages that the client has not yet fetched. diff --git a/docs/subsystems/full-text-search.md b/docs/subsystems/full-text-search.md index 96d9e7adac..18252a454e 100644 --- a/docs/subsystems/full-text-search.md +++ b/docs/subsystems/full-text-search.md @@ -1,7 +1,7 @@ # Full-text search Zulip supports full-text search, which can be combined arbitrarily -with Zulip's full suite of narrowing operators. By default, it only +with Zulip's full suite of narrowing operators. By default, it only supports English text, but there is an experimental [PGroonga](https://pgroonga.github.io/) integration that provides full-text search for all languages. @@ -19,7 +19,7 @@ search results. In order to optimize the performance of delivering messages, the full-text search index is updated for newly sent messages in the -background, after the message has been delivered. This background +background, after the message has been delivered. This background updating is done by `puppet/zulip/files/postgresql/process_fts_updates`, which is usually deployed on the database server, but could be deployed on an @@ -31,7 +31,7 @@ Zulip also supports using [PGroonga](https://pgroonga.github.io/) for full-text search. While PostgreSQL's built-in full-text search feature supports only one language at a time (in Zulip's case, English), the PGroonga full-text search engine supports all languages -simultaneously, including Japanese and Chinese. Once we have tested +simultaneously, including Japanese and Chinese. Once we have tested this new backend sufficiently, we expect to switch Zulip deployments to always use PGroonga. diff --git a/docs/subsystems/hashchange-system.md b/docs/subsystems/hashchange-system.md index 3a67c183fe..5e3e72acc5 100644 --- a/docs/subsystems/hashchange-system.md +++ b/docs/subsystems/hashchange-system.md @@ -13,19 +13,19 @@ Some examples are: - `/#streams/11/announce`: Streams overlay with stream ID 11 (called "announce") selected. - `/#narrow/stream/42-android/topic/fun`: Message feed showing stream - "android" and topic "fun". (The `42` represents the id of the + "android" and topic "fun". (The `42` represents the id of the stream. The main module in the frontend that manages this all is `static/js/hashchange.js` (plus `hash_util.js` for all the parsing -code), which is unfortunately one of our thorniest modules. Part of +code), which is unfortunately one of our thorniest modules. Part of the reason that it's thorny is that it needs to support a lot of different flows: - The user clicking on an in-app link, which in turn opens an overlay. For example the streams overlay opens when the user clicks the small cog symbol on the left sidebar, which is in fact a link to - `/#streams`. This makes it easy to have simple links around the app + `/#streams`. This makes it easy to have simple links around the app without custom click handlers for each one. - The user uses the "back" button in their browser (basically equivalent to the previous one, as a *link* out of the browser history @@ -70,7 +70,7 @@ Internally you have these functions: - `hashchange.do_hashchange_normal` handles most cases, like loading the main page (but maybe with a specific URL if you are narrowed to a stream or topic or PMs, etc.). -- `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have +- `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have some minor complexity related to remembering the page from which the overlay was launched, as well as optimizing in-page transitions (i.e. don't close/re-open the overlay if you can @@ -84,19 +84,19 @@ reload itself: - If the browser has been offline for more than 10 minutes, the browser's [event queue][events-system] will have been garbage-collected by the server, meaning the browser can no longer - get real-time updates altogether. In this case, the browser - auto-reloads immediately in order to reconnect. We have coded an + get real-time updates altogether. In this case, the browser + auto-reloads immediately in order to reconnect. We have coded an unsuspend callback (based on some clever time logic) that ensures we check immediately when a client unsuspends; grep for `watchdog` to see the code. - If a new version of the server has been deployed, we want to reload - the browser so that it will start running the latest code. However, - we don't want server deploys to be disruptive. So, the backend + the browser so that it will start running the latest code. However, + we don't want server deploys to be disruptive. So, the backend preserves user-side event queues (etc.) and just pushes a special - `restart` event to all clients. That event causes the browser to + `restart` event to all clients. That event causes the browser to start looking for a good time to reload, based on when the user is idle (ideally, we'd reload when they're not looking and restore - state so that the user never knew it happened!). The logic for + state so that the user never knew it happened!). The logic for doing this is in `static/js/reload.js`; but regardless we'll reload within 30 minutes unconditionally. diff --git a/docs/subsystems/hotspots.md b/docs/subsystems/hotspots.md index 4f01d64555..4e54abf366 100644 --- a/docs/subsystems/hotspots.md +++ b/docs/subsystems/hotspots.md @@ -48,7 +48,7 @@ However, if you would like to fix the orientation of a hotspot popover, a To test your hotspot in the development environment, set `ALWAYS_SEND_ALL_HOTSPOTS = True` in `zproject/dev_settings.py`, and invoke `hotspots.initialize()` in your browser console. Every hotspot -should be displayed. Note that this setting has a bug that can result +should be displayed. Note that this setting has a bug that can result in multiple copies of hotspots appearing; you can clear that by reloading the browser. diff --git a/docs/subsystems/html-css.md b/docs/subsystems/html-css.md index 8863e5cc39..817412cf1a 100644 --- a/docs/subsystems/html-css.md +++ b/docs/subsystems/html-css.md @@ -3,12 +3,12 @@ ## Zulip CSS organization The Zulip application's CSS can be found in the `static/styles/` -directory. Zulip uses [Bootstrap](https://getbootstrap.com/) as its +directory. Zulip uses [Bootstrap](https://getbootstrap.com/) as its main third-party CSS library. -Zulip uses PostCSS for its CSS files. There are two high-level sections +Zulip uses PostCSS for its CSS files. There are two high-level sections of CSS: the "portico" (logged-out pages like /help/, /login/, etc.), -and the app. The portico CSS lives under the `static/styles/portico` +and the app. The portico CSS lives under the `static/styles/portico` subdirectory. ## Editing Zulip CSS @@ -33,8 +33,8 @@ browser window (following backend changes). Without care, it's easy for a web application to end up with thousands of lines of duplicated CSS code, which can make it very difficult to -understand the current styling or modify it. We would very much like -to avoid such a fate. So please make an effort to reuse existing +understand the current styling or modify it. We would very much like +to avoid such a fate. So please make an effort to reuse existing styling, clean up now-unused CSS, etc., to keep things maintainable. ### Be consistent with existing similar UI @@ -84,7 +84,7 @@ found [here][jconditionals]. The context for Jinja2 templates is assembled from a few places: -- `zulip_default_context` in `zerver/context_processors.py`. This is +- `zulip_default_context` in `zerver/context_processors.py`. This is the default context available to all Jinja2 templates. - As an argument in the `render` call in the relevant function that @@ -151,10 +151,10 @@ relevant background as well. ### Primary build process Zulip's frontend is primarily JavaScript in the `static/js` directory; -we are working on migrating these to TypeScript modules. Stylesheets +we are working on migrating these to TypeScript modules. Stylesheets are written in CSS extended by various PostCSS plugins; they are converted from plain CSS, and we have yet to take full advantage of -the features PostCSS offers. We use Webpack to transpile and build JS +the features PostCSS offers. We use Webpack to transpile and build JS and CSS bundles that the browser can understand, one for each entry points specified in `tools/webpack.*assets.json`; source maps are generated in the process for better debugging experience. @@ -188,9 +188,9 @@ first add it to the appropriate place under `static/`. version of third-party libraries. - Third-party files that we have patched should all go in `static/third/`. Tag the commit with "[third]" when adding or - modifying a third-party package. Our goal is to the extent possible + modifying a third-party package. Our goal is to the extent possible to eliminate patched third-party code from the project. -- Our own JavaScript and TypeScript files live under `static/js`. Ideally, +- Our own JavaScript and TypeScript files live under `static/js`. Ideally, new modules should be written in TypeScript (details on this policy below). - CSS files live under `static/styles`. - Portico JavaScript ("portico" means for logged-out pages) lives under @@ -225,18 +225,18 @@ If you want to test minified files in development, look for the A few useful notes are: - Zulip installs static assets in production in -`/home/zulip/prod-static`. When a new version is deployed, before the +`/home/zulip/prod-static`. When a new version is deployed, before the server is restarted, files are copied into that directory. - We use the VFL (versioned file layout) strategy, where each file in the codebase (e.g. `favicon.ico`) gets a new name - (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each + (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each deployment, has a manifest file (e.g. `/home/zulip/deployments/current/staticfiles.json`) that maps - codebase filenames to serving filenames for that deployment. The + codebase filenames to serving filenames for that deployment. The benefit of this VFL approach is that all the static files for past deployments can coexist, which in turn eliminates most classes of race condition bugs where browser windows opened just before a - deployment can't find their static assets. It also is necessary for + deployment can't find their static assets. It also is necessary for any incremental rollout strategy where different clients get different versions of the site. - Some paths for files (e.g. emoji) are stored in the @@ -261,24 +261,24 @@ where one is moving code from an existing JavaScript module, the new commit should just move the code, not translate it to TypeScript). TypeScript provides more accurate information to development tools, allowing for better refactoring, auto-completion and static analysis. -TypeScript also uses the ES6 module system. See our documentation on +TypeScript also uses the ES6 module system. See our documentation on [TypeScript static types](../testing/typescript). Webpack does not ordinarily allow modules to be accessed directly from the browser console, but for debugging convenience, we have a custom webpack plugin (`tools/debug-require-webpack-plugin.ts`) that exposes a version of the `require()` function to the development environment -browser console for this purpose. For example, you can access our +browser console for this purpose. For example, you can access our `people` module by evaluating `people = require("./static/js/people")`, or the third-party `lodash` -module with `_ = require("lodash")`. This mechanism is **not** a +module with `_ = require("lodash")`. This mechanism is **not** a stable API and should not be used for any purpose other than interactive debugging. We have one module, `zulip_test`, that’s exposed as a global variable using `expose-loader` for direct use in Puppeteer tests and in the -production browser console. If you need to access a variable or -function in those scenarios, add it to `zulip_test`. This is also +production browser console. If you need to access a variable or +function in those scenarios, add it to `zulip_test`. This is also **not** a stable API. [Jinja2]: http://jinja.pocoo.org/ diff --git a/docs/subsystems/input-pills.md b/docs/subsystems/input-pills.md index 0d4e3f44ba..344d49f842 100644 --- a/docs/subsystems/input-pills.md +++ b/docs/subsystems/input-pills.md @@ -27,17 +27,17 @@ var pills = input_pill.create({ ``` You can look at `static/js/user_pill.js` to see how the above -methods are implemented. Essentially you just need to convert +methods are implemented. Essentially you just need to convert from raw data (like an email) to structured data (like an object with display_value, email, and user_id for a user), and vice -versa. The most important field to supply is `display_value`. +versa. The most important field to supply is `display_value`. For user pills `pill_item.display_value === user.full_name`. ## Typeahead Pills almost always work in conjunction with typeahead, and you will want to provide a `source` function to typeahead -that can exclude items from the prior pills. Here is an +that can exclude items from the prior pills. Here is an example snippet from our user group settings code. ```js diff --git a/docs/subsystems/logging.md b/docs/subsystems/logging.md index 2c7859456f..f450f54d08 100644 --- a/docs/subsystems/logging.md +++ b/docs/subsystems/logging.md @@ -1,12 +1,12 @@ # Logging and error reporting Having a good system for logging error reporting is essential to -making a large project like Zulip successful. Without reliable error +making a large project like Zulip successful. Without reliable error reporting, one has to rely solely on bug reports from users in order to produce a working product. Our goal as a project is to have zero known 500 errors on the backend -and zero known JavaScript exceptions on the frontend. While there +and zero known JavaScript exceptions on the frontend. While there will always be new bugs being introduced, that goal is impossible without an efficient and effective error reporting framework. @@ -21,7 +21,7 @@ The [Django][django-errors] framework provides much of the infrastructure needed by our error reporting system: - The ability to send emails to the server's administrators with any - 500 errors, using the `mail_admins` function. We enhance these data + 500 errors, using the `mail_admins` function. We enhance these data with extra details (like what user was involved in the error) in `zerver/logging_handlers.py`, and then send them to the administrator in `zerver/lib/error_notify.py` (which also supports @@ -48,10 +48,10 @@ exception, and the full request headers which triggered it. ### Backend logging [Django's logging system][django-logging] uses the standard -[Python logging infrastructure][python-logging]. We have configured +[Python logging infrastructure][python-logging]. We have configured them so that `logging.exception` and `logging.error` get emailed to the server maintainer, while `logging.warning` will just appear in -`/var/log/zulip/errors.log`. Lower log levels just appear in the main +`/var/log/zulip/errors.log`. Lower log levels just appear in the main server log (as well as in the log for corresponding process, be it `django.log` for the main Django processes or the appropriate `events_*` log file for a queue worker). @@ -60,7 +60,7 @@ server log (as well as in the log for corresponding process, be it The main Zulip server log contains a line for each backend request. It also contains warnings, errors, and the full tracebacks for any -Python exceptions. In production, it goes to +Python exceptions. In production, it goes to `/var/log/zulip/server.log`; in development, it goes to the terminal where you run `run-dev.py`. @@ -105,7 +105,7 @@ processor, in memcached, or in other Python code. One useful thing to note, however, is that the database time is only the time spent connecting to and receiving a response from the -database. Especially when response are large, there can often be a +database. Especially when response are large, there can often be a great deal of Python processing overhead to marshall the data from the database into Django objects that is not accounted for in these numbers. @@ -114,13 +114,13 @@ numbers. We have a custom library, called `blueslip` (named after the form used at MIT to report problems with the facilities), that takes care of -reporting JavaScript errors. In production, this means emailing the +reporting JavaScript errors. In production, this means emailing the server administrators (though the setting controlling this, `BROWSER_ERROR_REPORTING`, is disabled by default, since most problems are unlikely to be addressable by a system administrator, and it's very hard to make JavaScript errors not at least somewhat spammy due to the variety of browser versions and sets of extensions that someone -might use). In development, this means displaying a highly visible +might use). In development, this means displaying a highly visible overlay over the message view area, to make exceptions in testing a new feature hard to miss. @@ -131,23 +131,23 @@ new feature hard to miss. and assertion failures. - Blueslip keeps a log of all the notices it has received during a browser session, and includes them in reports to the server, so that - one can see cases where exceptions chained together. You can print + one can see cases where exceptions chained together. You can print this log from the browser console using `blueslip = require("./static/js/blueslip"); blueslip.get_log()`. Blueslip supports several error levels: - `throw new Error(…)`: For fatal errors that cannot be easily - recovered from. We try to avoid using it, since it kills the + recovered from. We try to avoid using it, since it kills the current JS thread, rather than returning execution to the caller. - `blueslip.error`: For logging of events that are definitely caused by a bug and thus sufficiently important to be reported, but where we can handle the error without creating major user-facing problems (e.g. an exception when handling a presence update). - `blueslip.warn`: For logging of events that are a problem but not - important enough to send an email about in production. They are, + important enough to send an email about in production. They are, however, highlighted in the JS console in development. - `blueslip.log` (and `blueslip.info`): Logged to the JS console in - development and also in the blueslip log in production. Useful for + development and also in the blueslip log in production. Useful for data that might help discern what state the browser was in during an error (e.g. whether the user was in a narrow). - `blueslip.debug`: Similar to `blueslip.log`, but are not printed to diff --git a/docs/subsystems/management-commands.md b/docs/subsystems/management-commands.md index 516412be21..cac10977d4 100644 --- a/docs/subsystems/management-commands.md +++ b/docs/subsystems/management-commands.md @@ -5,7 +5,7 @@ live under `{zerver,zilencer,analytics}/management/commands/`. If you need some Python code to run with a Zulip context (access to the database, etc.) in a script, it should probably go in a management -command. The key thing distinguishing these from production scripts +command. The key thing distinguishing these from production scripts (`scripts/`) and development scripts (`tools/`) is that management commands can access the database. @@ -33,14 +33,14 @@ installation, e.g. `checkconfig`, `send_test_email`. ## Writing management commands It's generally pretty easy to template off an existing management -command to write a new one. Some good examples are -`change_user_email` and `deactivate_realm`. The Django documentation +command to write a new one. Some good examples are +`change_user_email` and `deactivate_realm`. The Django documentation is good, but we have a few pieces advice specific to the Zulip project. - If you need to access a realm or user, use the `ZulipBaseCommand` class in `zerver/lib/management.py` so you don't need to write the - tedious code of looking those objects up. This is especially + tedious code of looking those objects up. This is especially important for users, since the library handles the issues around looking up users by email well (if there's a unique user with that email, just modify it without requiring the user to specify the @@ -48,8 +48,8 @@ project. - Avoid writing a lot of code in management commands; management commands are annoying to unit test, and thus easier to maintain if all the interesting logic is in a nice function that is unit tested - (and ideally, also used in Zulip's existing code). Look for code in - `zerver/lib/` that already does what you need. For most actions, + (and ideally, also used in Zulip's existing code). Look for code in + `zerver/lib/` that already does what you need. For most actions, you can just call a `do_change_foo` type function from `zerver/lib/actions.py` to do all the work; this is usually far better than manipulating the database directly, since the library diff --git a/docs/subsystems/markdown.md b/docs/subsystems/markdown.md index 76476183c6..5fd01bd97c 100644 --- a/docs/subsystems/markdown.md +++ b/docs/subsystems/markdown.md @@ -1,13 +1,13 @@ # Markdown implementation Zulip uses a special flavor of Markdown/CommonMark for its message -formatting. Our Markdown flavor is unique primarily to add important +formatting. Our Markdown flavor is unique primarily to add important extensions, such as quote blocks and math blocks, and also to do -previews and correct issues specific to the chat context. Beyond +previews and correct issues specific to the chat context. Beyond that, it has a number of minor historical variations resulting from its history predacting CommonMark (and thus Zulip choosing different solutions to some problems) and based in part on Python-Markdown, -which is proudly a classic Markdown implementation. We reduce these +which is proudly a classic Markdown implementation. We reduce these variations with every major Zulip release. Zulip has two implementations of Markdown. The backend implementation @@ -15,11 +15,11 @@ at `zerver/lib/markdown/` is based on [Python-Markdown](https://pypi.python.org/pypi/Markdown) and is used to authoritatively render messages to HTML (and implements slow/expensive/complex features like querying the Twitter API to -render tweets nicely). The frontend implementation is in JavaScript, +render tweets nicely). The frontend implementation is in JavaScript, based on [marked.js](https://github.com/chjj/marked) (`static/js/echo.js`), and is used to preview and locally echo messages the moment the sender hits Enter, without waiting for round -trip from the server. Those frontend renderings are only shown to the +trip from the server. Those frontend renderings are only shown to the sender of a message, and they are (ideally) identical to the backend rendering. @@ -28,12 +28,12 @@ The JavaScript Markdown implementation has a function, contains any syntax that needs to be rendered to HTML on the backend. If `markdown.contains_backend_only_syntax` returns true, the frontend simply won't echo the message for the sender until it receives the rendered HTML -from the backend. If there is a bug where `markdown.contains_backend_only_syntax` +from the backend. If there is a bug where `markdown.contains_backend_only_syntax` returns false incorrectly, the frontend will discover this when the backend returns the newly sent message, and will update the HTML based on the authoritative backend rendering (which would cause a change in the rendering that is visible only to the sender shortly after a -message is sent). As a result, we try to make sure that +message is sent). As a result, we try to make sure that `markdown.contains_backend_only_syntax` is always correct. ## Testing @@ -46,7 +46,7 @@ The Python-Markdown implementation is tested by A shared set of fixed test data ("test fixtures") is present in `zerver/tests/fixtures/markdown_test_cases.json`, and is automatically used by both test suites; as a result, it is the preferred place to add new -tests for Zulip's Markdown system. Some important notes on reading +tests for Zulip's Markdown system. Some important notes on reading this file: - `expected_output` is the expected output for the backend Markdown @@ -57,13 +57,13 @@ this file: `markdown.contains_backend_only_syntax` rejects the syntax, ensuring it will be rendered only by the backend processor. - When the two processors disagree, we set `marked_expected_output` in - the fixtures; this will ensure that the syntax stays that way. If + the fixtures; this will ensure that the syntax stays that way. If the differences are important (i.e. not just whitespace), we should also open an issue on GitHub to track the problem. - For mobile push notifications, we need a text version of the rendered content, since the APNS and GCM push notification systems - don't support richer markup. Mostly, this involves stripping HTML, - but there's some syntax we take special care with. Tests for what + don't support richer markup. Mostly, this involves stripping HTML, + but there's some syntax we take special care with. Tests for what this plain-text version of content should be are stored in the `text_content` field. @@ -72,10 +72,10 @@ implementation, the easiest way to do this is as follows: 1. Log in to your development server. 2. Stop your Zulip server with Ctrl-C, leaving the browser open. -3. Compose and send the messages you'd like to test. They will be +3. Compose and send the messages you'd like to test. They will be locally echoed using the frontend rendering. -This procedure prevents any server-side rendering. If you don't do +This procedure prevents any server-side rendering. If you don't do this, backend will likely render the Markdown you're testing and swap it in before you can see the frontend's rendering. @@ -118,14 +118,14 @@ Important considerations for any changes are: accidentally triggering Markdown syntax or typeahead that isn't related to what they are trying to express. - Performance: Zulip can render a lot of messages very quickly, and - we'd like to keep it that way. New regular expressions similar to + we'd like to keep it that way. New regular expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about expensive computations or third-party API requests. - Database: The backend Markdown processor runs inside a Python thread (as part of how we implement timeouts for third-party API queries), and for that reason we currently should avoid making database - queries inside the Markdown processor. This is a technical + queries inside the Markdown processor. This is a technical implementation detail that could be changed with a few days of work, but is an important detail to know about until we do that work. - Testing: Every new feature should have both positive and negative @@ -135,7 +135,7 @@ Important considerations for any changes are: ## Per-realm features Zulip's Markdown processor's rendering supports a number of features -that depend on realm-specific or user-specific data. For example, the +that depend on realm-specific or user-specific data. For example, the realm could have [linkifiers](https://zulip.com/help/add-a-custom-linkifier) or [custom emoji](https://zulip.com/help/add-custom-emoji) @@ -144,7 +144,7 @@ groups (which depend on data like users' names, IDs, etc.). At a backend code level, these are controlled by the `message_realm` object and other arguments passed into `do_convert` (`sent_by_bot`, -`translate_emoticons`, `mention_data`, etc.). Because +`translate_emoticons`, `mention_data`, etc.). Because Python-Markdown doesn't support directly passing arguments into the Markdown processor, our logic attaches these data to the Markdown processor object via e.g. `_md_engine.zulip_db_data`, and then @@ -154,7 +154,7 @@ For non-message contexts (e.g. an organization's profile (aka the thing on the right-hand side of the login page), stream descriptions, or rendering custom profile fields), one needs to just pass in a `message_realm` (see, for example, `zulip_default_context` for the -organization profile code for this). But for messages, we need to +organization profile code for this). But for messages, we need to pass in attributes like `sent_by_bot` and `translate_emoticons` that indicate details about how the user sending the message is configured. @@ -171,21 +171,21 @@ plain text (e.g. emails) that it helps more than getting in the way. The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to -fix the formatting after typing it the first time. While that's +fix the formatting after typing it the first time. While that's basically fine when writing a blog, it gets annoying very fast in a chat product; even though you can edit messages to fix formatting -mistakes, you don't want to be doing that often. There are basically +mistakes, you don't want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip: - What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it - to render in a reasonable way? This is the "accidental Markdown + to render in a reasonable way? This is the "accidental Markdown syntax" problem, common with Markdown syntax like the italics syntax interacting with talking about `char *`s. - What fraction of the time do users attempting to use a particular - Markdown syntax actually succeed at doing so correctly? Syntax like + Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figure substantially. @@ -223,7 +223,7 @@ accurate. - Disable special use of `\` to escape other syntax. Rendering `\\` as `\` was hugely controversial, but having no escape syntax is also - controversial. We may revisit this. For now you can always put + controversial. We may revisit this. For now you can always put things in code blocks. ### Lists @@ -271,7 +271,7 @@ accurate. - Disabled images with `![]()` (images from links are shown as an inline preview). -- Allow embedding any avatar as a tiny (list bullet size) image. This +- Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integrations. - We added the `~~~ quote` block quote syntax. diff --git a/docs/subsystems/notifications.md b/docs/subsystems/notifications.md index 53cc9dcaf8..5fb596eab4 100644 --- a/docs/subsystems/notifications.md +++ b/docs/subsystems/notifications.md @@ -37,9 +37,9 @@ as follows: - The `presence_idle_user_ids` set, containing the subset of recipient users who are mentioned, are PM recipients, have alert words, or otherwise would normally get a notification, but have not - interacted with a Zulip client in the last few minutes. (Users who + interacted with a Zulip client in the last few minutes. (Users who have generally will not receive a notification unless the - `enable_online_push_notifications` flag is enabled). This data + `enable_online_push_notifications` flag is enabled). This data structure ignores users for whom the message is not notifiable, which is important to avoid this being thousands of `user_ids` for messages to large streams with few currently active users. @@ -97,7 +97,7 @@ as follows: - The queue processors for those queues make the final determination for whether to send a notification, and do the work to generate an email (`zerver/lib/email_notifications.py`) or mobile - (`zerver/lib/push_notifications.py`) notification. We'll detail + (`zerver/lib/push_notifications.py`) notification. We'll detail this process in more detail for each system below, but it's important to know that it's normal for a message to sit in these queues for minutes (and in the future, possibly hours). @@ -115,7 +115,7 @@ as follows: messages into a single email. These features are unnecessary for mobile push notifications, because we can live-update those details with a future notification, whereas emails cannot be readily - updated once sent. Zulip's email notifications are styled similarly + updated once sent. Zulip's email notifications are styled similarly to GitHub's email notifications, with a clean, simple design that makes replying from an email client possible (using the [incoming email integration](../production/email-gateway.md)). @@ -139,7 +139,7 @@ structure of the system, when thinking about changes to it: to Tornado via RabbitMQ for scalability reasons. - **Tornado doesn't do database queries**. Because the Tornado system is an asynchronous event-driven framework, and our Django database - library is synchronous, database queries are very expensive. So + library is synchronous, database queries are very expensive. So these queries need to be done in either `do_send_messages` or the queue processor logic. (For example, this means `presence` data should be checked in either `do_send_messages` or the queue diff --git a/docs/subsystems/performance.md b/docs/subsystems/performance.md index b36e115c11..da10697369 100644 --- a/docs/subsystems/performance.md +++ b/docs/subsystems/performance.md @@ -1,7 +1,7 @@ # Performance and scalability This page aims to give some background to help prioritize work on the -Zulip's server's performance and scalability. By scalability, we mean +Zulip's server's performance and scalability. By scalability, we mean the ability of the Zulip server on given hardware to handle a certain workload of usage without performance materially degrading. @@ -9,14 +9,14 @@ First, a few notes on philosophy. - We consider it an important technical goal for Zulip to be fast, because that's an important part of user experience for a real-time - collaboration tool like Zulip. Many UI features in the Zulip webapp + collaboration tool like Zulip. Many UI features in the Zulip webapp are designed to load instantly, because all the data required for them is present in the initial HTTP response, and both the Zulip API and webapp are architected around that strategy. - The Zulip database model and server implementation are carefully designed to ensure that every common operation is efficient, with automated tests designed to prevent the accidental introductions of - inefficient or excessive database queries. We much prefer doing + inefficient or excessive database queries. We much prefer doing design/implementation work to make requests fast over the operational work of running 2-5x as much hardware to handle the same load. @@ -30,9 +30,9 @@ important to understand the load profiles for production uses. Zulip servers typically involve a mixture of two very different types of load profiles: - Open communities like open source projects, online classes, - etc. have large numbers of users, many of whom are idle. (Many of + etc. have large numbers of users, many of whom are idle. (Many of the others likely stopped by to ask a question, got it answered, and - then didn't need the community again for the next year). Our own + then didn't need the community again for the next year). Our own [chat.zulip.org](../contributing/chat-zulip-org.md) is a good example for this, with more than 15K total user accounts, of which only several hundred have logged in during the last few weeks. @@ -42,7 +42,7 @@ of load profiles: scalability and request latency. - Fulltime teams, like your typical corporate Zulip installation, have users who are mostly active for multiple hours a day and sending a - high volume of messages each. This load profile is most important + high volume of messages each. This load profile is most important for self-hosted servers, since many of those are used exclusively by the employees of the organization running the server. @@ -53,7 +53,7 @@ organizations from each of those two load profiles. It's important to understand that Zulip has a handful of endpoints that result in the vast majority of all server load, and essentially -every other endpoint is not important for scalability. We still put +every other endpoint is not important for scalability. We still put effort into making sure those other endpoints are fast for latency reasons, but were they to be 10x faster (a huge optimization!), it wouldn't materially improve Zulip's scalability. @@ -63,7 +63,7 @@ around the several specific endpoints that have a combination of request volume and cost that makes them important. That said, it is important to distinguish the load associated with an -API endpoint from the load associated with a feature. Almost any +API endpoint from the load associated with a feature. Almost any significant new feature is likely to result in its data being sent to the client in `page_params` or `GET /messages`, i.e. one of the endpoints important to scalability here. As a result, it is important @@ -77,7 +77,7 @@ optimizations which save a few milliseconds that would be invisible to the end u if they carry any cost in code readability. In Zulip's documentation, our general rule is to primarily write facts -that are likely to remain true for a long time. While the numbers +that are likely to remain true for a long time. While the numbers presented here vary with hardware, usage patterns, and time (there's substantial oscillation within a 24 hour period), we expect the rough sense of them (as well as the list of important endpoints) is not @@ -100,7 +100,7 @@ POST /users/me/* 50ms 0.04% 20 The "Average impact" above is computed by multiplying request volume by average time; this tells you roughly that endpoint's **relative** -contribution to the steady-state total CPU load of the system. It's +contribution to the steady-state total CPU load of the system. It's not precise -- waiting for a network request is counted the same as active CPU time, but it's extremely useful for providing intuition for what code paths are most important to optimize, especially since @@ -110,7 +110,7 @@ memcached to do work. As one can see, there are two categories of endpoints that are important for scalability: those with extremely high request volumes, and those with moderately high request volumes that are also -expensive. It doesn't matter how expensive, for example, +expensive. It doesn't matter how expensive, for example, `POST /users/me/subscriptions` is for scalability, because the volume is negligible. @@ -119,7 +119,7 @@ is negligible. Zulip's Tornado-based [real-time push system](../subsystems/events-system.md), and in particular `GET /events`, accounts for something like 50% of all HTTP requests to -a production Zulip server. Despite `GET /events` being extremely +a production Zulip server. Despite `GET /events` being extremely high-volume, the typical request takes 1-3ms to process, and doesn't use the database at all (though it will access `memcached` and `redis`), so they aren't a huge contributor to the overall CPU usage @@ -132,16 +132,16 @@ usage of a Zulip installation. It's worth noting that most (~80%) Tornado requests end the longpolling via a `heartbeat` event, which are issued to idle -connections after about a minute. These `heartbeat` events are +connections after about a minute. These `heartbeat` events are useless aside from avoiding problems with networks/proxies/NATs that are configured poorly and might kill HTTP connections that have been -idle for a minute. It's likely that with some strategy for detecting +idle for a minute. It's likely that with some strategy for detecting such situations, we could reduce their volume (and thus overall Tornado load) dramatically. Currently, Tornado is sharded by realm, which is sufficient for arbitrary scaling of the number of organizations on a multi-tenant -system like zulip.com. With a somewhat straightforward set of work, +system like zulip.com. With a somewhat straightforward set of work, one could change this to sharding by `user_id` instead, which will eventually be important for individual large organizations with many thousands of concurrent users. @@ -151,9 +151,9 @@ thousands of concurrent users. `POST /users/me/presence` requests, which submit the current user's presence information and return the information for all other active users in the organization, account for about 36% of all HTTP requests -on production Zulip servers. See +on production Zulip servers. See [presence](../subsystems/presence.md) for details on this system and -how it's optimized. For this article, it's important to know that +how it's optimized. For this article, it's important to know that presence is one of the most important scalability concerns for any chat system, because it cannot be cached long, and is structurally a quadratic problem. @@ -162,7 +162,7 @@ Because typical presence requests consume 10-50ms of server-side processing time (to fetch and send back live data on all other active users in the organization), and are such a high volume, presence is the single most important source of steady-state load for a Zulip -server. This is true for most other chat server implementations as +server. This is true for most other chat server implementations as well. There is an ongoing [effort to rewrite the data model for @@ -182,7 +182,7 @@ data required for the entire Zulip webapp in this single request, which is part of why the Zulip webapp loads very quickly -- one only needs a single round trip aside from cacheable assets (avatars, images, JS, CSS). Data on other users in the organization, streams, supported -emoji, custom profile fields, etc., is all included. The nice thing +emoji, custom profile fields, etc., is all included. The nice thing about this model is that essentially every UI element in the Zulip client can be rendered immediately without paying latency to the server; this is critical to Zulip feeling performant even for users @@ -210,7 +210,7 @@ history](#fetching-message-history). The cost for fetching `page_params` varies dramatically based primarily on the organization's size, varying from 90ms-300ms for a typical organization but potentially multiple seconds for large open -organizations with 10,000s of users. There is also smaller +organizations with 10,000s of users. There is also smaller variability based on a individual user's personal data state, primarily in that having 10,000s of unread messages results in a somewhat expensive query to find which streams/topics those are in. @@ -221,7 +221,7 @@ greater than a second to be a bug, and there is ongoing work to fix that. It can help when thinking about this to imagine `page_params` as what in another webapp would have been 25 or so HTTP GET requests, each fetching data of a given type (users, streams, custom emoji, etc.); in -Zulip, we just do all of those in a single API request. In the +Zulip, we just do all of those in a single API request. In the future, we will likely move to a design that does much of the database fetching work for different features in parallel to improve latency. @@ -234,7 +234,7 @@ of active optimization work. Bulk requests for message content and metadata ([`GET /messages`](https://zulip.com/api/get-messages)) account for -~3% of total HTTP requests. The zulip webapp has a few major reasons +~3% of total HTTP requests. The zulip webapp has a few major reasons it does a large number of these requests: - Most of these requests are from users clicking into different views @@ -243,21 +243,21 @@ it does a large number of these requests: relevant stream/topic cached locally. - When a browser opens the Zulip webapp, it will eventually fetch and cache in the browser all messages newer than the oldest unread - message in a non-muted context. This can be in total extremely + message in a non-muted context. This can be in total extremely expensive for users with 10,000s of unread messages, resulting in a single browser doing 100 of these requests. - When a new version of the Zulip server is deployed, every browser will reload within 30 minutes to ensure they are running the latest - code. For installations that deploy often like chat.zulip.org and + code. For installations that deploy often like chat.zulip.org and zulip.com, this can result in a thundering herd effect for both `/` - and `GET /messages`. A great deal of care has been taking in + and `GET /messages`. A great deal of care has been taking in designing this [auto-reload system](../subsystems/hashchange-system.html#server-initiated-reloads) to spread most of that herd over several minutes. Typical requests consume 20-100ms to process, much of which is waiting to fetch message IDs from the database and then their content from -memcached. While not large in an absolute sense, these requests are +memcached. While not large in an absolute sense, these requests are expensive relative to most other Zulip endpoints. Some requests, like full-text search for commonly used words, can be @@ -265,7 +265,7 @@ more expensive, but they are sufficiently rare in an absolute sense so as to be immaterial to the overall scalability of the system. This server-side code path is already heavily optimized on a -per-request basis. However, we have technical designs for optimizing +per-request basis. However, we have technical designs for optimizing the overall frequency with which clients need to make these requests in two major ways: @@ -274,7 +274,7 @@ in two major ways: caching of narrows that the user has viewed in the current session, avoiding repeat fetches of message content during a given session. - Adjusting the behavior for clients with 10,000s of unread messages - to not fetch as much old message history into the cache. See [this + to not fetch as much old message history into the cache. See [this issue](https://github.com/zulip/zulip/issues/16697) for relevant design work. @@ -284,7 +284,7 @@ scalability cost of fetching message history dramatically. ### User uploads Requests to fetch uploaded files (including user avatars) account for -about 5% of total HTTP requests. Zulip spends consistently ~10-15ms +about 5% of total HTTP requests. Zulip spends consistently ~10-15ms processing one of these requests (mostly authorization logic), before handing off delivery of the file to `nginx` or S3 (depending on the configured [upload backend](../production/upload-backends.md)). @@ -306,7 +306,7 @@ sent to 50 users triggers ~50 `GET /events` requests. A typical message-send request takes 20-70ms, with more expensive requests typically resulting from [Markdown -rendering](../subsystems/markdown.md) of more complex syntax. As a +rendering](../subsystems/markdown.md) of more complex syntax. As a result, these requests are not material to Zulip's scalability. Editing messages and adding emoji reactions are very similar to sending them for the purposes of performance and scalability, since @@ -339,11 +339,11 @@ The above doesn't cover all of the work that a production Zulip server does; various tasks like sending outgoing emails or recording the data that powers [/stats](https://zulip.com/help/analytics) are run by [queue processors](../subsystems/queuing.md) and cron jobs, not in -response to incoming HTTP requests. In practice, all of these have +response to incoming HTTP requests. In practice, all of these have been written such that they are immaterial to total load and thus architectural scalability, though we do from time to time need to do operational work to add additional queue processors for particularly -high-traffic queues. For all of our queue processors, any +high-traffic queues. For all of our queue processors, any serialization requirements are at most per-user, and thus it would be straightforward to shard by `user_id` or `realm_id` if required. @@ -355,7 +355,7 @@ services (memcached, redis, rabbitmq, and most importantly postgres), as well as queue processors (which might get backlogged). In practice, efforts to make an individual endpoint faster will very -likely reduce the load on these services as well. But it is worth +likely reduce the load on these services as well. But it is worth considering that database time is a more precious resource than Python/CPU time (being harder to scale horizontally). diff --git a/docs/subsystems/pointer.md b/docs/subsystems/pointer.md index 9094c844a5..d8b77fe5be 100644 --- a/docs/subsystems/pointer.md +++ b/docs/subsystems/pointer.md @@ -21,7 +21,7 @@ First a bit of terminology: the messages the user has access to. - The blue cursor box (the "pointer") is around is called the - "selected" message. Zulip ensures that the currently selected + "selected" message. Zulip ensures that the currently selected message is always in-view. ## Pointer logic @@ -53,7 +53,7 @@ streams.) ### Unnarrow: previous sequence -When you unnarrow using e.g. the `a` key, you will automatically be +When you unnarrow using e.g. the `a` key, you will automatically be taken to the same message that was selected in the All messages view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped forward to the first unread and non-muted @@ -76,10 +76,10 @@ see [the architectural overview](../overview/architecture-overview.md). How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might -use the product. The algorithm is as follows: +use the product. The algorithm is as follows: - Any message which is selected or above a message which is selected - is marked as read. So messages are marked as read as you scroll + is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. - If the whitespace at the very bottom of the feed is in view, all diff --git a/docs/subsystems/presence.md b/docs/subsystems/presence.md index dcefd6ce8b..c426f3fd43 100644 --- a/docs/subsystems/presence.md +++ b/docs/subsystems/presence.md @@ -5,41 +5,41 @@ This document explains the model for Zulip's presence. In a chat tool like Zulip, users expect to see the “presence” status of other users: is the person I want to talk to currently online? If not, were they last online 5 minutes ago, or more like an hour ago, or -a week? Presence helps set expectations for whether someone is likely -to respond soon. To a user, this feature can seem like a simple thing -that should be easy. But presence is actually one of the hardest +a week? Presence helps set expectations for whether someone is likely +to respond soon. To a user, this feature can seem like a simple thing +that should be easy. But presence is actually one of the hardest scalability problems for a team chat tool like Zulip. There's a lot of performance-related details in the backend and -network protocol design that we won't get into here. The focus of +network protocol design that we won't get into here. The focus of this is what one needs to know to correctly implement a Zulip client's presence implementation (e.g. webapp, mobile app, terminal client, or other tool that's intended to represent whether a user is online and using Zulip). A client should report to the server every minute a `POST` request to -`/users/me/presence`, containing the current user's status. The -requests contains a few parameters. The most important is "status", +`/users/me/presence`, containing the current user's status. The +requests contains a few parameters. The most important is "status", which had 2 valid values: - "active" -- this means the user has interacted with the client - recently. We use this for the "green" state in the webapp. + recently. We use this for the "green" state in the webapp. - "idle" -- the user has not interacted with the client recently. This is important for the case where a user left a Zulip tab open on - their desktop at work and went home for the weekend. We use this + their desktop at work and went home for the weekend. We use this for the "orange" state in the webapp. The client receives in the response to that request a data set that, for each user, contains their status and timestamp that we last heard -from that client. There are a few important details to understand +from that client. There are a few important details to understand about that data structure: - It's really important that the timestamp is the last time we heard - from the client. A client can only interpret the status to display + from the client. A client can only interpret the status to display about another user by doing a simple computation using the (status, - timestamp) pair. E.g. a user who last used Zulip 1 week ago will - have a timestamp of 1 week ago and a status of "active". Why? - Because this correctly handles the race conditions. For example, if + timestamp) pair. E.g. a user who last used Zulip 1 week ago will + have a timestamp of 1 week ago and a status of "active". Why? + Because this correctly handles the race conditions. For example, if the threshold for displaying a user as "offline" was 5 minutes since the user was last online, the client can at any time accurately compute whether that user is offline (even if the last diff --git a/docs/subsystems/queuing.md b/docs/subsystems/queuing.md index 726fadf809..00bef9c08d 100644 --- a/docs/subsystems/queuing.md +++ b/docs/subsystems/queuing.md @@ -1,6 +1,6 @@ # Queue processors -Zulip uses RabbitMQ to manage a system of internal queues. These are +Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: - Asynchronously doing expensive operations like sending email @@ -14,7 +14,7 @@ used for a variety of purposes: - Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events - system. Example events might be that a new message was sent, a user + system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. - Processing mobile push notifications and email mirroring system @@ -37,10 +37,10 @@ To add a new queue processor: - Define the processor in `zerver/worker/queue_processors.py` using the `@assign_queue` decorator; it's pretty easy to get the template - for an existing similar queue processor. This suffices to test your + for an existing similar queue processor. This suffices to test your queue worker in the Zulip development environment (`tools/run-dev.py` will automatically restart the queue processors - and start running your new queue processor code). You can also run + and start running your new queue processor code). You can also run a single queue processor manually using e.g. `./manage.py process_queue --queue=user_activity`. @@ -51,7 +51,7 @@ To add a new queue processor: The queue will automatically be added to the list of queues tracked by `scripts/nagios/check-rabbitmq-consumers`, so Nagios can properly -check whether a queue processor is running for your queue. You still +check whether a queue processor is running for your queue. You still need to update the sample Nagios configuration in `puppet/zulip_ops` manually. @@ -61,12 +61,12 @@ You can publish events to a RabbitMQ queue using the `queue_json_publish` function defined in `zerver/lib/queue.py`. An interesting challenge with queue processors is what should happen -when queued events in Zulip's backend tests. Our current solution is +when queued events in Zulip's backend tests. Our current solution is that in the tests, `queue_json_publish` will (by default) simple call -the `consume` method for the relevant queue processor. However, +the `consume` method for the relevant queue processor. However, `queue_json_publish` also supports being passed a function that should be called in the tests instead of the queue processor's `consume` -method. Where possible, we prefer the model of calling `consume` in +method. Where possible, we prefer the model of calling `consume` in tests since that's more predictable and automatically covers the queue processor's code path, but it isn't always possible. @@ -79,7 +79,7 @@ If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue user_activity ``` -You can also use the amqp tools directly. Install `amqp-tools` from +You can also use the amqp tools directly. Install `amqp-tools` from apt and then run: ```bash diff --git a/docs/subsystems/realms.md b/docs/subsystems/realms.md index 63985b41f3..8ea8b90ccf 100644 --- a/docs/subsystems/realms.md +++ b/docs/subsystems/realms.md @@ -8,7 +8,7 @@ user documentation as an organization (the name "realm" comes from Wherever possible, we avoid using the term `realm` in any user-facing string or documentation; "Organization" is the equivalent term used in those contexts (and we have linters that attempt to enforce this rule -in translateable strings). We may in the future modify Zulip's +in translateable strings). We may in the future modify Zulip's internals to use `organization` instead. The @@ -30,16 +30,16 @@ There are two main methods for creating realms. The above command will output a URL which can be used for creating a new realm and an administrator user for that realm. The link expires -after the creation of the realm. The link also expires if not used +after the creation of the realm. The link also expires if not used within 7 days. The expiration period can be changed by modifying `REALM_CREATION_LINK_VALIDITY_DAYS` in settings.py. ### Enabling open realm creation If you want anyone to be able to create new realms on your server, you -can enable open realm creation. This will add a **Create new +can enable open realm creation. This will add a **Create new organization** link to your Zulip homepage footer, and anyone can -create a new realm by visiting this link (**/new**). This +create a new realm by visiting this link (**/new**). This feature is disabled by default in production instances, and can be enabled by setting `OPEN_REALM_CREATION = True` in settings.py. @@ -57,14 +57,14 @@ records so that the subdomains point to your Zulip installation IP. An job. We also recommend upgrading to at least Zulip 1.7, since older Zulip -releases had much less nice handling for subdomains. See our +releases had much less nice handling for subdomains. See our [docs on using subdomains](../production/multiple-organizations.md) for user-facing documentation on this. ### Working with subdomains in development environment By default, Linux does not provide a convenient way to use subdomains -in your local development environment. To solve this problem, we use +in your local development environment. To solve this problem, we use the **zulipdev.com** domain, which has a wildcard A record pointing to 127.0.0.1. You can use zulipdev.com to connect to your Zulip development server instead of localhost. The default realm with the @@ -73,8 +73,8 @@ visiting **zulip.zulipdev.com**. If you are behind a **proxy server**, this method won't work. When you make a request to load zulipdev.com in your browser, the proxy server -will try to get the page on your behalf. Since zulipdev.com points -to 127.0.0.1 the proxy server is likely to give you a 503 error. The +will try to get the page on your behalf. Since zulipdev.com points +to 127.0.0.1 the proxy server is likely to give you a 503 error. The workaround is to disable your proxy for `*.zulipdev.com`. The DNS lookup should still work even if you disable proxy for *.zulipdev.com. If it doesn't you can add zulipdev.com records in diff --git a/docs/subsystems/release-checklist.md b/docs/subsystems/release-checklist.md index 9ecd642050..224a4ec4e1 100644 --- a/docs/subsystems/release-checklist.md +++ b/docs/subsystems/release-checklist.md @@ -11,7 +11,7 @@ preparing a new release. `pip list --outdated`). - [Upload strings to Transifex](../translating/internationalization.html#translation-process) - using `push-translations`. Post a Transifex + using `push-translations`. Post a Transifex [Announcement](https://www.transifex.com/zulip/zulip/announcements/) notifying translators that we're approaching a release. - Merge draft updates to the [changelog](../overview/changelog.md) @@ -22,7 +22,7 @@ preparing a new release. remove any backwards-compatibility code in this release. - Create a burn-down list of issues that need to be fixed before we can release, and make sure all of them are being worked on. -- Draft the release blog post (a.k.a. the release notes) in Paper. In +- Draft the release blog post (a.k.a. the release notes) in Paper. In it, list the important changes in the release, from most to least notable. @@ -36,9 +36,9 @@ preparing a new release. release, on both Bionic and Focal. - Repeat until release is ready. - Send around the Paper blog post draft for review. -- Move the blog post draft to Ghost. (For a draft in Dropbox Paper, +- Move the blog post draft to Ghost. (For a draft in Dropbox Paper, use "··· > Export > Markdown" to get a pretty good markup - conversion.) Proofread the post, especially for formatting. Tag + conversion.) Proofread the post, especially for formatting. Tag the post with "Release announcements" in Ghost. ### Executing the release @@ -82,8 +82,8 @@ preparing a new release. - On the release branch, update `ZULIP_VERSION` in `version.py` to the present release with a `+git` suffix, e.g. `4.0+git`. - On `main`, update `ZULIP_VERSION` to the future major release with - a `-dev+git` suffix, e.g. `5.0-dev+git`. Make a Git tag for this - update commit with a `-dev` suffix, e.g. `5.0-dev`. Push the tag + a `-dev+git` suffix, e.g. `5.0-dev+git`. Make a Git tag for this + update commit with a `-dev` suffix, e.g. `5.0-dev`. Push the tag to both zulip.git and zulip-internal.git to get a correct version number for future Cloud deployments. - Consider removing a few old releases from ReadTheDocs; we keep about diff --git a/docs/subsystems/schema-migrations.md b/docs/subsystems/schema-migrations.md index c4a8a3b6f4..8ec72fa2c0 100644 --- a/docs/subsystems/schema-migrations.md +++ b/docs/subsystems/schema-migrations.md @@ -24,7 +24,7 @@ migrations. code as part of the migration, it's best to read past migrations to understand how to write them well. `git grep RunPython zerver/migrations/02*` will find many good - examples. Before writing migrations of this form, you should read + examples. Before writing migrations of this form, you should read Django's docs and the sections below. - **Numbering conflicts across branches**: If you've done your schema change in a branch, and meanwhile another schema change has taken @@ -32,13 +32,13 @@ migrations. number. There are two easy way to fix this: - If your migrations were automatically generated using `manage.py makemigrations`, a good option is to just remove your - migration and rerun the command after rebasing. Remember to + migration and rerun the command after rebasing. Remember to `git rebase` to do this in the the commit that changed `models.py` if you have a multi-commit branch. - If you wrote code as part of preparing your migrations, or prefer this workflow, you can use run `./tools/renumber-migrations`, which renumbers your migration(s) and fixes up the "dependencies" - entries in your migration(s). The tool could use a bit of work to + entries in your migration(s). The tool could use a bit of work to prompt unnecessarily less, but it will update the working tree for you automatically (you still need to do all the `git add` commands, etc.). @@ -54,14 +54,14 @@ migrations. to be built while the server is active. While in historical migrations we've used `RunSQL` directly, newer versions of Django add the corresponding operation `AddIndexConcurrently` and thus that's what should normally be used. -- **Atomicity**. By default, each Django migration is run atomically - inside a transaction. This can be problematic if one wants to do +- **Atomicity**. By default, each Django migration is run atomically + inside a transaction. This can be problematic if one wants to do something in a migration that touches a lot of data and would best be done in batches of e.g. 1000 objects (e.g. a `Message` or - `UserMessage` table change). There is a [useful Django + `UserMessage` table change). There is a [useful Django feature][migrations-non-atomic] that makes it possible to add `atomic=False` at the top of a `Migration` class and thus not have - the entire migration in a transaction. This should make it possible + the entire migration in a transaction. This should make it possible to use the batch update tools in `zerver/lib/migrate.py` (originally written to work with South) for doing larger database migrations. @@ -101,25 +101,25 @@ migrations. non-atomic, splitting it into two migration files (recommended), or replacing the `RunPython` logic with pure SQL (though this can generally be difficult). -- **Making large migrations work**. Major migrations should have a +- **Making large migrations work**. Major migrations should have a few properties: - - **Unit tests**. You'll want to carefully test these, so you might + - **Unit tests**. You'll want to carefully test these, so you might as well write some unit tests to verify the migration works - correctly, rather than doing everything by hand. This often saves + correctly, rather than doing everything by hand. This often saves a lot of time in re-testing the migration process as we make adjustments to the plan. - - **Run in batches**. Updating more than 1K-10K rows (depending on - type) in a single transaction can lock up a database. It's best + - **Run in batches**. Updating more than 1K-10K rows (depending on + type) in a single transaction can lock up a database. It's best to do lots of small batches, potentially with a brief sleep in between, so that we don't block other operations from finishing. - - **Rerunnability/idempotency**. Good migrations are ones where if + - **Rerunnability/idempotency**. Good migrations are ones where if operational concerns (e.g. it taking down the Zulip server for users) interfere with it finishing, it's easy to restart the - migration without doing a bunch of hand investigation. Ideally, + migration without doing a bunch of hand investigation. Ideally, the migration can even continue where it left off, without needing to redo work. - - **Multi-step migrations**. For really big migrations, one wants + - **Multi-step migrations**. For really big migrations, one wants to split the transition into into several commits that are each individually correct, and can each be deployed independently: @@ -138,7 +138,7 @@ few properties: ## Automated testing for migrations Zulip has support for writing automated tests for your database -migrations, using the `MigrationsTestCase` test class. This system is +migrations, using the `MigrationsTestCase` test class. This system is inspired by [a great blog post][django-migration-test-blog-post] on the subject. @@ -148,7 +148,7 @@ from `test_classes.py` and friends from inside the tests (which is normally not possible in Django's migrations framework). If you find yourself writing logic in a `RunPython` migration, we -highly recommend adding a test using this framework. We may end up +highly recommend adding a test using this framework. We may end up deleting the test later (they can get slow once they are many migrations away from current), but it can help prevent disaster where an incorrect migration messes up a database in a way that's impossible diff --git a/docs/subsystems/sending-messages.md b/docs/subsystems/sending-messages.md index cfa34a78f6..4a52f31e5d 100644 --- a/docs/subsystems/sending-messages.md +++ b/docs/subsystems/sending-messages.md @@ -5,7 +5,7 @@ lot of underlying complexity required to make a professional-quality experience. This document aims to explain conceptually what happens when a message -is sent in Zulip, and why that is correct behavior. It assumes the +is sent in Zulip, and why that is correct behavior. It assumes the reader is familiar with our [real-time sync system](../subsystems/events-system.md) for server-to-client communication and @@ -34,7 +34,7 @@ and narrowing). ## Compose area The compose box does a lot of fancy things that are out of scope for -this article. But it also does a decent amount of client-side +this article. But it also does a decent amount of client-side validation before sending a message off to the server, especially around mentions (E.g. checking the stream name is a valid stream, displaying a warning about the number of recipients before a user can @@ -49,12 +49,12 @@ process described in our This section details the ways in which it is different: - There is significant custom code inside the `process_message_event` -function in `zerver/tornado/event_queue.py`. This custom code has a +function in `zerver/tornado/event_queue.py`. This custom code has a number of purposes: - Triggering [email and mobile push notifications](../subsystems/notifications.md) for any users who do not have active clients and have settings of the form "push - notifications when offline". In order to avoid doing any real + notifications when offline". In order to avoid doing any real computational work inside the Tornado codebase, this logic aims to just do the check for whether a notification should be generated, and then put an event into an appropriate @@ -65,7 +65,7 @@ number of purposes: was `mentioned`) into the events. - Handling the [local echo details](#local-echo). - Handling certain client configuration options that affect - messages. E.g. determining whether to send the + messages. E.g. determining whether to send the plaintext/Markdown raw content or the rendered HTML (e.g. the `apply_markdown` and `client_gravatar` features in our [events API docs](https://zulip.com/api/register-queue)). @@ -76,20 +76,20 @@ number of purposes: basically everything that can fail due to bad user input. - The core `do_send_messages` function (which handles actually sending the message) in `zerver/lib/actions.py` is one of the most optimized and thus complex parts of - the system. But in short, its job is to atomically do a few key + the system. But in short, its job is to atomically do a few key things: - Store a `Message` row in the database. - Store one `UserMessage` row in the database for each user who is a recipient of the message (including the sender), with appropriate `flags` for whether the user was mentioned, an alert - word appears, etc. See + word appears, etc. See [the section on soft deactivation](#soft-deactivation) for a clever optimization we use here that is important for large open organizations. - Do all the database queries to fetch relevant data for and then send a `message` event to the [events system](../subsystems/events-system.md) containing the - data it will need for the calculations described above. This + data it will need for the calculations described above. This step adds a lot of complexity, because the events system cannot make queries to the database directly. - Trigger any other deferred work caused by the current message, @@ -104,7 +104,7 @@ number of purposes: An essential feature for a good chat is experience is local echo (i.e. having the message appear in the feed the moment the user hits -send, before the network round trip to the server). This is essential +send, before the network round trip to the server). This is essential both for freeing up the compose box (for the user to send more messages) as well as for the experience to feel snappy. @@ -117,24 +117,24 @@ Zulip aims for a near-perfect local echo experience, which requires is why our [Markdown system](../subsystems/markdown.md) requires both an authoritative (backend) Markdown implementation and a secondary (frontend) Markdown implementation, the latter used only for the local -echo feature. Read our Markdown documentation for all the tricky +echo feature. Read our Markdown documentation for all the tricky details on how that works and is tested. The rest of this section details how Zulip manages locally echoed messages. - The core function in the frontend codebase - `echo.try_deliver_locally`. This checks whether correct local echo + `echo.try_deliver_locally`. This checks whether correct local echo is possible (via `markdown.contains_backend_only_syntax`) and useful (whether the message would appear in the current view), and if so, causes Zulip to insert the message into the relevant feed(s). - Since the message hasn't been confirmed by the server yet, it - doesn't have a message ID. The frontend makes one up, via + doesn't have a message ID. The frontend makes one up, via `local_message.next_local_id`, by taking the highest message ID it - has seen and adding the decimal `0.01`. The use of a floating point + has seen and adding the decimal `0.01`. The use of a floating point value is critical, because it means the message should sort correctly with other messages (at the bottom) and also won't be - duplicated by a real confirmed-by-the-backend message ID. We choose + duplicated by a real confirmed-by-the-backend message ID. We choose just above the `max_message_id`, because we want any new messages that other users send to the current view to be placed after it in the feed (this decision is somewhat arbitrary; in any case we'll @@ -143,30 +143,30 @@ messages. visually). - The `POST /messages` API request to the server to send the message is passed two special parameters that clients not implementing local - echo don't use: `queue_id` and `local_id`. The `queue_id` is the ID + echo don't use: `queue_id` and `local_id`. The `queue_id` is the ID of the client's event queue; here, it is used just as a unique identifier for the specific client (e.g. a browser tab) that sent - the message. And the `local_id` is, by the construction above, a + the message. And the `local_id` is, by the construction above, a unique value within that namespace identifying the message. - The `do_send_messages` backend code path includes the `queue_id` and `local_id` in the data it passes to the - [events system](../subsystems/events-system.md). The events + [events system](../subsystems/events-system.md). The events system will extend the `message` event dictionary it delivers to the client containing the `queue_id` with `local_message_id` field, containing the `local_id` that the relevant client used when sending - the message. This allows the client to know that the `message` + the message. This allows the client to know that the `message` event it is receiving is the same message it itself had sent. - Using that information, rather than adding the "new message" to the relevant message feed, it updates the (locally echoed) message's properties (at the very least, message ID and timestamp) and - rerenders it in any message lists where it appears. This is + rerenders it in any message lists where it appears. This is primarily done in the `process_from_server` function in `static/js/echo.js`. ### Local echo in message editing Zulip also supports local echo in the message editing code path for -edits to just the content of a message. The approach is analogous +edits to just the content of a message. The approach is analogous (using `markdown.contains_backend_only_syntax`, etc.)), except we don't need any of the `local_id` tracking logic, because the message already has a permanent message id; as a result, the whole @@ -191,7 +191,7 @@ one place: system). - The events system processes, and dispatches that event to all clients subscribed to receive notifications for users who should - receive the message (including the sender). As a side effect, it + receive the message (including the sender). As a side effect, it adds queue items to the email and push notification queues (which, in turn, may trigger those notifications). - Other clients receive the event and display the new message. @@ -201,14 +201,14 @@ one place: display timestamp to the message). - The `send_message_backend` view function returns a 200 `HTTP` response; the client receives that response and mostly - does nothing with it other than update some logging details. (This + does nothing with it other than update some logging details. (This may happen before or after the client receives the event notifying it about the new message via its event queue.) ## Message editing Message editing uses a very similar principle to how sending messages -works. A few details are worth mentioning: +works. A few details are worth mentioning: - `maybe_enqueue_notifications_for_message_update` is an analogue of `maybe_enqueue_notifications`, and exists to handle cases like a @@ -226,7 +226,7 @@ works. A few details are worth mentioning: ### Inline URL previews Zulip's inline URL previews feature (`zerver/lib/url_preview/`) uses -variant of the message editing/local echo behavior. The reason is +variant of the message editing/local echo behavior. The reason is that for inline URL previews, the backend needs to fetch the content from the target URL, and for slow websites, this could result in a significant delay in rendering the message and delivering it to other @@ -260,7 +260,7 @@ in each room, channel, or stream). We track these data in the backend in the `UserMessage` table, storing rows `(message_id, user_id, flags)`, where `flags` is 32 bits of space for boolean data like whether the user has read or starred the -message. All the key queries needed for accessing message history, +message. All the key queries needed for accessing message history, full-text search, and other key features can be done efficiently with the database indexes on this table (with joins to the `Message` table containing the actual message content where required). @@ -292,17 +292,17 @@ even simple questions). A key insight for addressing this problem is that there isn’t much of a use case for long chat discussions among 1000s of users who are all -continuously online and actively participating. Streams with a very +continuously online and actively participating. Streams with a very large number of active users are likely to only be used for occasional announcements, where some latency before everyone sees the message is -fine. Even in giant organizations, almost all messages are sent to +fine. Even in giant organizations, almost all messages are sent to smaller streams with dozens or hundreds of active users, representing some organizational unit within the community or company. However, large, active streams are common in open source projects, standards bodies, professional development groups, and other large communities with the rough structure of the Zulip development -community. These communities usually have thousands of user accounts +community. These communities usually have thousands of user accounts subscribed to all the default streams, even if they only have dozens or hundreds of those users active in any given month. Many of the other accounts may be from people who signed up just to check the @@ -312,7 +312,7 @@ be seen again. The key technical insight is that if we can make the latency scale with the number of users who actually participate in the community, not the total size of the community, then our database write limited -send latency of 1 second per 2000 users is totally fine. But we need +send latency of 1 second per 2000 users is totally fine. But we need to do this in a way that doesn’t create problems if any of the thousands of “inactive” users come back (or one of the active users sends a private message to one of the inactive users), since it’s @@ -322,7 +322,7 @@ back or will eventually be interacted with by an existing user. We solved this problem with a solution we call “soft deactivation”; users that are soft-deactivated consume less resources from Zulip in a way that is designed to be invisible both to other users and to the -user themself. If a user hasn’t logged into a given Zulip +user themself. If a user hasn’t logged into a given Zulip organization for a few weeks, they are tagged as soft-deactivated. The way this works internally is: @@ -333,24 +333,24 @@ users when a message is sent to a stream where they are subscribed. - If/when the user ever returns to Zulip, we can at that time reconstruct the UserMessage rows that they missed, and create the rows at that time (or, to avoid a latency spike if/when the user returns to -Zulip, this work can be done in a nightly cron job). We can construct +Zulip, this work can be done in a nightly cron job). We can construct those rows later because we already have the data for when the user might have been subscribed or unsubscribed from streams by other users, and, importantly, we also know that the user didn’t interact with the UI since the message was sent (and thus we can safely assume -that the messages have not been marked as read by the user). This is +that the messages have not been marked as read by the user). This is done in the `add_missing_messages` function, which is the core of the soft-deactivation implementation. - The “usually” above is because there are a few flags that result from content in the message (e.g., a message that mentions a user results in a “mentioned” flag in the UserMessage row), that we need to -keep track of. Since parsing a message can be expensive (>10ms of +keep track of. Since parsing a message can be expensive (>10ms of work, depending on message content), it would be too inefficient to need to re-parse every message when a soft-deactivated user comes back -to Zulip. Conveniently, those messages are rare, and so we can just +to Zulip. Conveniently, those messages are rare, and so we can just create UserMessage rows which would have “interesting” flags at the -time they were sent without any material performance impact. And then +time they were sent without any material performance impact. And then `add_missing_messages` skips any messages that already have a `UserMessage` row for that user when doing its backfill. @@ -358,7 +358,7 @@ The end result is the best of both worlds: - Nobody's view of the world is different because the user was soft-deactivated (resulting in no visible user-experience impact), at -least if one is running the cron job. If one does not run the cron +least if one is running the cron job. If one does not run the cron job, then users returning after being away for a very long time will potentially have a (very) slow loading experience as potentially 100,000s of UserMessage rows might need to be reconstructed at once. @@ -367,7 +367,7 @@ server only needs to do work for users who are currently interacting with Zulip. Empirically, we've found this technique completely resolved the "send -latency" scaling problem. The latency of sending a message to a stream +latency" scaling problem. The latency of sending a message to a stream now scales only with the number of active subscribers, so one can send a message to a stream with 5K subscribers of which 500 are active, and it’ll arrive in the couple hundred milliseconds one would expect if @@ -375,12 +375,12 @@ the extra 4500 inactive subscribers didn’t exist. There are a few details that require special care with this system: - [Email and mobile push - notifications](../subsystems/notifications.md). We need to make + notifications](../subsystems/notifications.md). We need to make sure these are still correctly delivered to soft-deactivated users; making this work required careful work for those code paths that assumed a `UserMessage` row would always exist for a message that triggers a notification to a given user. - Digest emails, which use the `UserMessage` table extensively to - determine what has happened in streams the user can see. We can use + determine what has happened in streams the user can see. We can use the user's subscriptions to construct what messages they should have access to for this feature. diff --git a/docs/subsystems/settings.md b/docs/subsystems/settings.md index 21ce6efdca..e35379d2c4 100644 --- a/docs/subsystems/settings.md +++ b/docs/subsystems/settings.md @@ -27,7 +27,7 @@ from shareable configuration. Zulip uses the [Django settings system](https://docs.djangoproject.com/en/2.2/topics/settings/), which means that the settings files are Python programs that set a lot of -variables with all-capital names like `EMAIL_GATEWAY_PATTERN`. You can +variables with all-capital names like `EMAIL_GATEWAY_PATTERN`. You can access these anywhere in the Zulip Django code using e.g.: ```python @@ -45,19 +45,19 @@ $ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the ~1000 lines -of settings needed by the Zulip Django app. As a result, there are a +of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settings for server administrators. In a production environment, we have: - `/etc/zulip/settings.py` (the template is in the Zulip repo at `zproject/prod_settings_template.py`) is the main system - administrator-facing settings file for Zulip. It contains all the + administrator-facing settings file for Zulip. It contains all the server-specific settings, such as how to send outgoing email, the hostname of the PostgreSQL database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in - `zproject/default_settings.py`, and then override it here. As this + `zproject/default_settings.py`, and then override it here. As this is the main documentation for Zulip settings, we recommend that production installations [carefully update `/etc/zulip/settings.py` every major @@ -66,9 +66,9 @@ In a production environment, we have: - `/etc/zulip/zulip-secrets.conf` (generated by `scripts/setup/generate_secrets.py` as part of installation) - contains secrets used by the Zulip installation. These are read + contains secrets used by the Zulip installation. These are read using the standard Python `ConfigParser`, and accessed in - `zproject/computed_settings.py` by the `get_secret` function. All + `zproject/computed_settings.py` by the `get_secret` function. All secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the `get_secret` function in `zproject/config.py`. @@ -96,11 +96,11 @@ additionally: environment; these are set after importing `prod_settings_template.py`. - `zproject/dev-secrets.conf` replaces - `/etc/zulip/zulip-secrets.conf`, and is not tracked by Git. This + `/etc/zulip/zulip-secrets.conf`, and is not tracked by Git. This allows you to configure your development environment to support features like [authentication options](../development/authentication.md) that require secrets to - work. It is also used to set certain settings that in production + work. It is also used to set certain settings that in production belong in `/etc/zulip/settings.py`, e.g. `SOCIAL_AUTH_GITHUB_KEY`. You can see a full list with `git grep development_only=True`, or add additional settings of this form if needed. @@ -139,10 +139,10 @@ want those settings. ### Testing non-default settings You can write tests for settings using e.g. -`with self.settings(TERMS_OF_SERVICE=None)`. However, this only works +`with self.settings(TERMS_OF_SERVICE=None)`. However, this only works for settings which are checked at runtime, not settings which are only accessed in initialization of Django (or Zulip) internals -(e.g. `DATABASES`). See the [Django docs on overriding settings in +(e.g. `DATABASES`). See the [Django docs on overriding settings in tests][django-test-settings] for more details. [django-test-settings]: https://docs.djangoproject.com/en/2.2/topics/testing/tools/#overriding-settings @@ -151,9 +151,9 @@ tests][django-test-settings] for more details. Realm settings are preferred for any configuration that is a matter of organizational policy (as opposed to technical capabilities of the -server). As a result, configuration options for user-facing +server). As a result, configuration options for user-facing functionality is almost always added as a new realm setting, not a -server setting. The [new feature tutorial][doc-newfeat] documents the +server setting. The [new feature tutorial][doc-newfeat] documents the process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be diff --git a/docs/subsystems/typing-indicators.md b/docs/subsystems/typing-indicators.md index f12124ed04..710de2f41c 100644 --- a/docs/subsystems/typing-indicators.md +++ b/docs/subsystems/typing-indicators.md @@ -3,12 +3,12 @@ Zulip supports a feature called "typing indicators." Typing indicators are status messages that tell you when -another user is composing a message to you. Zulip's typing UI +another user is composing a message to you. Zulip's typing UI is similar to what you see in other chat/text systems. This document describes how we have implemented the feature in Zulip, and our main audience is developers who want to understand the -system and possibly improve it. This document assumes that the +system and possibly improve it. This document assumes that the client is our web app, but any client can play along with this protocol. @@ -22,7 +22,7 @@ There are two major roles for users in this system: ready to shift their attention elsewhere). Any Zulip user can play either one of these roles, and sometimes -they can be playing both roles at once. Having said that, you +they can be playing both roles at once. Having said that, you can generally understand the system in terms of a single message being composed by the "writing user." @@ -42,26 +42,26 @@ function that facilitates this is called `send_typing_notification_ajax`. If the "writing user" is composing a long message, we want to send repeated updates to the server, so that downstream clients know that the -user is still typing. (Zulip messages tend to be longer than +user is still typing. (Zulip messages tend to be longer than messages in other chat/text clients, so this detail is important.) We have a small state machine in `static/shared/js/typing_status.js` that makes sure subsequent "start" requests get sent out every ten -seconds. (This document is intended to describe the high level +seconds. (This document is intended to describe the high level architecture; the actual time values may be tuned in future releases. See the constant `TYPING_STARTED_WAIT_PERIOD`, for example.) If the "writing user" goes more than five seconds without any text -input, then we send a request with an `op` of `stop`. We also send +input, then we send a request with an `op` of `stop`. We also send "stop" messages when the user explicitly aborts composing a message by closing the compose box (or other actions). A common scenario is that a user is just pausing to think for a few -seconds, but they still intend to finish the message. Of course, +seconds, but they still intend to finish the message. Of course, that's hard to distinguish from the scenario of the user got pulled -away from their desk. For the former case, where the "writing user" +away from their desk. For the former case, where the "writing user" completes the message with lots of pauses for thinking, a series of -"start" and "stop" messages may be sent over time. Timeout values +"start" and "stop" messages may be sent over time. Timeout values reflect tradeoffs, where we have to guess how quickly people type, how long they pause to think, and how frequently they get interrupted. @@ -79,15 +79,15 @@ As such, the server piece here is basically a single Django view function with a small bit of library support to send out events to clients. -Requests come into `/json/typing`. The view mostly calls out +Requests come into `/json/typing`. The view mostly calls out to `check_send_typing_notification` to do the heavy lifting. One of the main things that the server does is to simply validate that the `to` users are for valid, active users in the realm. Once the request has been validated, the server sends events to -potential recipients of the message. The event type for that -payload is `typing`. See the function `do_send_typing_notification` +potential recipients of the message. The event type for that +payload is `typing`. See the function `do_send_typing_notification` for more details. ## Receiving user @@ -115,7 +115,7 @@ The main goal is then to triage which events should lead to display changes. The web app client maintains a list of incoming "typists" using -code in `static/js/typing_data.js`. The API here has functions +code in `static/js/typing_data.js`. The API here has functions like the following: - `add_typist` @@ -124,9 +124,9 @@ like the following: - `get_all_typists` One subtle thing that the client has to do here is to maintain -timers for typing notifications. The constant +timers for typing notifications. The constant `TYPING_STARTED_EXPIRY_PERIOD` is used to determine that the -"writing user" has abandoned the message. Of course, the +"writing user" has abandoned the message. Of course, the "writing user" will also explicitly send us `stop` notifications at certain times. @@ -141,8 +141,8 @@ that affect the overall system. In general, client developers should agree on timeout parameters for how frequently we "kickstart" typing notifications for users -sending long messages. This means standardizing the "writing -user" piece of the system. It's possible that certain clients +sending long messages. This means standardizing the "writing +user" piece of the system. It's possible that certain clients will have slightly different mechanisms for detecting that users have abandoned messages, but the re-transmit frequency should be similar. @@ -150,7 +150,7 @@ similar. When implementing the "receiving user" piece, it's important to realize how clients behave on the other end of the protocol. It's possible, for example, to never receive a "stop" notification -from a client that was shut down abruptly. You should allow +from a client that was shut down abruptly. You should allow reasonable amounts of time for the other side to send notifications, allowing for network delays and server delays, but you should not let the notifications become too "sticky" either. @@ -158,22 +158,22 @@ not let the notifications become too "sticky" either. ## Roadmap This document is being written just under a year after typing -indicators were first implemented. The feature has been popular, +indicators were first implemented. The feature has been popular, and after some initial cleanup, it has not required a lot of maintenance. The most likely big change to typing indicators is that we will -add them for stream conversations. This will require some thought +add them for stream conversations. This will require some thought for large streams, in terms of both usability and performance. Another area for refinement is to tune the timing values a bit. Right now we are probably too aggressive about sending `stop` -messages, when users often are just pausing. It's possible +messages, when users often are just pausing. It's possible to better account for typing speed or other heuristic things like how much of the message has already been typed. From an infrastructure perspective, we should be mindful of -bandwidth concerns. A fairly easy change would be to send +bandwidth concerns. A fairly easy change would be to send user ids instead of emails. Some folks may want to turn off typing indicators, so we will diff --git a/docs/subsystems/widgets.md b/docs/subsystems/widgets.md index 87dbd33094..58a772b531 100644 --- a/docs/subsystems/widgets.md +++ b/docs/subsystems/widgets.md @@ -1,11 +1,11 @@ # Widgets (experimental) [Note: this document is currently intended to be a roadmap/design -document. It may be converted over time to permanent documentation.] +document. It may be converted over time to permanent documentation.] ## Overview -During 2018 we built out a "widget system" in Zulip. It includes +During 2018 we built out a "widget system" in Zulip. It includes these features: - **/ping** @@ -18,15 +18,15 @@ of this writing. There's a strong overlap between **widgets** and **slash commands**, and many widgets are launched by slash commands. -A few exceptions are worth noting. If you type "/me shrugs" +A few exceptions are worth noting. If you type "/me shrugs" in the compose box, it's just a message that gets -slightly customized rendering. And +slightly customized rendering. And if you type "/settings", it's just a shortcut to open -the settings popup. Neither of these are really "widgets," +the settings popup. Neither of these are really "widgets," per se. Another exception, in the opposite direction, is our -trivia_quiz bot. It does not involve slash commands. +trivia_quiz bot. It does not involve slash commands. Instead it sends "extra_data" in messages to invoke **zforms** (which enable button-based UIs in the messages). @@ -62,7 +62,7 @@ They send commands to the server using the `/json/command` endpoint. In the case of "/ping", the server code in `zcommand.py` -basically just acks the client. The client then computes +basically just acks the client. The client then computes the round trip time and shows a little message above the compose box that the user can see and then dismiss. @@ -70,21 +70,21 @@ For commands like "/day" and "/night", the server does a little bit of logic to toggle the user's night mode setting, and this is largely done inside `zcommand.py`. The server sends a very basic response, and then -the client actually changes the display colors. The +the client actually changes the display colors. The client also shows the user a little message above the compose box instructing them how to reverse the change. It's a bit of a stretch to label "/ping" and "/day" -as **widgets**. In some ways they're just compose-box -shortcuts for doing UI tasks. The commands share +as **widgets**. In some ways they're just compose-box +shortcuts for doing UI tasks. The commands share the new "zcommand" namespace in the code, and both have some common UI for talking to users. (It's possible that we don't really need a general `/json/zcommand` endpoint for these, and we may decide later to just use custom -API endpoints for each command. There's some logic +API endpoints for each command. There's some logic in having a central API for these, though, since they are typically things that only UI-based clients will invoke, and they may share validation code.) @@ -92,7 +92,7 @@ invoke, and they may share validation code.) ### Availability The above commands are available for all Zulip servers -that use 1.9 or above. You must use the webapp client to +that use 1.9 or above. You must use the webapp client to get the features; other clients will send the messages without any translation (e.g. "/day" will just be a message that says "/day" if you use the mobile app). @@ -124,29 +124,29 @@ We'll use the poll widget as a concrete example. The `SubMessage` table, as the name indicates, allows you to associate multiple submessages to any given -`Message` row. When a message gets sent, there's a +`Message` row. When a message gets sent, there's a hook inside of `widget.py` that will detect slash -commands like "/poll". If a message needs to be +commands like "/poll". If a message needs to be widgetized, an initial `SubMessage` row will be created with an appropriate `msg_type` (and persisted -to the database). This data will also be included -in the normal Zulip message event payload. Clients +to the database). This data will also be included +in the normal Zulip message event payload. Clients can choose to ignore the submessage-related data, in which case they'll gracefully degrade to seeing "/poll". Of course, the webapp client actually recognizes the appropriate widgets. The webapp client will next collect poll options and votes -from users. The webapp client has +from users. The webapp client has code in `submessage.js` that dispatches events to `widgetize.js`, which in turns sends events to -individual widgets. The widgets know how to render +individual widgets. The widgets know how to render themselves and set up click/input handlers to collect -data. They can then post back to `/json/submessage` +data. They can then post back to `/json/submessage` to attach more data to the message (and the -details are encapsulated with a callback). The server +details are encapsulated with a callback). The server will continue to persist `SubMessage` rows in the -database. These rows are encoded as JSON, and the +database. These rows are encoded as JSON, and the schema of the messages is driven by the individual widgets. Most of the logic is in the client; things are fairly opaque to the server at this point. @@ -156,31 +156,31 @@ Our todo list widget uses the same architecture as "poll". If a client joins Zulip after a message has accumulated several submessage events, it will see all of those -events the first time it sees the parent message. Clients +events the first time it sees the parent message. Clients need to know how to build/rebuild their state as each -submessage comes in. They also need to tolerate +submessage comes in. They also need to tolerate misformatted data, ideally just dropping data on the floor. If a widget throws an exception, it's caught before the rest of the message feed is affected. As far as rendering is concerned, each widget module is given a parent `elem` when its `activate` function -is called. This is just a `
` inside of the parent -message in the message pane. The widget has access to +is called. This is just a `
` inside of the parent +message in the message pane. The widget has access to jQuery and template.render, and the developer can create new templates in `static/templates/widgets/`. A good way to learn the system is to read the code -in `static/js/poll_widget.js`. It is worth noting that +in `static/js/poll_widget.js`. It is worth noting that writing a new widget requires only minor backend -changes in the current architecture. This could change +changes in the current architecture. This could change in the future, but for now a frontend developer mostly needs to know JS, CSS, and HTML. It may be useful to think of widgets in terms of a -bunch of clients exchanging peer-to-peer messages. The +bunch of clients exchanging peer-to-peer messages. The server's only real role is to decide who gets delivered -which submessages. It's a lot like a "subchat" system. +which submessages. It's a lot like a "subchat" system. ### Backward compatibility @@ -190,14 +190,14 @@ breaking old messages. Widget developers can revise code to improve a widget's visual polish without too much concern -for breaking how old messages get widgetized. They will need to +for breaking how old messages get widgetized. They will need to be more cautious if they change the actual data structures passed around in the submessage payloads. For significant schema changes, it would be worthwhile to add some kind of versioning scheme inside of `SubMessages`, either at the DB level or more at the JSON level within fields. -This has yet to be designed. One thing to consider is that +This has yet to be designed. One thing to consider is that most widgets are somewhat ephemeral in nature, so it's not the end of the world if upgrades cause some older messages to be obsolete, as long as the code degrades gracefully. @@ -205,7 +205,7 @@ to be obsolete, as long as the code degrades gracefully. Mission critical widgets should have a deprecation strategy. For example, you could add optional features for one version bump and then only make them mandatory for the next version, -as long as you don't radically change the data model. And +as long as you don't radically change the data model. And if you're truly making radical changes, you can always write a Django migration for the `SubMessage` data. @@ -216,7 +216,7 @@ they are served up by the core Zulip server implementation. Of course, anybody who wishes to build their own widget has the option of forking the server code and self-hosting, but we want to encourage folks to submit widget -code to our codebase in PRs. If we get to a critical mass +code to our codebase in PRs. If we get to a critical mass of contributed widgets, we will want to explore a more dynamic mechanism for "plugging in" code from outside sources, but that is not in our immediate roadmap. @@ -225,23 +225,23 @@ This is sort of a segue to the next section of this document. Suppose you want to write your own custom bot, and you want to allow users to click buttons to respond to options, but you don't want to have to modify the Zulip server codebase -to turn on those features. This is where our "zform" +to turn on those features. This is where our "zform" architecture comes to the rescue. ## zform (trivia quiz bot) This section will describe our "zform" architecture. -For context, imagine a naive triva bot. The trivia bot +For context, imagine a naive triva bot. The trivia bot sends a question with the answers labeled as A, B, C, -and D. Folks who want to answer the bot send back an +and D. Folks who want to answer the bot send back an answer have to send an actual Zulip message with something like `@trivia_bot answer A to Q01`, which is kind of -tedious to type. Wouldn't it be nice if the bot could +tedious to type. Wouldn't it be nice if the bot could serve up some kind of buttons with canned replies, so that the user just hits a button? -That is where zforms come in. Zulip's trivia bot sends +That is where zforms come in. Zulip's trivia bot sends the Zulip server a JSON representation of a form it wants rendered, and then the client renders a generic "**zform**" with buttons corresponding to `short_name` fields @@ -288,7 +288,7 @@ Here is what an example payload looks like: When users click on the buttons, **generic** click handlers automatically simulate a client reply using a field called `reply` (in `choices`) as the content -of the message reply. Then the bot sees the reply +of the message reply. Then the bot sees the reply and grades the answer using ordinary chat-bot coding. The beautiful thing is that any third party developer @@ -361,7 +361,7 @@ recipients of the parent message. When the message gets to the client, the codepath for **zform** is actually quite similar to what happens with a more -customized widget like **poll**. (In fact, **zform** is a +customized widget like **poll**. (In fact, **zform** is a sibling of **poll** and **zform** just has a somewhat more generic job to do.) In `static/js/widgetize.js` you will see where this code converges, with snippets like this: @@ -407,5 +407,5 @@ such as the "/me" command. If certain widget features are behind feature flags, this will slightly complicate the typeahead -implementation. Mostly we just need the server +implementation. Mostly we just need the server to share any relevant settings with the client. diff --git a/docs/testing/continuous-integration.md b/docs/testing/continuous-integration.md index cb5497c3b7..f7ca8f63dd 100644 --- a/docs/testing/continuous-integration.md +++ b/docs/testing/continuous-integration.md @@ -8,7 +8,7 @@ GitHub Actions and debugging issues with it. ## Goals The overall goal of our CI is to avoid regressions and minimize the -total time spent debugging Zulip. We do that by trying to catch as +total time spent debugging Zulip. We do that by trying to catch as many possible future bugs as possible, while minimizing both latency and false positives, both of which can waste a lot of developer time. There are a few implications of this overall goal: @@ -28,14 +28,14 @@ run to iteratively debug something. - GitHub Actions stores timestamps for every line in the logs. They are hidden by default; you can see them by toggling the -`Show timestamps` option in the menu on any job's log page. (You can +`Show timestamps` option in the menu on any job's log page. (You can get this sort of timestamp in a development environment by piping output to `ts`). - GitHub Actions runs on every branch you push on your Zulip fork. This is helpful when debugging something complicated. -- You can also ssh into a container to debug failures. SSHing into +- You can also ssh into a container to debug failures. SSHing into the containers can be helpful, especially in rare cases where the tests are passing in your computer but failing in the CI. There are various @@ -108,7 +108,7 @@ generated Dockerfiles. An important element of making GitHub Actions perform effectively is caching between jobs the various caches that live under `/srv/` in a Zulip -development or production environment. In particular, we cache the +development or production environment. In particular, we cache the following: - Python virtualenvs @@ -122,7 +122,7 @@ We have designed these caches carefully (they are also used in production and the Zulip development environment) to ensure that each is named by a hash of its dependencies and ubuntu distribution name, so Zulip should always be using the same version of dependencies it -would have used had the cache not existed. In practice, bugs are +would have used had the cache not existed. In practice, bugs are always possible, so be mindful of this possibility. A consequence of this caching is that test jobs for branches which diff --git a/docs/testing/linters.md b/docs/testing/linters.md index 33073ae4b6..b9bf5a5683 100644 --- a/docs/testing/linters.md +++ b/docs/testing/linters.md @@ -11,7 +11,7 @@ but for other files, we are quite thorough about checking semantic correctness. Obviously, a large reason for linting code is to enforce the [Zulip -coding standards](../contributing/code-style.md). But we also use the linters to +coding standards](../contributing/code-style.md). But we also use the linters to prevent common coding errors. We borrow some open source tools for much of our linting, and the links @@ -44,7 +44,7 @@ You can also run them individually or pass specific files: ``` `./tools/lint` has many useful options; you can read about them in its -internal documentation using `./tools/lint --help`. Of particular +internal documentation using `./tools/lint --help`. Of particular note are: - `--fix`: Several of our linters support automatically fixing basic issues; this option will ask `tools/lint` to run those. @@ -82,9 +82,9 @@ And, of course, for minor oversights, `lint` is your friend, not your foe. Occasionally, our linters will complain about things that are more of an artifact of the linter limitations than any actual problem with your -code. There is usually a mechanism where you can bypass the linter in +code. There is usually a mechanism where you can bypass the linter in extreme cases, but often it can be a simple matter of writing your code -in a slightly different style to appease the linter. If you have +in a slightly different style to appease the linter. If you have problems getting something to lint, you can submit an unfinished PR and ask the reviewer to help you work through the lint problem, or you can find other people in the [Zulip Community](../contributing/chat-zulip-org.md) @@ -95,12 +95,12 @@ find limitations in either the Zulip home-grown stuff or our third party tools, feedback will be highly appreciated. Finally, one way to clean up your code is to thoroughly exercise it -with tests. The [Zulip test documentation](../testing/testing.md) +with tests. The [Zulip test documentation](../testing/testing.md) describes our test system in detail. ## Lint checks -Most of our lint checks get performed by `./tools/lint`. These include the +Most of our lint checks get performed by `./tools/lint`. These include the following checks: - Check Python code with pyflakes. @@ -113,10 +113,10 @@ following checks: - Check HTML templates for matching tags and indentations. - Check CSS for parsability and formatting. - Check JavaScript code for addClass calls. -- Running `mypy` to check static types in Python code. Our +- Running `mypy` to check static types in Python code. Our [documentation on using mypy](../testing/mypy.md) covers mypy in more detail. -- Running `tsc` to compile TypeScript code. Our [documentation on +- Running `tsc` to compile TypeScript code. Our [documentation on TypeScript](typescript.md) covers TypeScript in more detail. The rest of this document pertains to the checks that occur in `./tools/lint`. @@ -134,14 +134,14 @@ In order for our entire lint suite to run in a timely fashion, the `lint` script performs several lint checks in parallel by forking out subprocesses. Note that our project does custom regex-based checks on the code, and we -also customize how we call pyflakes and pycodestyle (pep8). The code for these +also customize how we call pyflakes and pycodestyle (pep8). The code for these types of checks mostly lives [here](https://github.com/zulip/zulip/tree/main/tools/linter_lib). ### Special options -You can use the `-h` option for `lint` to see its usage. One particular +You can use the `-h` option for `lint` to see its usage. One particular flag to take note of is the `--modified` flag, which enables you to only run -lint checks against files that are modified in your Git repo. Most of the +lint checks against files that are modified in your Git repo. Most of the "sub-linters" respect this flag, but some will continue to process all the files. Generally, a good workflow is to run with `--modified` when you are iterating on the code, and then run without that option right before committing new code. @@ -157,7 +157,7 @@ various file types. #### Generic source code checks -We check almost our entire codebase for trailing whitespace. Also, we +We check almost our entire codebase for trailing whitespace. Also, we disallow tab (\t) characters in all but two files. We also have custom regex-based checks that apply to specific file types. @@ -165,7 +165,7 @@ For relatively minor files like Markdown files and JSON fixtures, this is the extent of our checking. Finally, we're checking line length in Python code (and hope to extend -this to other parts of the codebase soon). You can use +this to other parts of the codebase soon). You can use `#ignorelinelength` for special cases where a very long line makes sense (e.g. a link in a comment to an extremely long URL). @@ -173,19 +173,19 @@ sense (e.g. a link in a comment to an extremely long URL). Our Python code is formatted using Black (using the options in the `[tool.black]` section of `pyproject.toml`) and isort (using the -options in `.isort.cfg`). The `lint` script enforces this by running +options in `.isort.cfg`). The `lint` script enforces this by running Black and isort in check mode, or in write mode with `--fix`. -The bulk of our Python linting gets outsourced to the "pyflakes" tool. We +The bulk of our Python linting gets outsourced to the "pyflakes" tool. We call "pyflakes" in a fairly vanilla fashion, and then we post-process its output to exclude certain specific errors that Zulip is comfortable ignoring. Zulip also has custom regex-based rules that it applies to Python code. -Look for `python_rules` in the source code for `lint`. Note that we +Look for `python_rules` in the source code for `lint`. Note that we provide a mechanism to exclude certain lines of codes from these checks. Often, it is simply the case that our regex approach is too crude to -correctly exonerate certain valid constructs. In other cases, the code +correctly exonerate certain valid constructs. In other cases, the code that we exempt may be deemed not worthwhile to fix. #### JavaScript code @@ -198,7 +198,7 @@ We check our JavaScript code in a few different ways: #### Puppet manifests We use Puppet as our tool to manage configuration files, using -Puppet "manifests." To lint Puppet manifests, we use the "parser validate" +Puppet "manifests." To lint Puppet manifests, we use the "parser validate" option of Puppet. #### HTML templates @@ -209,7 +209,7 @@ Zulip uses two HTML templating systems: - [handlebars](https://handlebarsjs.com/) Zulip has an internal tool that validates both types of templates for -correct indentation and matching tags. You can find the code here: +correct indentation and matching tags. You can find the code here: - driver: [check-templates](https://github.com/zulip/zulip/blob/main/tools/check-templates) - engine: [lib/template_parser.py](https://github.com/zulip/zulip/blob/main/tools/lib/template_parser.py) @@ -227,7 +227,7 @@ for the rules we currently enforce. #### Shell scripts Zulip uses [shellcheck](https://github.com/koalaman/shellcheck) to -lint our shell scripts. We recommend the +lint our shell scripts. We recommend the [shellcheck gallery of bad code][shellcheck-bad-code] as a resource on how to not write bad shell. diff --git a/docs/testing/manual-testing.md b/docs/testing/manual-testing.md index 8ecf9739fb..01f1f43617 100644 --- a/docs/testing/manual-testing.md +++ b/docs/testing/manual-testing.md @@ -1,14 +1,14 @@ # Manual testing # As a general rule, we like to have automated tests for everything that -can be practically tested. However, there are certain types of bugs +can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called -manual QA). Manual testing not only catches bugs, but it also helps +manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they're working on. This doc assumes you know how to set up a local development server -and open the Zulip app in the browser. It also assumes a basic +and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip. ## Basic stuff ## @@ -16,24 +16,24 @@ knowledge of how to use Zulip. When testing Zulip manually, here are things to focus on: - The best bugs to catch are security/permissions bugs. -- Don't rush manual testing. Look for small details like +- Don't rush manual testing. Look for small details like display glitches. - Always test with multiple users (you can use incognito windows to facilitate this). - Always keep the inspector console open and watch for warnings or errors. -- Be methodical about collecting information on bugs. (You will +- Be methodical about collecting information on bugs. (You will eventually want to create tickets, but you may want to consolidate your own notes before filing tickets.) You generally want to test with Cordelia as the primary user, -and use Hamlet as her primary conversation partner. Use Iago -when you need to test administrative functions. Send messages +and use Hamlet as her primary conversation partner. Use Iago +when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her. The rest of this document groups tasks into basic areas of -functionality of the system. If you have multiple people testing +functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc. ### Message view ### @@ -150,9 +150,9 @@ Here are some tasks: Zulip uses the term "narrowing" to refer to opening different views of your messages, whether by clicking on sidebar options, recipient -bars, or by using search. The main focus of these tasks should -be watching unread counts. Of course, you also want to see messages -show up in the message pane. And, finally, you should make sure +bars, or by using search. The main focus of these tasks should +be watching unread counts. Of course, you also want to see messages +show up in the message pane. And, finally, you should make sure that no messages outside the narrow show up in Cordelia's view. @@ -191,14 +191,14 @@ messages after each narrow): - Go to Denmark view. - Go to Denmark/foo view. -There are 56 things to test here. If you can get into a rhythm +There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs. ### Composing messages ### We have pretty good automated tests for our Markdown processor, so -manual testing is targeted more to other interactions. For composing +manual testing is targeted more to other interactions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed. @@ -258,9 +258,9 @@ populated and where the focus is placed. ### Popover menus ### For this task you just want to go through all of our popover menus -and exercise them. The main nuance here is that you occasionally want +and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if -the popover menu is "too sticky." Also, occasionally actions will be +the popover menu is "too sticky." Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view. @@ -307,7 +307,7 @@ Here are the things to test: ### Sidebar filtering ### This is a fairly quick task where we test the search filters on the left sidebar -and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to +and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that stream. - Streams filtering @@ -350,7 +350,7 @@ First, we start off with "positive" tests. users. For negative tests, we want to dig a little deeper to find back -doors for Cordelia to access the stream. Here are some techniques +doors for Cordelia to access the stream. Here are some techniques to try: - Try to have her compose a message to the stream by @@ -360,7 +360,7 @@ to try: - Go to stream settings and see if the stream shows up. For public streams, it's ok for Cordelia to know the stream exists, -and she can subsequently subscribe. For private streams, she should +and she can subsequently subscribe. For private streams, she should not even know they exist (until she's invited, of course). - Negative tests @@ -381,9 +381,9 @@ not even know they exist (until she's invited, of course). ### Search ### The main task for testing search is to play around with search -suggestions (autocomplete). Once you select an option, verify the +suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar -reflects the current narrow. If a search comes up legitimately +reflects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search. Here are searches you should be able to do with autocomplete: @@ -531,7 +531,7 @@ Here are the tasks: ### To be continued... ### -This document does not cover settings/admin options yet. The main +This document does not cover settings/admin options yet. The main things to do when testing the settings system are: - Verify that changes are synced to other users. - Verify error messages appear if you do something wrong and look right. diff --git a/docs/testing/mypy.md b/docs/testing/mypy.md index 835368e2c1..77fd38e8b3 100644 --- a/docs/testing/mypy.md +++ b/docs/testing/mypy.md @@ -1,9 +1,9 @@ # Python static type checker (mypy) [mypy](http://mypy-lang.org/) is a compile-time static type checker -for Python, allowing optional, gradual typing of Python code. Zulip +for Python, allowing optional, gradual typing of Python code. Zulip was fully annotated with mypy's Python 2 syntax in 2016, before our -migration to Python 3 in late 2017. In 2018 and 2020, we migrated +migration to Python 3 in late 2017. In 2018 and 2020, we migrated essentially the entire codebase to the nice PEP 484 (Python 3 only) and PEP 526 (Python 3.6) syntax for static types: @@ -19,7 +19,7 @@ You can learn more about it at: - The [mypy cheat sheet for Python 3](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html) is the best resource for quickly understanding how to write the PEP - 484 type annotations used by mypy correctly. The + 484 type annotations used by mypy correctly. The [Python 2 cheat sheet](https://mypy.readthedocs.io/en/latest/cheat_sheet.html) is useful for understanding the type comment syntax needed for our few modules that need to support both Python 2 and 3. @@ -38,7 +38,7 @@ CI testing process in the `backend` build. ## Installing mypy -mypy is installed by default in the Zulip development environment. If +mypy is installed by default in the Zulip development environment. If you'd like to install just the version of `mypy` that we're using (useful if e.g. you want `mypy` installed on your laptop outside the Vagrant guest), you can do that with @@ -52,7 +52,7 @@ To run mypy on Zulip's python code, you can run the command: tools/run-mypy ``` -Mypy outputs errors in the same style as a compiler would. For +Mypy outputs errors in the same style as a compiler would. For example, if your code has a type error like this: ```python @@ -70,7 +70,7 @@ test.py:200: error: Incompatible types in assignment (expression has type "str", ## Mypy is there to find bugs in Zulip before they impact users For the purposes of Zulip development, you can treat `mypy` like a -much more powerful linter that can catch a wide range of bugs. If, +much more powerful linter that can catch a wide range of bugs. If, after running `tools/run-mypy` on your Zulip branch, you get mypy errors, it's important to get to the bottom of the issue, not just do something quick to silence the warnings, before we merge the changes. @@ -86,7 +86,7 @@ Possible explanations include: Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way that makes the Zulip -codebase better. If you're having trouble, silence the warning with +codebase better. If you're having trouble, silence the warning with an `Any` or `# type: ignore[code]` so you're not blocked waiting for help, add a `# TODO: ` comment so it doesn't get forgotten in code review, and ask for help in chat.zulip.org. @@ -105,7 +105,7 @@ root of the project, letting `mypy` know that it's third-party code, or add type stubs to the `stubs/` directory, which has type stubs that mypy can use to type-check calls into that third-party module. -It's easy to add new stubs! Just read the docs, look at some of +It's easy to add new stubs! Just read the docs, look at some of existing examples to see how they work, and remember to remove the `ignore_missing_imports` entry in `mypy.ini` when you add them. @@ -116,9 +116,9 @@ to use `mypy`!), but means the code can't be fully type-checked. ## `type_debug.py` -`zerver/lib/type_debug.py` has a useful decorator `print_types`. It +`zerver/lib/type_debug.py` has a useful decorator `print_types`. It prints the types of the parameters of the decorated function and the -return type whenever that function is called. This can help find out +return type whenever that function is called. This can help find out what parameter types a function is supposed to accept, or if parameters with the wrong types are being passed to a function. @@ -145,8 +145,8 @@ func([int, ...], [int, ...]) -> [int, ...] [1, 2, 3, 4, 5, 6, 7] ``` -`print_all` prints the type of the first item of lists. So `[int, ...]` represents -a list whose first element's type is `int`. Types of all items are not printed +`print_all` prints the type of the first item of lists. So `[int, ...]` represents +a list whose first element's type is `int`. Types of all items are not printed because a list can have many elements, which would make the output too large. Similarly in dicts, one key's type and the corresponding value's type are printed. @@ -156,13 +156,13 @@ So `{1: 'a', 2: 'b', 3: 'c'}` will be printed as `{int: str, ...}`. Sometimes, a function's type is most precisely expressed as a few possibilities, and which possibility can be determined by looking at -the arguments. You can express that idea in a way mypy understands -using `@overload`. For example, `check_list` returns a `Validator` +the arguments. You can express that idea in a way mypy understands +using `@overload`. For example, `check_list` returns a `Validator` function that verifies that an object is a list, raising an exception if it isn't. It supports being passed a `sub_validator`, which will verify that -each element in the list has a given type as well. One can express +each element in the list has a given type as well. One can express the idea "If `sub_validator` validates that something is a `ResultT`, `check_list(sub_validator)` validators that something is a `List[ResultT]` as follows: @@ -186,7 +186,7 @@ type logic for the case where we are passed a `sub_validator`. **Warning:** Mypy only checks the body of an overloaded function against the final signature and not against the more restrictive -`@overload` signatures. This allows some type errors to evade +`@overload` signatures. This allows some type errors to evade detection by mypy: ```python @@ -201,7 +201,7 @@ x: int = f("three!!") ``` Due to this potential for unsafety, we discourage overloading unless -it's absolutely necessary. Consider writing multiple functions with +it's absolutely necessary. Consider writing multiple functions with different names instead. See the [mypy overloading documentation][mypy-overloads] for more details. @@ -213,12 +213,12 @@ See the [mypy overloading documentation][mypy-overloads] for more details. ### When is a type annotation justified? Usually in fully typed code, mypy will protect you from writing a type -annotation that isn't justified by the surrounding code. But when you +annotation that isn't justified by the surrounding code. But when you need to write annotations at the border between untyped and typed code, keep in mind that **a type annotation should always represent a -guarantee,** not an aspiration. If you have validated that some value -is an `int`, it can go in an `int` annotated variable. If you are -going to validate it later, it should not. When in doubt, an `object` +guarantee,** not an aspiration. If you have validated that some value +is an `int`, it can go in an `int` annotated variable. If you are +going to validate it later, it should not. When in doubt, an `object` annotation is always safe. Mypy understands many Python constructs like `assert`, `if`, @@ -238,7 +238,7 @@ def f(x: object, y: Optional[str]) -> None: It won't be able do this narrowing if the validation is hidden behind a function call, so sometimes it's helpful for a validation function to return the type-narrowed value back to the caller even though the -caller already has it. (The validators in `zerver/lib/validator.py` +caller already has it. (The validators in `zerver/lib/validator.py` are examples of this pattern.) ### Avoid the `Any` type @@ -248,7 +248,7 @@ type](https://mypy.readthedocs.io/en/stable/dynamic_typing.html) for interoperability with untyped code, but it is completely unchecked. You can put an value of an arbitrary type into an expression of type `Any`, and get an value of an arbitrary type out, and mypy will make -no effort to check that the input and output types match. So using +no effort to check that the input and output types match. So using `Any` defeats the type safety that mypy would otherwise provide. ```python @@ -281,8 +281,8 @@ alternatives first: `instance(value, str)` test. - If you really have no information about the type of a value, use the - **`object` type**. Since every type is a subtype of `object`, you - can correctly annotate any value as `object`. The [difference + **`object` type**. Since every type is a subtype of `object`, you + can correctly annotate any value as `object`. The [difference between `Any` and `object`](https://mypy.readthedocs.io/en/stable/dynamic_typing.html#any-vs-object) is that mypy will check that you safely validate an `object` with @@ -290,17 +290,17 @@ alternatives first: type. - A common way for `Any` annotations to sneak into your code is the - interaction with untyped third-party libraries. Mypy treats any + interaction with untyped third-party libraries. Mypy treats any value imported from an untyped library as annotated with `Any`, and treats any type imported from an untyped library as equivalent to - `Any`. Consider providing real type annotations for the library by + `Any`. Consider providing real type annotations for the library by [**writing a stub file**](#mypy-stubs-for-third-party-modules). ### Avoid `cast()` The [`cast` function](https://mypy.readthedocs.io/en/stable/casts.html) lets you -provide an annotation that Mypy will not verify. Obviously, this is +provide an annotation that Mypy will not verify. Obviously, this is completely unsafe in general. ```python @@ -329,7 +329,7 @@ Mypy allows you to ignore any type checking error with a [`# type: ignore` comment](https://mypy.readthedocs.io/en/stable/common_issues.html#spurious-errors-and-locally-silencing-the-checker), but you should avoid this in the absence of a very good reason, such -as a bug in mypy itself. If there are no safe options for dealing +as a bug in mypy itself. If there are no safe options for dealing with the error, prefer an unchecked `cast`, since its unsafety is somewhat more localized. @@ -347,7 +347,7 @@ issue. not checked against the `@overload` signatures. - **Avoid `Callable[..., T]`** (with literal ellipsis `...`), since - mypy cannot check the types of arguments passed to it. Provide the + mypy cannot check the types of arguments passed to it. Provide the specific argument types (`Callable[[int, str], T]`) in simple cases, or use [callback protocols](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols) @@ -357,11 +357,11 @@ issue. The [`Optional` type](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#built-in-types) -is for optional values, which are values that could be `None`. For +is for optional values, which are values that could be `None`. For example, `Optional[int]` is equivalent to `Union[int, None]`. The `Optional` type is **not for optional parameters** (unless they -are also optional values as above). This signature does not use the +are also optional values as above). This signature does not use the `Optional` type: ```python @@ -371,7 +371,7 @@ def func(flag: bool = False) -> str: A collection such as `List` should only be `Optional` if `None` would have a different meaning than the natural meaning of an empty -collection. For example: +collection. For example: - An include list where the default is to include everything should be `Optional` with default `None`. @@ -397,7 +397,7 @@ The basic Python collections [`Dict`](https://docs.python.org/3/library/typing.html#typing.Dict), and [`Set`](https://docs.python.org/3/library/typing.html#typing.Set) are mutable, but it's confusing for a function to mutate a collection -that was passed to it as an argument, especially by accident. To +that was passed to it as an argument, especially by accident. To avoid this, prefer annotating function parameters with read-only types: @@ -422,14 +422,14 @@ In some cases the more general [`Collection`](https://docs.python.org/3/library/typing.html#typing.Collection) or [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable) -types might be appropriate. (But don’t use `Iterable` for a value +types might be appropriate. (But don’t use `Iterable` for a value that might be iterated multiple times, since a one-use iterator is `Iterable` too.) A function's return type can be mutable if the return value is always a freshly created collection, since the caller ends up with the only reference to the value and can freely mutate it without risk of -confusion. But a read-only return type might be more appropriate for +confusion. But a read-only return type might be more appropriate for a function that returns a reference to an existing collection. Read-only types have the additional advantage of being [covariant @@ -497,7 +497,7 @@ def f(s: str) -> str: But Mypy doesn't yet support the advanced type annotations that would be needed to correctly type generic signature-changing decorators, such as `zerver.decorator.authenticated_json_view`, which passes an -extra argument to the inner function. For these decorators we must +extra argument to the inner function. For these decorators we must unfortunately give up some type safety by falling back to `Callable[..., T]`. @@ -505,7 +505,7 @@ unfortunately give up some type safety by falling back to All of our linters, including mypy, are designed to only check files that have been added in git (this is by design, since it means you -have untracked files in your Zulip checkout safely). So if you get a +have untracked files in your Zulip checkout safely). So if you get a `mypy` error like this after adding a new file that is referenced by the existing codebase: diff --git a/docs/testing/philosophy.md b/docs/testing/philosophy.md index c6a1b37a02..064e8bda64 100644 --- a/docs/testing/philosophy.md +++ b/docs/testing/philosophy.md @@ -1,13 +1,13 @@ # Testing philosophy Zulip's automated tests are a huge part of what makes the project able -to make progress. This page records some of the key principles behind +to make progress. This page records some of the key principles behind how we have designed our automated test suites. ## Effective testing allows us to move quickly Zulip's engineering strategy can be summarized as "move quickly -without breaking things". Despite reviewing many code submissions +without breaking things". Despite reviewing many code submissions from new contributors without deep expertise in the code they are changing, Zulip's maintainers spend most of the time they spend integrating changes on product decisions and code @@ -17,9 +17,9 @@ lower-level issues. This is possible because we have spent years systematically investing in testing, tooling, code structure, documentation, and development practices to help ensure that our contributors write code that needs -relatively few changes before it can be merged. The testing element +relatively few changes before it can be merged. The testing element of this is to have reliable, extensive, easily extended test suites -that cover most classes of bugs. Our testing systems have been +that cover most classes of bugs. Our testing systems have been designed to minimize the time spent manually testing or otherwise investigating whether changes are correct. @@ -34,7 +34,7 @@ While not every part of Zulip has a great test suite, many components do, and for those components, the tests mean that new contributors can often make substantive changes and have them be more or less correct by the time they share the -changes for code review. More importantly, it means that maintainers +changes for code review. More importantly, it means that maintainers save most of the time that would otherwise be spent verifying that the changes are simply correct, and instead focus on making sure that the codebase remains readable, well-structured, and well-tested. @@ -43,12 +43,12 @@ codebase remains readable, well-structured, and well-tested. When automated test suites are slow or unreliable, developers will avoid running them, and furthermore, avoid working on improving them -(both the system and individual tests). Because changes that make +(both the system and individual tests). Because changes that make tests slow or unreliable are often unintentional side effects of other development, problems in this area tend to accumulate as a codebase -grows. As a result, barring focused effort to prevent this outcome, +grows. As a result, barring focused effort to prevent this outcome, any large software project will eventually have its test suite rot -into one that is slow, unreliable, untrustworthy, and hated. We aim +into one that is slow, unreliable, untrustworthy, and hated. We aim for Zulip to avoid that fate. So we consider it essential to maintaining every automated test suite @@ -62,7 +62,7 @@ suite (`test-js-with-node`) to run in under 10 seconds. It'd be a long blog post to summarize everything we do to help achieve these goals, but a few techniques are worth highlighting: - Our test suites are designed to not access the Internet, since the - Internet might be down or unreliable in the test environment. Where + Internet might be down or unreliable in the test environment. Where outgoing HTTP requests are required to test something, we mock the responses with libraries like `responses`. - We carefully avoid the potential for contamination of data inside @@ -70,7 +70,7 @@ these goals, but a few techniques are worth highlighting: - Every test case prepends a unique random prefix to all keys it uses when accessing Redis and memcached. - Every test case runs inside a database transaction, which is - aborted after the test completes. Each test process interacts + aborted after the test completes. Each test process interacts only with a fresh copy of a special template database used for server tests that is destroyed after the process completes. - We rigorously investigate non-deterministically failing tests as though @@ -79,9 +79,9 @@ these goals, but a few techniques are worth highlighting: ## Integration testing or unit testing? Developers frequently ask whether they should write "integration -tests" or "unit tests". Our view is that tests should be written +tests" or "unit tests". Our view is that tests should be written against interfaces that you're already counting on keeping stable, or -already promising people you'll keep stable. In other words, +already promising people you'll keep stable. In other words, interfaces that you or other people are already counting on mostly not changing except in compatible ways. @@ -104,26 +104,26 @@ tiny internal functions, then any time you refactor something and change the internal interfaces -- even though you just made them up, and they're completely internal to that codebase so there's nothing that will break if you change them at will -- you have to go deal with -editing a bunch of tests to match the new interfaces. That's +editing a bunch of tests to match the new interfaces. That's especially a lot of work if you try to take the tests seriously, because you have to think through whether the tests breaking are telling you something you should actually listen to. In some big codebases, this can lead to tests feeling a lot like busywork... and it's because a lot of those tests really are -busywork. And that leads to developers not being committed to +busywork. And that leads to developers not being committed to maintaining and expanding the test suite in a thoughtful way. But if your tests are written against an external API, and you make some refactoring change and a bunch of tests break... now that's -telling you something very real! You can always edit the tests... but +telling you something very real! You can always edit the tests... but the tests are stand-ins for real users and real code out there beyond your reach, which will break the same way. So you can still make the change... but you have to deal with figuring out an appropriate migration or backwards-compatibility strategy for all those real users out there. Updating the tests is one of the easy -parts. And those changes to the tests are a nice reminder to code +parts. And those changes to the tests are a nice reminder to code reviewers that you've changed an interface, and the reviewer should think carefully about whether those interface changes will be a problem for any existing clients and whether they're properly reflected @@ -167,7 +167,7 @@ So, to summarize our approach to integration vs. unit testing: ## Avoid duplicating code with security impact Developing secure software with few security bugs is extremely -difficult. An important part of our strategy for avoiding security +difficult. An important part of our strategy for avoiding security logic bugs is to design patterns for how all of our code that processes untrusted user input can be well tested without either writing (and reviewing!) endless tests or requiring every developer to @@ -177,7 +177,7 @@ Our strategy for this is to write a small number of carefully-designed functions like `access_stream_by_id` that we test carefully, and then use linting and other coding conventions to require that all access to data from code paths that might share that data with users be mediated -through those functions. So rather than having each view function do +through those functions. So rather than having each view function do it own security checks for whether the user can access a given stream, and needing to test each of those copies of the logic, we only need to do that work once for each major type of data structure and level of @@ -211,7 +211,7 @@ test conditions. The benefit of this strategy is that you guarantee that the test setup only differs as intended: Done well, it helps avoid the otherwise extremely common failure mode where a `test_foo_failure` test passes -for the wrong reason. (E.g. the action fails not because of the +for the wrong reason. (E.g. the action fails not because of the permission check, but because a required HTTP parameter was only added to an adjacent `test_foo_success`). @@ -228,7 +228,7 @@ designed to be robust to future refactoring. But for some things, like documentation and CSS, the only way to test is to view the element in a browser and try things that might not -work. What to test will vary with what is likely to break. For +work. What to test will vary with what is likely to break. For example, after a significant change to Zulip's Markdown documentation, if you haven't verified every special bit of formatting visually and clicked every new link, there's a good chance that you've introduced a diff --git a/docs/testing/testing-with-django.md b/docs/testing/testing-with-django.md index 265dfdf84d..25644fd831 100644 --- a/docs/testing/testing-with-django.md +++ b/docs/testing/testing-with-django.md @@ -2,21 +2,21 @@ ## Overview -Zulip uses the Django framework for its Python backend. We +Zulip uses the Django framework for its Python backend. We use the testing framework from [django.test](https://docs.djangoproject.com/en/1.10/topics/testing/) -to test our code. We have over a thousand automated tests that verify that +to test our code. We have over a thousand automated tests that verify that our backend works as expected. -All changes to the Zulip backend code should be supported by tests. We +All changes to the Zulip backend code should be supported by tests. We enforce our testing culture during code review, and we also use -coverage tools to measure how well we test our code. We mostly use +coverage tools to measure how well we test our code. We mostly use tests to prevent regressions in our code, but the tests can have ancillary benefits such as documenting interfaces and influencing the design of our software. If you have worked on other Django projects that use unit testing, you -will probably find familiar patterns in Zulip's code. This document +will probably find familiar patterns in Zulip's code. This document describes how to write tests for the Zulip backend, with a particular emphasis on areas where we have either wrapped Django's test framework or just done things that are kind of unique in Zulip. @@ -26,7 +26,7 @@ or just done things that are kind of unique in Zulip. Our tests live in `zerver/tests/`. You can run them with `./tools/test-backend`. The tests run in parallel using multiple threads in your development environment, and can finish in under 30s -on a fast machine. When you are in iterative mode, you can run +on a fast machine. When you are in iterative mode, you can run individual tests or individual modules, following the dotted.test.name convention below: @@ -36,7 +36,7 @@ cd /srv/zulip ``` There are many command line options for running Zulip tests, such -as a `--verbose` option. The +as a `--verbose` option. The best way to learn the options is to use the online help: ```bash @@ -44,23 +44,23 @@ best way to learn the options is to use the online help: ``` We also have ways to instrument our tests for finding code coverage, -URL coverage, and slow tests. Use the `-h` option to discover these -features. We also have a `--profile` option to facilitate profiling +URL coverage, and slow tests. Use the `-h` option to discover these +features. We also have a `--profile` option to facilitate profiling tests. Another thing to note is that our tests generally "fail fast," i.e. they -stop at the first sign of trouble. This is generally a good thing for +stop at the first sign of trouble. This is generally a good thing for iterative development, but you can override this behavior with the -`--nonfatal-errors` option. A useful option to combine with that is +`--nonfatal-errors` option. A useful option to combine with that is the `--rerun` option, which will rerun just the tests that failed in the last test run. -**Webhook integrations**. For performance, `test-backend` with no +**Webhook integrations**. For performance, `test-backend` with no arguments will not run webhook integration tests (`zerver/webhooks/`), which would otherwise account for about 25% of the total runtime. When working on webhooks, we recommend instead running `test-backend zerver/webhooks` manually (or better, the direction for -the specific webhooks you're working on). And of course our CI is +the specific webhooks you're working on). And of course our CI is configured to always use `test-backend --include-webhooks` and run all of the tests. @@ -71,10 +71,10 @@ the rest of this document, and you can also read some of the existing tests in `zerver/tests` to get a feel for the patterns we use. A good practice is to get a "failing test" before you start to implement -your feature. First, it is a useful exercise to understand what needs to happen +your feature. First, it is a useful exercise to understand what needs to happen in your tests before you write the code, as it can help drive out simple -design or help you make incremental progress on a large feature. Second, -you want to avoid introducing tests that give false positives. Ensuring +design or help you make incremental progress on a large feature. Second, +you want to avoid introducing tests that give false positives. Ensuring that a test fails before you implement the feature ensures that if somebody accidentally regresses the feature in the future, the test will catch the regression. @@ -87,7 +87,7 @@ which contains our `ZulipTestCase` and `WebhookTestCase` classes. ### Setting up data for tests -All tests start with the same fixture data. (The tests themselves +All tests start with the same fixture data. (The tests themselves update the database, but they do so inside a transaction that gets rolled back after each of the tests complete. For more details on how the fixture data gets set up, refer to `tools/setup/generate-fixtures`.) @@ -118,7 +118,7 @@ Here are some example action methods that tests may use for data setup: ### Testing code that accesses the filesystem Some tests need to access the filesystem (e.g. `test_upload.py` tests -for `LocalUploadBackend` and the data import tests). Doing +for `LocalUploadBackend` and the data import tests). Doing this correctly requires care to avoid problems like: - Leaking files after every test (which are clutter and can eventually run the development environment out of disk) or @@ -138,7 +138,7 @@ To avoid these problems, you can do the following: Our common testing infrastructure handles some of this for you, e.g. it replaces `settings.LOCAL_UPLOADS_DIR` for each test process -with a unique path under `/var//test-backend`. And +with a unique path under `/var//test-backend`. And `UploadSerializeMixin` manages some of the cleanup work for `test_upload.py`. @@ -164,7 +164,7 @@ Additionally, you can observe any calls made to your mocked object. When writing tests, it often occurs that you make calls to functions taking complex arguments. Creating a real instance of such an argument would require the use of various different libraries, a lot of -boilerplate code, etc. Another scenario is that the tested code +boilerplate code, etc. Another scenario is that the tested code accesses files or objects that don't exist at testing time. Finally, it is good practice to keep tests independent from others. Mocks help you to isolate test cases by simulating objects and methods irrelevant @@ -314,8 +314,8 @@ On the other hand, if we had used `import os.urandom`, we would need to call `mo #### Zulip mocking practices For mocking we generally use the "mock" library and use `mock.patch` as -a context manager or decorator. We also take advantage of some context managers -from Django as well as our own custom helpers. Here is an example: +a context manager or decorator. We also take advantage of some context managers +from Django as well as our own custom helpers. Here is an example: ```python with self.settings(RATE_LIMITING=True): @@ -335,7 +335,7 @@ find several examples of doing this. ## Zulip testing philosophy If there is one word to describe Zulip's philosophy for writing tests, -it is probably "flexible." (Hopefully "thorough" goes without saying.) +it is probably "flexible." (Hopefully "thorough" goes without saying.) When in doubt, unless speed concerns are prohibitive, you usually want your tests to be somewhat end-to-end, particularly @@ -346,8 +346,8 @@ test suite... ### Endpoint tests -We strive to test all of our URL endpoints. The vast majority of Zulip -endpoints support a JSON interface. Regardless of the interface, an +We strive to test all of our URL endpoints. The vast majority of Zulip +endpoints support a JSON interface. Regardless of the interface, an endpoint test generally follows this pattern: - Set up the data. @@ -384,7 +384,7 @@ via Django. ### Fixture-driven tests Particularly for testing Zulip's integrations with third party systems, -we strive to have a highly data-driven approach to testing. To give a +we strive to have a highly data-driven approach to testing. To give a specific example, when we test our GitHub integration, the test code reads a bunch of sample inputs from a JSON fixture file, feeds them to our GitHub integration code, and then verifies the output against @@ -410,20 +410,20 @@ A detailed description of mocks, along with useful coded snippets, can be found In [zerver/tests/test_templates.py](https://github.com/zulip/zulip/blob/main/zerver/tests/test_templates.py) we have a test that renders all of our backend templates with a "dummy" context, to make sure the templates don't have obvious -errors. (These tests won't catch all types of errors; they are +errors. (These tests won't catch all types of errors; they are just a first line of defense.) ### SQL performance tests A common class of bug with Django systems is to handle bulk data in an inefficient way, where the backend populates objects for join tables -with a series of individual queries that give O(N) latency. (The +with a series of individual queries that give O(N) latency. (The remedy is often just to call `select_related()`, but sometimes it requires a more subtle restructuring of the code.) We try to prevent these bugs in our tests by using a context manager called `queries_captured()` that captures the SQL queries used by -the backend during a particular operation. We make assertions about +the backend during a particular operation. We make assertions about those queries, often simply asserting that the number of queries is below some threshold. @@ -432,21 +432,21 @@ below some threshold. The Zulip backend has a mechanism where it will fetch initial data for a client from the database, and then it will subsequently apply some queued up events to that data to the data structure before notifying -the client. The `BaseAction.do_test()` helper helps tests +the client. The `BaseAction.do_test()` helper helps tests verify that the application of those events via apply_events() produces the same data structure as performing an action that generates said event. This is a bit esoteric, but if you read the tests, you will see some of -the patterns. You can also learn more about our event system in the +the patterns. You can also learn more about our event system in the [new feature tutorial](../tutorials/new-feature-tutorial.html#handle-database-interactions). ### Negative tests It is important to verify error handling paths for endpoints, particularly situations where we need to ensure that we don't return results to clients -with improper authentication or with limited authorization. A typical test +with improper authentication or with limited authorization. A typical test will call the endpoint with either a non-logged in client, an invalid API -key, or missing input fields. Then the test will call `assert_json_error()` +key, or missing input fields. Then the test will call `assert_json_error()` to verify that the endpoint is properly failing. ## Testing considerations @@ -458,26 +458,26 @@ If you have several tests repeating the same type of test setup, consider making a setUp() method or a test helper. - **Network independence** Our tests should still work if you don't -have an internet connection. For third party clients, you can simulate -their behavior using fixture data. For third party servers, you can +have an internet connection. For third party clients, you can simulate +their behavior using fixture data. For third party servers, you can typically simulate their behavior using mocks. - **Coverage** We have 100% line coverage on several of our backend -modules. You can use the `--coverage` option to generate coverage +modules. You can use the `--coverage` option to generate coverage reports, and new code should have 100% coverage, which generally requires testing not only the "happy path" but also error handling -code and edge cases. It will generate a nice HTML report that you can +code and edge cases. It will generate a nice HTML report that you can view right from your browser (the tool prints the URL where the report is exposed in your development environment). - **Console output** A properly written test should print nothing to the console; use `with self.assertLogs` to capture and verify any -logging output. Note that we reconfigure various loggers in +logging output. Note that we reconfigure various loggers in `zproject/test_extra_settings.py` where the output is unlikely to be interesting when running our test suite. `test-backend --ban-console-output` checks for stray print statements. Note that `test-backend --coverage` will assert that various specific files in the project have 100% test coverage and -throw an error if their coverage has fallen. One of our project goals +throw an error if their coverage has fallen. One of our project goals is to expand that checking to ever-larger parts of the codebase. diff --git a/docs/testing/testing-with-node.md b/docs/testing/testing-with-node.md index 4f9558ce69..da8097a080 100644 --- a/docs/testing/testing-with-node.md +++ b/docs/testing/testing-with-node.md @@ -1,7 +1,7 @@ # JavaScript/TypeScript unit tests Our node-based unit tests system is the preferred way to test -JavaScript/TypeScript code in Zulip. We prefer it over the [Puppeteer +JavaScript/TypeScript code in Zulip. We prefer it over the [Puppeteer black-box whole-app testing](../testing/testing-with-puppeteer.md), system since it is much (>100x) faster and also easier to do correctly than the Puppeteer system. @@ -15,8 +15,8 @@ See `test-js-with-node --help` for useful options; even though the whole suite is quite fast, it still saves time to run a single test by name when debugging something. -The JS unit tests are written to work with node. You can find them -in `frontend_tests/node_tests`. Here is an example test from +The JS unit tests are written to work with node. You can find them +in `frontend_tests/node_tests`. Here is an example test from `frontend_tests/node_tests/stream_data.js`: ```js @@ -38,8 +38,8 @@ in `frontend_tests/node_tests`. Here is an example test from ``` The names of the node tests generally align with the names of the -modules they test. If you modify a JS module in `static/js` you should -see if there are corresponding test in `frontend_tests/node_tests`. If +modules they test. If you modify a JS module in `static/js` you should +see if there are corresponding test in `frontend_tests/node_tests`. If there are, you should strive to follow the patterns of the existing tests and add your own tests. @@ -69,7 +69,7 @@ working on or debugging the Zulip node tests. Conceptually, the `zjquery` library provides minimal versions of most `jQuery` DOM manipulation functions, and has a convenient system for -letting you set up return values for more complex functions. For +letting you set up return values for more complex functions. For example, if the code you'd like to test calls `$obj.find()`, you can use `$obj.set_find_results(selector, $value)` to set up `zjquery` so that calls to `$obj.find(selector)` will return `$value`. See the unit @@ -97,7 +97,7 @@ based on how other functions have been stubbed in the same file. The other big challenge with doing unit tests for a JavaScript project is that often one wants to limit the scope the production code being run, just to avoid doing extra setup work that isn't relevant to the -code you're trying to test. For that reason, each unit test file +code you're trying to test. For that reason, each unit test file explicitly declares all of the modules it depends on, with a few different types of declarations depending on whether we want to: @@ -178,8 +178,8 @@ data/logic modules (UI modules are lower priority for unit testing). Our node test system is pretty simple, and it's possible to configure the native debugger features of popular editors to allow stepping -through the code. Below we document the editors where someone has put -together detailed instructions for how to do so. Contributions of +through the code. Below we document the editors where someone has put +together detailed instructions for how to do so. Contributions of notes for other editors are welcome! ## Webstorm integration setup @@ -201,7 +201,7 @@ These instructions assume you're using the Vagrant development environment. - `Vagrant executable` should already be correctly `vagrant`. - `Environment Variables` is not needed. -3. You'll now need to set up a WebStorm "Debug Configuration". Open +3. You'll now need to set up a WebStorm "Debug Configuration". Open the `Run/Debug Configuration` menu and create a new `Node.js` config: 1. Under `Node interpreter:` click the 3 dots to the right side and click on the little plus in the bottom left of the @@ -217,7 +217,7 @@ These instructions assume you're using the Vagrant development environment. 1. Under `JavaScript file`, enter `frontend_tests/zjsunit/index.js` -- this is the root script for Zulip's node unit tests. -Congratulations! You've now set up the integration. +Congratulations! You've now set up the integration. ## Running tests with the debugger diff --git a/docs/testing/testing-with-puppeteer.md b/docs/testing/testing-with-puppeteer.md index 2f6628b758..97d0af5cb7 100644 --- a/docs/testing/testing-with-puppeteer.md +++ b/docs/testing/testing-with-puppeteer.md @@ -26,7 +26,7 @@ of various useful helper functions defined in The Puppeteer tests use a real Chromium browser (powered by [puppeteer](https://github.com/puppeteer/puppeteer)), connected to a -real Zulip development server. These are black-box tests: Steps in a +real Zulip development server. These are black-box tests: Steps in a Puppeteer test are largely things one might do as a user of the Zulip webapp, like "Type this key", "Wait until this HTML element appears/disappears", or "Click on this HTML element". @@ -45,7 +45,7 @@ async function test_private_message_compose_shortcut(page) { The test function presses the `x` key, waits for the `#private_message_recipient` input element to appear, verifies its -content is empty, and then closes the compose box. The +content is empty, and then closes the compose box. The `waitForSelector` step here (and in most tests) is critical; tests that don't wait properly often fail nonderministically, because the test will work or not depending on whether the browser updates the UI @@ -70,14 +70,14 @@ integration](../testing/continuous-integration.md): interactive to debug an issue in the normal Zulip development environment than in the Puppeteer test suite. - Does the change being tested adjust the HTML structure in a way that - affects any of the selectors used in the tests? If so, the test may + affects any of the selectors used in the tests? If so, the test may just need to be updated for your changes. - Does the test fail deterministically when you run it locally using E.g. `./tools/test-js-with-puppeteer compose.ts`? If so, you can iteratively debug to see the failure. -- Does the test fail nondeterministically? If so, the problem is +- Does the test fail nondeterministically? If so, the problem is likely that a `waitForSelector` statement is either missing or not - waiting for the right thing. Tests fail nondeterministically much + waiting for the right thing. Tests fail nondeterministically much more often on very slow systems like those used for Continuous Integration (CI) services because small races are amplified in those environments; this often explains failures in CI that cannot be @@ -115,7 +115,7 @@ These tools/features are often useful when debugging: extra [Django settings](../subsystems/settings.md) from `zproject/test_extra_settings.py` to configure an isolated database so that the tests will not interfere/interact with a normal - development environment. The console output while running the tests + development environment. The console output while running the tests includes the console output for the server; any Python exceptions are likely actual bugs in the changes being tested. @@ -143,22 +143,22 @@ notes above: assumes the last step was processed by the browser (E.g. after you click on a user's avatar, you need an explicit wait for the profile popover to appear before you can try to click on a menu item in that - popover). This means that before essentially every action in your + popover). This means that before essentially every action in your Puppeteer tests, you'll want to use `waitForSelector` or similar wait function to make sure the page or element is ready before you - interact with it. The [puppeteer docs site](https://pptr.dev/) is a + interact with it. The [puppeteer docs site](https://pptr.dev/) is a useful reference for the available wait functions. - When using `waitForSelector`, you always want to use the `{visible: true}` option; otherwise the test will stop waiting as soon as the target selector is present in the DOM even if it's - hidden. For the common UI pattern of having an element always be + hidden. For the common UI pattern of having an element always be present in the DOM whose presence is managed via show/hide rather than adding/removing it from the DOM, `waitForSelector` without `visible: true` won't wait at all. - The test suite uses a smaller set of default user accounts and other data initialized in the database than the normal development environment; specifically, it uses the same setup as the [backend - tests](../testing/testing-with-django.md). To see what differs from + tests](../testing/testing-with-django.md). To see what differs from the development environment, check out the conditions on `options["test_suite"]` in `zilencer/management/commands/populate_db.py`. diff --git a/docs/testing/testing.md b/docs/testing/testing.md index d7fb55117a..248ec16336 100644 --- a/docs/testing/testing.md +++ b/docs/testing/testing.md @@ -28,7 +28,7 @@ as follows: ``` However, you will rarely want to do this while actively developing, -because it takes a long time. Instead, your edit/refresh cycle will +because it takes a long time. Instead, your edit/refresh cycle will typically involve running subsets of the tests with commands like these: ```bash @@ -39,7 +39,7 @@ typically involve running subsets of the tests with commands like these: ./tools/test-js-with-node utils.js ``` -The commands above will all run in just a few seconds. Many more +The commands above will all run in just a few seconds. Many more useful options are discussed in each tool's documentation (e.g. `./tools/test-backend --help`). @@ -60,7 +60,7 @@ eventually work with, each with its own page detailing how it works: Additionally, Zulip also has about a dozen smaller tests suites: - `tools/test-migrations`: Checks whether the `zerver/migrations` - migration content the models defined in `zerver/models.py`. See our + migration content the models defined in `zerver/models.py`. See our [schema migration documentation](../subsystems/schema-migrations.md) for details on how to do database migrations correctly. - `tools/test-documentation`: Checks for broken links in this @@ -72,12 +72,12 @@ Additionally, Zulip also has about a dozen smaller tests suites: `zerver/openapi/python_examples.py`. - `test-locked-requirements`: Verifies that developers didn't forget to run `tools/update-locked-requirements` after modifying - `requirements/*.in`. See + `requirements/*.in`. See [our dependency documentation](../subsystems/dependencies.md) for details on the system this is verifying. - `tools/check-capitalization`: Checks whether translated strings (aka user-facing strings) correctly follow Zulip's capitalization - conventions. This requires some maintenance of an exclude list + conventions. This requires some maintenance of an exclude list (`tools.lib.capitalization.IGNORED_PHRASES`) of proper nouns mentioned in the Zulip project, but helps a lot in avoiding new strings being added that don't match our style. @@ -104,7 +104,7 @@ something valuable to helping keep Zulip bug-free. ## Internet access inside test suites As a policy matter, the Zulip test suites should never make outgoing -HTTP or other network requests. This is important for 2 major +HTTP or other network requests. This is important for 2 major reasons: - Tests that make outgoing Internet requests will fail when the user @@ -116,7 +116,7 @@ reasons: developer time, and we try to avoid them wherever possible. As a result, Zulip's major test suites should never access the -Internet directly. Since code in Zulip does need to access the +Internet directly. Since code in Zulip does need to access the Internet (e.g. to access various third-party APIs), this means that the Zulip tests use mocking to basically hardcode (for the purposes of the test) what responses should be used for any outgoing Internet @@ -125,7 +125,7 @@ requests that Zulip would make in the code path being tested. This is easy to do using test fixtures (a fancy word for fixed data used in tests) and the `mock.patch` function to specify what HTTP response should be used by the tests for every outgoing HTTP (or other -network) request. Consult +network) request. Consult [our guide on mocking](../testing/testing-with-django.html#zulip-mocking-practices) to learn how to mock network requests easily; there are also a number of examples throughout the codebase. @@ -133,7 +133,7 @@ examples throughout the codebase. We partially enforce this policy in the main Django/backend test suite by overriding certain library functions that are used in outgoing HTTP code paths (`httplib2.Http().request`, `requests.request`, etc.) to -throw an exception in the backend tests. While this is enforcement is +throw an exception in the backend tests. While this is enforcement is not complete (there a lot of other ways to use the Internet from Python), it is easy to do and catches most common cases of new code depending on Internet access. @@ -151,9 +151,9 @@ This enforcement code results in the following exception: The one exception to this policy is our documentation tests, which will attempt to verify that the links included in our documentation -aren't broken. Those tests end up failing nondeterministically fairly +aren't broken. Those tests end up failing nondeterministically fairly often, which is unfortunate, but there's simply no other correct way -to verify links other than attempting to access them. The compromise +to verify links other than attempting to access them. The compromise we've implemented is that in CI, these tests only verify links to websites controlled by the Zulip project (zulip.com, our GitHub, our ReadTheDocs), and not links to third-party websites. diff --git a/docs/testing/typescript.md b/docs/testing/typescript.md index dda223e067..1a18190bcb 100644 --- a/docs/testing/typescript.md +++ b/docs/testing/typescript.md @@ -2,7 +2,7 @@ Zulip is early in the process of migrating our codebase to use [TypeScript](https://www.typescriptlang.org/), the leading static type -system for JavaScript. It works as an extension of the ES6 JavaScript +system for JavaScript. It works as an extension of the ES6 JavaScript standard, and provides similar benefits to our use of [the mypy static type system for Python](../testing/mypy.md). @@ -33,7 +33,7 @@ The following resources are valuable for learning TypeScript: ## Type checking TypeScript types are checked by the TypeScript compiler, `tsc`, which -is run as part of our [lint checks](linters.md). You can run the +is run as part of our [lint checks](linters.md). You can run the compiler yourself with `tools/run-tsc`, which will check all the TypeScript files once, or `tools/run-tsc --watch`, which will continually recheck the files as you edit them. @@ -41,13 +41,13 @@ continually recheck the files as you edit them. ## Linting and style We use the Eslint plugin for TypeScript to lint TypeScript code, just -like we do for JavaScript. Our long-term goal is to use an idiomatic +like we do for JavaScript. Our long-term goal is to use an idiomatic TypeScript style for our TypeScript codebase. However, because we are migrating an established JavaScript codebase, we plan to start with a style that is closer to the existing JavaScript code, so that we can easily migrate individual modules -without too much code churn. A few examples: +without too much code churn. A few examples: - TypeScript generally prefers explicit `return undefined;`, whereas our existing JavasScript style uses just `return;`. @@ -72,22 +72,22 @@ Our plan is to order which modules we migrate carefully, starting with those that: - Appear frequently as reverse dependencies of other modules - (e.g. `people.js`). These are most valuable to do first because + (e.g. `people.js`). These are most valuable to do first because then we have types on the data being interacted with by other modules when we migrate those. - Don't have large open pull requests (to avoid merge conflicts); one can scan for these using [TinglingGit](https://github.com/zulip/TinglingGit). - Have good unit test coverage, which limits the risk of breaking - correctness through refactoring. Use + correctness through refactoring. Use `tools/test-js-with-node --coverage` to get a coverage report. When migrating a module, we want to be especially thoughtful about putting together a commit structure that makes mistakes unlikely and -the changes easy to verify. E.g.: +the changes easy to verify. E.g.: - First a commit that just converts the language to TypeScript adding - types. The result may potentially have some violations of the - long-term style we want (e.g. not using `const`). Depending on how + types. The result may potentially have some violations of the + long-term style we want (e.g. not using `const`). Depending on how we're handling linting, we set some override eslint rules at the top of the module at this stage so CI still passes. - Then a commit just migrating use of `var` to `const/let` without diff --git a/docs/translating/chinese.md b/docs/translating/chinese.md index 15ccdc8259..3b290a819f 100644 --- a/docs/translating/chinese.md +++ b/docs/translating/chinese.md @@ -117,7 +117,7 @@ readability considerations. - Mute/Unmute - **开启/关闭免打扰** "Mute" is mostly translated as "静音(Silent)", which is common in TV -set. Such a translation is not appropriate for Zulip. "开启/关闭免打 +set. Such a translation is not appropriate for Zulip. "开启/关闭免打 扰(Turn off/on Notification)" is a sense to sense translation, which is also borrowed from the WeChat. diff --git a/docs/translating/internationalization.md b/docs/translating/internationalization.md index 607a6fd45d..055d1b22e2 100644 --- a/docs/translating/internationalization.md +++ b/docs/translating/internationalization.md @@ -6,19 +6,19 @@ the Zulip UI in their preferred language. This article aims to teach Zulip contributors enough about internationalization and Zulip's tools for it so that they can make -correct decisions about how to tag strings for translation. A few +correct decisions about how to tag strings for translation. A few principles are important in how we think about internationalization: - Our goal is for **all end-user facing strings** in Zulip to be tagged for translation in both [HTML templates](#html-templates) and - code, and our linters attempt to enforce this. There are some + code, and our linters attempt to enforce this. There are some exceptions: we don't tag strings in Zulip's landing pages (e.g. /features) and other documentation (e.g. /help) for translation at this time (though we do aim for those pages to be usable with tools like Google Translate). - Translating all the strings in Zulip for a language and maintaining that translation is a lot of work, and that work scales with the - number of strings tagged for translation in Zulip. For this reason, + number of strings tagged for translation in Zulip. For this reason, we put significant effort into only tagging for translation content that will actually be displayed to users, and minimizing unnecessary user-facing strings in the product. @@ -29,7 +29,7 @@ principles are important in how we think about internationalization: policy](../translating/translating.html#capitalization) that we enforce using linters that check all strings tagged for translation in Zulip. -This article aims to provide a brief introduction. We recommend the +This article aims to provide a brief introduction. We recommend the [EdX i18n guide][edx-i18n] as a great resource for learning more about internationalization in general; we agree with essentially all of their style guidelines. @@ -42,23 +42,23 @@ There are a few critical details about human language that are important to understand when implementing an internationalized application: - **Punctuation** varies between languages (e.g. Japanese doesn't use - `.`s at the end of sentences). This means that you should always + `.`s at the end of sentences). This means that you should always include end-of-sentence symbols like `.` and `?` inside the to-be-translated strings, so that translators can correctly translate the content. - **Word order** varies between languages (e.g. some languages put - subjects before verbs, others the other way around). This means + subjects before verbs, others the other way around). This means that **concatenating translateable strings** produces broken results (more details with examples are below). - The **width of the string needed to express something** varies dramatically between languages; this means you can't just hardcode a button or widget to look great for English and expect it to work in - all languages. German is a good test case, as it has a lot of long + all languages. German is a good test case, as it has a lot of long words, as is Japanese (as character-based languages use a lot less width). - This is more about how i18n tooling works, but in code, the translation function must be passed the string to translate, not a - variable containing the target string. Otherwise, the parsers that + variable containing the target string. Otherwise, the parsers that extract the strings in a project to send to translators will not find your string. @@ -96,7 +96,7 @@ The end-to-end tooling process for translations in Zulip is as follows. `./tools/i18n/push-translations` command (which invokes a Transifex API tool, `tx push`, internally). -4. Translators translate the strings in the Transifex UI. (In theory, +4. Translators translate the strings in the Transifex UI. (In theory, it's possible to translate locally and then do `tx push`, but because our workflow is to sync translation data from Transifex to Zulip, making changes to translations in Zulip risks having the @@ -145,8 +145,8 @@ can use the `_()` function in the templates like this: If a piece of text contains both a literal string component and variables, you can use a block translation, which makes use of placeholders to -help translators to translate an entire sentence. To translate a -block, Jinja2 uses the [trans][] tag. So rather than writing +help translators to translate an entire sentence. To translate a +block, Jinja2 uses the [trans][] tag. So rather than writing something ugly and confusing for translators like this: ```jinja @@ -169,7 +169,7 @@ which can be imported as follows: from django.utils.translation import gettext as _ ``` -Zulip expects all the error messages to be translatable as well. To +Zulip expects all the error messages to be translatable as well. To ensure this, the error message passed to `json_error` and `JsonableError` should always be a literal string enclosed by `_()` function, e.g.: @@ -240,8 +240,8 @@ $("#foo").html( The only HTML tags allowed directly in translated strings are the simple HTML tags enumerated in `default_html_elements` -(`static/js/i18n.js`) with no attributes. This helps to avoid -exposing HTML details to translators. If you need to include more +(`static/js/i18n.js`) with no attributes. This helps to avoid +exposing HTML details to translators. If you need to include more complex markup such as a link, you can define a custom HTML tag locally to the translation: @@ -255,7 +255,7 @@ $t_html( ### Handlebars templates For translations in Handlebars templates we also use FormatJS, through two -Handlebars [helpers][] that Zulip registers. The syntax for simple strings is: +Handlebars [helpers][] that Zulip registers. The syntax for simple strings is: ```html+handlebars {{t 'English text' }} @@ -283,7 +283,7 @@ The syntax for block strings or strings containing variables is: ``` Just like in JavaScript code, variables are enclosed in *single* -braces (rather than the usual Handlebars double braces). Unlike in +braces (rather than the usual Handlebars double braces). Unlike in JavaScript code, variables are automatically escaped by our Handlebars helper. @@ -292,11 +292,11 @@ Handlebars expressions like `{{variable}}` or blocks like translated block, because they don't work properly with translation. The Handlebars expression would be evaluated before the string is processed by FormatJS, so that the string to be translated wouldn't be -constant. We have a linter to enforce that translated blocks don't +constant. We have a linter to enforce that translated blocks don't contain handlebars. Restrictions on including HTML tags in translated strings are the same -as in JavaScript. You can insert more complex markup using a local +as in JavaScript. You can insert more complex markup using a local custom HTML tag like this: ```html+handlebars diff --git a/docs/translating/russian.md b/docs/translating/russian.md index 287d4ffb7f..de9158b6fb 100644 --- a/docs/translating/russian.md +++ b/docs/translating/russian.md @@ -48,7 +48,7 @@ - invite-only - **закрытый** - public - **открытый** - name - **название** for things, **имя** for people -- id - **код** +- id - **код** - notifications - **оповещения** - @-mentions - **@-упоминания** - mute - **заглушить** diff --git a/docs/translating/translating.md b/docs/translating/translating.md index 5854dc2eb0..1f4d079ab0 100644 --- a/docs/translating/translating.md +++ b/docs/translating/translating.md @@ -6,7 +6,7 @@ Zulip. Additionally, the Zulip UI is translated into more than a dozen major languages, including Spanish, German, Hindi, French, Chinese, Russian, -and Japanese, and we're always excited to add more. If you speak a +and Japanese, and we're always excited to add more. If you speak a language other than English, your help with translating Zulip is be greatly appreciated! @@ -42,7 +42,7 @@ to any languages that you'd like to contribute to (or add new ones). usually takes less than a day. You should then be able to access Zulip's dashboard in Transifex. -1. Translate the strings for your language in Transifex. Zulip has +1. Translate the strings for your language in Transifex. Zulip has several resource files: - `mobile.json` is for the iOS/Android mobile apps. - `desktop.json` is for the parts of the Zulip desktop apps that @@ -55,7 +55,7 @@ to any languages that you'd like to contribute to (or add new ones). current [stable release series](../overview/release-lifecycle.md). Transifex is smart about only asking you to translate a string once - even if it appears in multiple resources. The `4-x--` type variants + even if it appears in multiple resources. The `4-x--` type variants allow translators to get a language to 100% translated for the current release. @@ -68,7 +68,7 @@ to any languages that you'd like to contribute to (or add new ones). Some useful tips for your translating journey: - Follow your language's [translation guide](#translation-style-guides). - Keeping it open in a tab while translating is very handy. If one + Keeping it open in a tab while translating is very handy. If one doesn't exist one, write one as you go; they're easiest to write as you go along and will help any future translators a lot. @@ -112,21 +112,21 @@ can usually just deploy the latest translations there. `tools/i18n/sync-translations` command (it will require some [initial setup](../translating/internationalization.html#transifex-cli-setup)). This command will download the resource files from Transifex and replace -your local resource files with them, and then compile them. You can +your local resource files with them, and then compile them. You can now test your translation work in the Zulip UI. There are a few ways to see your translations in the Zulip UI: -- You can insert the language code as a URL prefix. For example, you +- You can insert the language code as a URL prefix. For example, you can view the login page in German using - `http://localhost:9991/de/login/`. This works for any part of the + `http://localhost:9991/de/login/`. This works for any part of the Zulip UI, including portico (logged-out) pages. - For Zulip's logged-in UI (i.e. the actual webapp), you can [pick the language](https://zulip.com/help/change-your-language) in the Zulip UI. - If your system has languages configured in your OS/browser, Zulip's portico (logged-out) pages will automatically use your configured - language. Note that we only tag for translation strings in pages + language. Note that we only tag for translation strings in pages that individual users need to use (e.g. `/login/`, `/register/`, etc.), not marketing pages like `/features/`. - In case you need to understand how the above interact, Zulip figures @@ -171,7 +171,7 @@ Some translated languages don't have these, but we highly encourage translators for new languages (or those updating a language) write a style guide as they work , since it's easy to take notes as you translate, and doing so greatly increases the ability of future -translators to update the translations in a consistent way. See [our +translators to update the translations in a consistent way. See [our docs on this documentation](../documentation/overview.md) for how to submit your changes. @@ -179,7 +179,7 @@ submit your changes. We expect that all the English translatable strings in Zulip are properly capitalized in a way consistent with how Zulip does -capitalization in general. This means that: +capitalization in general. This means that: - The first letter of a sentence or phrase should be capitalized. - Correct: "Manage streams" diff --git a/docs/tutorials/new-feature-tutorial.md b/docs/tutorials/new-feature-tutorial.md index 04cf4df77c..e906dd25be 100644 --- a/docs/tutorials/new-feature-tutorial.md +++ b/docs/tutorials/new-feature-tutorial.md @@ -22,7 +22,7 @@ will also be helpful to review when creating a new feature. Many aspects of the structure will be familiar to Django developers. Visit [Django's documentation](https://docs.djangoproject.com/en/2.2/#index-first-steps) for more information about how Django projects are typically -organized. And finally, the +organized. And finally, the [message sending](../subsystems/sending-messages.md) documentation on the additional complexity involved in sending messages. @@ -199,8 +199,8 @@ dictionary. ``` **The majority of realm settings can be included in -`property_types`.** However, there are some properties that need custom -logic and thus cannot use this framework. For example: +`property_types`.** However, there are some properties that need custom +logic and thus cannot use this framework. For example: - The realm `authentication_methods` attribute is a bitfield and needs additional code for validation and updating. @@ -255,7 +255,7 @@ Like typical apps, we will need our backend to update the database and send some response to the client that made the request. Beyond that, we need to orchestrate notifications about the setting change -to *other* clients (or other users, if you will). Clients +to *other* clients (or other users, if you will). Clients find out about settings through two closely related code paths. When a client first contacts the server, the server sends the client its initial state. Subsequently, clients subscribe to "events," which can @@ -481,7 +481,7 @@ write automated backend tests for your new feature. To test the new setting syncs correctly with the `property_types` framework, one usually just needs to add a line in each of `test_events.py` and `test_realm.py` with a list of values to switch -between in the test. In the case of a boolean field, no action is +between in the test. In the case of a boolean field, no action is required, because those tests will correctly assume that the only values to test are `True` and `False`. @@ -541,7 +541,7 @@ In frontend, we have split the `property_types` into three objects: - `org_settings`: This contains properties for the "organization settings" page. Settings belonging to this section generally decide what features should be available to a user like deleting a - message, message edit history etc. Our `mandatory_topics` feature + message, message edit history etc. Our `mandatory_topics` feature belongs in this section. - `org_permissions`: This contains properties for the "organization @@ -560,7 +560,7 @@ before implementing it.* Note that some settings, like `realm_msg_edit_limit_setting`, require special treatment, because they don't match the common -pattern. We can't extract the property name and compare the value of +pattern. We can't extract the property name and compare the value of such input elements with those in `page_params`, so we have to manually handle such situations in a couple key functions: @@ -570,7 +570,7 @@ manually handle such situations in a couple key functions: compare and set the values of corresponding DOM element. - `settings_org.update_dependent_subsettings`: This handles settings - whose value and state depend on other elements. For example, + whose value and state depend on other elements. For example, `realm_waiting_period_threshold` is only shown for with the right state of `realm_waiting_period_setting`. @@ -586,7 +586,7 @@ backend, so no UI updates are required.). However, if you had written a function to update the UI after a given setting has changed, your function should be referenced in the -`realm_settings` of `server_events_dispatch.js`. See for example +`realm_settings` of `server_events_dispatch.js`. See for example `settings_emoji.update_custom_emoji_ui`. ``` diff @@ -619,7 +619,7 @@ Here are few important cases you should consider when testing your changes: properly. - If your setting is dependent on another setting, carefully check - that both are properly synchronized. For example, the input element + that both are properly synchronized. For example, the input element for `realm_waiting_period_threshold` is shown only when we have selected the custom time limit option in the `realm_waiting_period_setting` dropdown. diff --git a/docs/tutorials/reading-list.md b/docs/tutorials/reading-list.md index 56fdf3a1d8..6ebf581d0a 100644 --- a/docs/tutorials/reading-list.md +++ b/docs/tutorials/reading-list.md @@ -76,7 +76,7 @@ Some titles have been shortened for organizational purposes. *Tutorial* - [clean-code-javascript Software engineering principles](https://github.com/ryanmcdermott/clean-code-javascript) -*Course* - [React native and redux course](https://www.udemy.com/course/the-complete-react-native-and-redux-course/) (*Not free!*) +*Course* - [React native and redux course](https://www.udemy.com/course/the-complete-react-native-and-redux-course/) (*Not free!*) *Slides* - [TypeScript vs. CoffeeScript vs. ES6](https://www.slideshare.net/NeilGreen1/type-script-vs-coffeescript-vs-es6) diff --git a/docs/tutorials/shell-tips.md b/docs/tutorials/shell-tips.md index fe69ff83a9..636a0b4f68 100644 --- a/docs/tutorials/shell-tips.md +++ b/docs/tutorials/shell-tips.md @@ -300,11 +300,11 @@ $ /usr/bin/env python3 ./my_program.py The purpose of `/usr/bin/env` in our shebangs is as a way to locate the `python3` program in your current environment, the same one the -shell would use if you ran `python3 my_program.py`. You may see +shell would use if you ran `python3 my_program.py`. You may see Python scripts outside of Zulip with a shebang like `#!/usr/bin/python3`; but because of the way Python virtualenvs work, this has the effect of running the script outside of any currently -activated virtualenv. We use `/usr/bin/env` to keep our scripts +activated virtualenv. We use `/usr/bin/env` to keep our scripts running inside the virtualenv where we've installed all our dependencies. diff --git a/docs/tutorials/writing-views.md b/docs/tutorials/writing-views.md index e60a4a35ff..a461cc85c0 100644 --- a/docs/tutorials/writing-views.md +++ b/docs/tutorials/writing-views.md @@ -84,7 +84,7 @@ valid session cookie) before providing the view for this route, or redirects the browser to a login page. This is used in the root path (`/`) of the website for the web client. If a request comes from a browser without a valid session cookie, they are redirected to a login -page. It is a small fork of Django's +page. It is a small fork of Django's [login_required][login-required-link], adding a few extra checks specific to Zulip. @@ -104,7 +104,7 @@ Templates for the main website are found in ## Writing API REST endpoints These are code-parseable views that take x-www-form-urlencoded or JSON -request bodies, and return JSON-string responses. Almost all Zulip +request bodies, and return JSON-string responses. Almost all Zulip view code is in the implementations of API REST endpoints. The REST API does authentication of the user through `rest_dispatch`, @@ -124,9 +124,9 @@ string given via HTTP basic auth for API clients. ### Request variables Most API views will have some arguments that are passed as part of the -request to control the behavior of the view. In any well-engineered +request to control the behavior of the view. In any well-engineered view, you need to write code to parse and validate that the arguments -exist and have the correct form. For many applications, this leads to +exist and have the correct form. For many applications, this leads to one of several bad outcomes: - The code isn't written, so arguments aren't validated, leading to @@ -139,7 +139,7 @@ one of several bad outcomes: In Zulip, we solve this problem with a the special decorator called `has_request_variables` which allows a developer to declare the arguments a view function takes and validate their types all within -the `def` line of the function. We like this framework because we +the `def` line of the function. We like this framework because we have found it makes the validation code compact, readable, and conveniently located in the same place as the method it is validating arguments for. @@ -158,11 +158,11 @@ def create_user_backend(request, user_profile, email=REQ(), password=REQ(), ``` You will notice the special `REQ()` in the keyword arguments to -`create_user_backend`. `has_request_variables` parses the declared +`create_user_backend`. `has_request_variables` parses the declared keyword arguments of the decorated function, and for each that has an instance of `REQ` as the default value, it extracts the HTTP parameter with that name from the request, parses it as JSON, and passes it to -the function. It will return an nicely JSON formatted HTTP 400 error +the function. It will return an nicely JSON formatted HTTP 400 error in the event that an argument is missing, doesn't parse as JSON, or otherwise is invalid. @@ -194,7 +194,7 @@ REQ also helps us with request variable validation. For example: not automatically marshall the input from JSON). - Since there is no need to JSON-encode strings, usually simply - `my_string=REQ()` is correct. One can pass e.g. + `my_string=REQ()` is correct. One can pass e.g. `str_validator=check_string_in(...)` where one wants to run a validator on the value of a string. @@ -205,7 +205,7 @@ for more validators and their documentation. ### Deciding which HTTP verb to use When writing a new API view, you should writing a view to do just one -type of thing. Usually that's either a read or write operation. +type of thing. Usually that's either a read or write operation. If you're reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a resource is available to @@ -243,18 +243,18 @@ change the server more than once or cause unwanted side effects. ### Making changes to the database If the view does any modification to the database, that change is done -in a helper function in `zerver/lib/actions.py`. Those functions are +in a helper function in `zerver/lib/actions.py`. Those functions are responsible for doing a complete update to the state of the server, which often entails both updating the database and sending any events -to notify clients about the state change. When possible, we prefer to +to notify clients about the state change. When possible, we prefer to design a clean boundary between the view function and the actions function is such that all user input validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for atomically executing the change (this is usually signalled by having the actions function have a name starting with -`do_`. So in most cases, errors in an actions function will be the +`do_`. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection to the -database) and lead to a 500 error. If an actions function is +database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a name starting with `check_`. diff --git a/static/shared/README.md b/static/shared/README.md index 5d350fd127..73ae7aea33 100644 --- a/static/shared/README.md +++ b/static/shared/README.md @@ -10,7 +10,7 @@ Note that the deployment cycles are different: - In the mobile app, this code is deployed in the same way as the rest of the mobile app: it's bundled up into the app binary which - is uploaded to app stores and users install on their devices. The + is uploaded to app stores and users install on their devices. The client will be running the version built into their version of the mobile app, which may be newer, older, or simply different from the version on the server. diff --git a/tools/droplets/README.md b/tools/droplets/README.md index 78319078d9..c31d91be2c 100644 --- a/tools/droplets/README.md +++ b/tools/droplets/README.md @@ -161,7 +161,7 @@ Rough steps: key of `base.zulipdev.org` during this step. 1. Create a user called `zulipdev` and add it to the `sudo` group. 1. Make sudo of `zulipdev` user passwordless by including - `zulipdev ALL=(ALL) NOPASSWD:ALL` in `/etc/sudoers.d/90-cloud-init-users` + `zulipdev ALL=(ALL) NOPASSWD:ALL` in `/etc/sudoers.d/90-cloud-init-users` 1. Copy the `authorized_keys` file of `root` user to the `.ssh` directory of `zulipdev` user 1. Switch to `zulipdev` user and set the permissions for the `.ssh` folder to `700` and `.ssh/authorized_keys` to `600`.