From 0147c6adce7089ff720a34bd74dc8f298d9792c3 Mon Sep 17 00:00:00 2001 From: Anders Kaseorg Date: Fri, 20 Aug 2021 12:45:39 -0700 Subject: [PATCH] docs: Apply bullet style changes from Prettier. Signed-off-by: Anders Kaseorg (cherry picked from commit 915884bff7f8b8dd4326158e8014d850086ff1e7) --- CODE_OF_CONDUCT.md | 32 ++-- CONTRIBUTING.md | 86 ++++----- README.md | 14 +- docs/contributing/accessibility.md | 28 +-- docs/contributing/bug-reports.md | 16 +- docs/contributing/chat-zulip-org.md | 40 ++--- docs/contributing/code-reviewing.md | 56 +++--- docs/contributing/code-style.md | 12 +- docs/contributing/gsoc-ideas.md | 12 +- docs/contributing/version-control.md | 16 +- docs/contributing/zulipbot-usage.md | 26 +-- docs/development/authentication.md | 74 ++++---- docs/development/overview.md | 8 +- docs/development/remote.md | 46 ++--- docs/development/request-remote.md | 8 +- docs/development/setup-advanced.md | 32 ++-- docs/development/setup-vagrant.md | 58 +++--- docs/development/using.md | 48 ++--- docs/documentation/api.md | 34 ++-- docs/documentation/integrations.md | 30 ++-- docs/documentation/openapi.md | 8 +- docs/documentation/overview.md | 22 +-- docs/documentation/user.md | 82 ++++----- docs/git/zulip-tools.md | 6 +- docs/overview/architecture-overview.md | 24 +-- docs/overview/changelog.md | 178 +++++++++---------- docs/overview/directory-structure.md | 82 ++++----- docs/overview/release-lifecycle.md | 42 ++--- docs/production/authentication-methods.md | 100 +++++------ docs/production/deployment.md | 28 +-- docs/production/email-gateway.md | 8 +- docs/production/email.md | 38 ++-- docs/production/export-and-import.md | 64 +++---- docs/production/install.md | 34 ++-- docs/production/management-commands.md | 34 ++-- docs/production/mobile-push-notifications.md | 40 ++--- docs/production/multiple-organizations.md | 10 +- docs/production/postgresql.md | 16 +- docs/production/requirements.md | 74 ++++---- docs/production/security-model.md | 78 ++++---- docs/production/settings.md | 6 +- docs/production/ssl-certificates.md | 18 +- docs/production/troubleshooting.md | 44 ++--- docs/production/upgrade-or-modify.md | 74 ++++---- docs/production/video-calls.md | 12 +- docs/subsystems/billing.md | 16 +- docs/subsystems/caching.md | 12 +- docs/subsystems/dependencies.md | 44 ++--- docs/subsystems/django-upgrades.md | 14 +- docs/subsystems/email.md | 36 ++-- docs/subsystems/emoji.md | 44 ++--- docs/subsystems/events-system.md | 50 +++--- docs/subsystems/hashchange-system.md | 38 ++-- docs/subsystems/html-css.md | 22 +-- docs/subsystems/logging.md | 60 +++---- docs/subsystems/management-commands.md | 16 +- docs/subsystems/markdown.md | 76 ++++---- docs/subsystems/notifications.md | 56 +++--- docs/subsystems/performance.md | 24 +-- docs/subsystems/pointer.md | 8 +- docs/subsystems/presence.md | 12 +- docs/subsystems/queuing.md | 14 +- docs/subsystems/realms.md | 4 +- docs/subsystems/release-checklist.md | 74 ++++---- docs/subsystems/schema-migrations.md | 42 ++--- docs/subsystems/sending-messages.md | 86 ++++----- docs/subsystems/settings.md | 46 ++--- docs/subsystems/typing-indicators.md | 22 +-- docs/testing/continuous-integration.md | 16 +- docs/testing/linters.md | 8 +- docs/testing/mypy.md | 46 ++--- docs/testing/philosophy.md | 26 +-- docs/testing/testing-with-django.md | 24 +-- docs/testing/testing-with-node.md | 10 +- docs/testing/testing-with-puppeteer.md | 26 +-- docs/testing/testing.md | 4 +- docs/testing/typescript.md | 20 +-- docs/translating/chinese.md | 36 ++-- docs/translating/french.md | 38 ++-- docs/translating/german.md | 80 ++++----- docs/translating/hindi.md | 70 ++++---- docs/translating/internationalization.md | 16 +- docs/translating/polish.md | 18 +- docs/translating/russian.md | 92 +++++----- docs/translating/spanish.md | 86 ++++----- docs/translating/translating.md | 40 ++--- docs/tutorials/life-of-a-request.md | 14 +- docs/tutorials/new-feature-tutorial.md | 4 +- docs/tutorials/writing-views.md | 14 +- scripts/README.md | 4 +- static/shared/README.md | 4 +- tools/droplets/README.md | 4 +- tools/linter_lib/custom_check.py | 2 +- tools/oneclickapps/README.md | 32 ++-- 94 files changed, 1674 insertions(+), 1674 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index e9010f80dc..9113aef564 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -18,15 +18,15 @@ all of us and the technical communities in which we participate. The following behaviors are expected and requested of all community members: -* Participate. In doing so, you contribute to the health and longevity of +- Participate. In doing so, you contribute to the health and longevity of the community. -* Exercise consideration and respect in your speech and actions. -* Attempt collaboration before conflict. Assume good faith. -* Refrain from demeaning, discriminatory, or harassing behavior and speech. -* Take action or alert community leaders if you notice a dangerous +- Exercise consideration and respect in your speech and actions. +- Attempt collaboration before conflict. Assume good faith. +- Refrain from demeaning, discriminatory, or harassing behavior and speech. +- Take action or alert community leaders if you notice a dangerous situation, someone in distress, or violations of this code, even if they seem inconsequential. -* Community event venues may be shared with members of the public; be +- Community event venues may be shared with members of the public; be respectful to all patrons of these locations. ## Unacceptable behavior @@ -34,24 +34,24 @@ The following behaviors are expected and requested of all community members: The following behaviors are considered harassment and are unacceptable within the Zulip community: -* Jokes or derogatory language that singles out members of any race, +- Jokes or derogatory language that singles out members of any race, ethnicity, culture, national origin, color, immigration status, social and economic class, educational level, language proficiency, sex, sexual orientation, gender identity and expression, age, size, family status, political belief, religion, and mental and physical ability. -* Violence, threats of violence, or violent language directed against +- Violence, threats of violence, or violent language directed against another person. -* Disseminating or threatening to disseminate another person's personal +- Disseminating or threatening to disseminate another person's personal information. -* Personal insults of any sort. -* Posting or displaying sexually explicit or violent material. -* Inappropriate photography or recording. -* Deliberate intimidation, stalking, or following (online or in person). -* Unwelcome sexual attention. This includes sexualized comments or jokes, +- Personal insults of any sort. +- Posting or displaying sexually explicit or violent material. +- Inappropriate photography or recording. +- Deliberate intimidation, stalking, or following (online or in person). +- Unwelcome sexual attention. This includes sexualized comments or jokes, inappropriate touching or groping, and unwelcomed sexual advances. -* Sustained disruption of community events, including talks and +- Sustained disruption of community events, including talks and presentations. -* Advocating for, or encouraging, any of the behaviors above. +- Advocating for, or encouraging, any of the behaviors above. ## Reporting and enforcement diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e27092fd3d..1b3684cbae 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,29 +26,29 @@ To make a code or documentation contribution, read our [step-by-step guide](#your-first-codebase-contribution) to getting started with the Zulip codebase. A small sample of the type of work that needs doing: -* Bug squashing and feature development on our Python/Django +- Bug squashing and feature development on our Python/Django [backend](https://github.com/zulip/zulip), web [frontend](https://github.com/zulip/zulip), React Native [mobile app](https://github.com/zulip/zulip-mobile), or Electron [desktop app](https://github.com/zulip/zulip-desktop). -* Building out our +- Building out our [Python API and bots](https://github.com/zulip/python-zulip-api) framework. -* [Writing an integration](https://zulip.com/api/integrations-overview). -* Improving our [user](https://zulip.com/help/) or +- [Writing an integration](https://zulip.com/api/integrations-overview). +- Improving our [user](https://zulip.com/help/) or [developer](https://zulip.readthedocs.io/en/latest/) documentation. -* [Reviewing code](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html) +- [Reviewing code](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html) and manually testing pull requests. **Non-code contributions**: Some of the most valuable ways to contribute don't require touching the codebase at all. We list a few of them below: -* [Reporting issues](#reporting-issues), including both feature requests and +- [Reporting issues](#reporting-issues), including both feature requests and bug reports. -* [Giving feedback](#user-feedback) if you are evaluating or using Zulip. -* [Sponsor Zulip](https://github.com/sponsors/zulip) through the GitHub sponsors program. -* [Translating](https://zulip.readthedocs.io/en/latest/translating/translating.html) +- [Giving feedback](#user-feedback) if you are evaluating or using Zulip. +- [Sponsor Zulip](https://github.com/sponsors/zulip) through the GitHub sponsors program. +- [Translating](https://zulip.readthedocs.io/en/latest/translating/translating.html) Zulip. -* [Outreach](#zulip-outreach): Star us on GitHub, upvote us +- [Outreach](#zulip-outreach): Star us on GitHub, upvote us on product comparison sites, or write for [the Zulip blog](https://blog.zulip.org/). ## Your first (codebase) contribution @@ -57,7 +57,7 @@ This section has a step by step guide to starting as a Zulip codebase contributor. It's long, but don't worry about doing all the steps perfectly; no one gets it right the first time, and there are a lot of people available to help. -* First, make an account on the +- First, make an account on the [Zulip community server](https://zulip.readthedocs.io/en/latest/contributing/chat-zulip-org.html), paying special attention to the community norms. If you'd like, introduce yourself in @@ -65,12 +65,12 @@ to help. your name as the topic. Bonus: tell us about your first impressions of Zulip, and anything that felt confusing/broken as you started using the product. -* Read [What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor). -* [Install the development environment](https://zulip.readthedocs.io/en/latest/development/overview.html), +- Read [What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor). +- [Install the development environment](https://zulip.readthedocs.io/en/latest/development/overview.html), getting help in [#development help](https://chat.zulip.org/#narrow/stream/49-development-help) if you run into any troubles. -* Read the +- Read the [Zulip guide to Git](https://zulip.readthedocs.io/en/latest/git/index.html) and do the Git tutorial (coming soon) if you are unfamiliar with Git, getting help in @@ -84,7 +84,7 @@ Now, you're ready to pick your first issue! There are hundreds of open issues in the main codebase alone. This section will help you find an issue to work on. -* If you're interested in +- If you're interested in [mobile](https://github.com/zulip/zulip-mobile/issues?q=is%3Aopen+is%3Aissue), [desktop](https://github.com/zulip/zulip-desktop/issues?q=is%3Aopen+is%3Aissue), or @@ -93,18 +93,18 @@ on. [#mobile](https://chat.zulip.org/#narrow/stream/48-mobile), [#desktop](https://chat.zulip.org/#narrow/stream/16-desktop), or [#integration](https://chat.zulip.org/#narrow/stream/127-integrations). -* For the main server and web repository, we recommend browsing +- For the main server and web repository, we recommend browsing recently opened issues to look for issues you are confident you can fix correctly in a way that clearly communicates why your changes are the correct fix. Our GitHub workflow bot, zulipbot, limits users who have 0 commits merged to claiming a single issue labeled with "good first issue" or "help wanted". -* We also partition all of our issues in the main repo into areas like +- We also partition all of our issues in the main repo into areas like admin, compose, emoji, hotkeys, i18n, onboarding, search, etc. Look through our [list of labels](https://github.com/zulip/zulip/labels), and click on some of the `area:` labels to see all the issues related to your areas of interest. -* If the lists of issues are overwhelming, post in +- If the lists of issues are overwhelming, post in [#new members](https://chat.zulip.org/#narrow/stream/95-new-members) with a bit about your background and interests, and we'll help you out. The most important thing to say is whether you're looking for a backend (Python), @@ -119,10 +119,10 @@ have a new feature you'd like to add, we recommend you start by posting in feature idea and the problem that you're hoping to solve. Other notes: -* For a first pull request, it's better to aim for a smaller contribution +- For a first pull request, it's better to aim for a smaller contribution than a bigger one. Many first contributions have fewer than 10 lines of changes (not counting changes to tests). -* The full list of issues explicitly looking for a contributor can be +- The full list of issues explicitly looking for a contributor can be found with the [good first issue](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) and @@ -133,7 +133,7 @@ Other notes: are fair game if Tim has written a clear technical design proposal in the issue, or it is a bug that you can reproduce and you are confident you can fix the issue correctly. -* For most new contributors, there's a lot to learn while making your first +- For most new contributors, there's a lot to learn while making your first pull request. It's OK if it takes you a while; that's normal! You'll be able to work a lot faster as you build experience. @@ -144,20 +144,20 @@ the issue thread. [Zulipbot](https://github.com/zulip/zulipbot) is a GitHub workflow bot; it will assign you to the issue and label the issue as "in progress". Some additional notes: -* You can only claim issues with the +- You can only claim issues with the [good first issue](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) or [help wanted](https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) labels. Zulipbot will give you an error if you try to claim an issue without one of those labels. -* You're encouraged to ask questions on how to best implement or debug your +- You're encouraged to ask questions on how to best implement or debug your changes -- the Zulip maintainers are excited to answer questions to help you stay unblocked and working efficiently. You can ask questions on chat.zulip.org, or on the GitHub issue or pull request. -* We encourage early pull requests for work in progress. Prefix the title of +- We encourage early pull requests for work in progress. Prefix the title of work in progress pull requests with `[WIP]`, and remove the prefix when you think it might be mergeable and want it to be reviewed. -* After updating a PR, add a comment to the GitHub thread mentioning that it +- After updating a PR, add a comment to the GitHub thread mentioning that it is ready for another review. GitHub only notifies maintainers of the changes when you post a comment, so if you don't, your PR will likely be neglected by accident! @@ -175,23 +175,23 @@ labels. Zulip has a lot of experience working with new contributors. In our experience, these are the best predictors of success: -* Posting good questions. This generally means explaining your current +- Posting good questions. This generally means explaining your current understanding, saying what you've done or tried so far, and including tracebacks or other error messages if appropriate. -* Learning and practicing +- Learning and practicing [Git commit discipline](https://zulip.readthedocs.io/en/latest/contributing/version-control.html#commit-discipline). -* Submitting carefully tested code. This generally means checking your work +- Submitting carefully tested code. This generally means checking your work through a combination of automated tests and manually clicking around the UI trying to find bugs in your work. See [things to look for](https://zulip.readthedocs.io/en/latest/contributing/code-reviewing.html#things-to-look-for) for additional ideas. -* Posting +- Posting [screenshots or GIFs](https://zulip.readthedocs.io/en/latest/tutorials/screenshot-and-gif-software.html) for frontend changes. -* Being responsive to feedback on pull requests. This means incorporating or +- Being responsive to feedback on pull requests. This means incorporating or responding to all suggested changes, and leaving a note if you won't be able to address things within a few days. -* Being helpful and friendly on chat.zulip.org. +- Being helpful and friendly on chat.zulip.org. These are also the main criteria we use to select candidates for all of our outreach programs. @@ -227,17 +227,17 @@ hear about your experience with the product. If you're not sure what to write, here are some questions we're always very curious to know the answer to: -* Evaluation: What is the process by which your organization chose or will +- Evaluation: What is the process by which your organization chose or will choose a group chat product? -* Pros and cons: What are the pros and cons of Zulip for your organization, +- Pros and cons: What are the pros and cons of Zulip for your organization, and the pros and cons of other products you are evaluating? -* Features: What are the features that are most important for your +- Features: What are the features that are most important for your organization? In the best-case scenario, what would your chat solution do for you? -* Onboarding: If you remember it, what was your impression during your first +- Onboarding: If you remember it, what was your impression during your first few minutes of using Zulip? What did you notice, and how did you feel? Was there anything that stood out to you as confusing, or broken, or great? -* Organization: What does your organization do? How big is the organization? +- Organization: What does your organization do? How big is the organization? A link to your organization's website? ## Outreach programs @@ -252,15 +252,15 @@ summer interns from Harvard, MIT, and Stanford. While each third-party program has its own rules and requirements, the Zulip community's approaches all of these programs with these ideas in mind: -* We try to make the application process as valuable for the applicant as +- We try to make the application process as valuable for the applicant as possible. Expect high-quality code reviews, a supportive community, and publicly viewable patches you can link to from your resume, regardless of whether you are selected. -* To apply, you'll have to submit at least one pull request to a Zulip +- To apply, you'll have to submit at least one pull request to a Zulip repository. Most students accepted to one of our programs have several merged pull requests (including at least one larger PR) by the time of the application deadline. -* The main criteria we use is quality of your best contributions, and +- The main criteria we use is quality of your best contributions, and the bullets listed at [What makes a great Zulip contributor](#what-makes-a-great-zulip-contributor). Because we focus on evaluating your best work, it doesn't hurt your @@ -307,23 +307,23 @@ for ZSoC, we'll contact you when the GSoC results are announced. perception of projects like Zulip. We've collected a few sites below where we know Zulip has been discussed. Doing everything in the following list typically takes about 15 minutes. -* Star us on GitHub. There are four main repositories: +- Star us on GitHub. There are four main repositories: [server/web](https://github.com/zulip/zulip), [mobile](https://github.com/zulip/zulip-mobile), [desktop](https://github.com/zulip/zulip-desktop), and [Python API](https://github.com/zulip/python-zulip-api). -* [Follow us](https://twitter.com/zulip) on Twitter. +- [Follow us](https://twitter.com/zulip) on Twitter. For both of the following, you'll need to make an account on the site if you don't already have one. -* [Like Zulip](https://alternativeto.net/software/zulip-chat-server/) on +- [Like Zulip](https://alternativeto.net/software/zulip-chat-server/) on AlternativeTo. We recommend upvoting a couple of other products you like as well, both to give back to their community, and since single-upvote accounts are generally given less weight. You can also [upvote Zulip](https://alternativeto.net/software/slack/) on their page for Slack. -* [Add Zulip to your stack](https://stackshare.io/zulip) on StackShare, star +- [Add Zulip to your stack](https://stackshare.io/zulip) on StackShare, star it, and upvote the reasons why people like Zulip that you find most compelling. Again, we recommend adding a few other products that you like as well. diff --git a/README.md b/README.md index 4b2e795171..85136eaa34 100644 --- a/README.md +++ b/README.md @@ -30,13 +30,13 @@ and tell us what's up! You might be interested in: -* **Contributing code**. Check out our +- **Contributing code**. Check out our [guide for new contributors](https://zulip.readthedocs.io/en/latest/overview/contributing.html) to get started. Zulip prides itself on maintaining a clean and well-tested codebase, and a stock of hundreds of [beginner-friendly issues][beginner-friendly]. -* **Contributing non-code**. +- **Contributing non-code**. [Report an issue](https://zulip.readthedocs.io/en/latest/overview/contributing.html#reporting-issues), [translate](https://zulip.readthedocs.io/en/latest/translating/translating.html) Zulip into your language, @@ -45,12 +45,12 @@ You might be interested in: [give us feedback](https://zulip.readthedocs.io/en/latest/overview/contributing.html#user-feedback). We would love to hear from you, even if you're just trying the product out. -* **Supporting Zulip**. Advocate for your organization to use Zulip, become a [sponsor](https://github.com/sponsors/zulip), write a +- **Supporting Zulip**. Advocate for your organization to use Zulip, become a [sponsor](https://github.com/sponsors/zulip), write a review in the mobile app stores, or [upvote Zulip](https://zulip.readthedocs.io/en/latest/overview/contributing.html#zulip-outreach) on product comparison sites. -* **Checking Zulip out**. The best way to see Zulip in action is to drop by +- **Checking Zulip out**. The best way to see Zulip in action is to drop by the [Zulip community server](https://zulip.readthedocs.io/en/latest/contributing/chat-zulip-org.html). We also recommend reading Zulip for @@ -58,17 +58,17 @@ You might be interested in: [companies](https://zulip.com/for/companies/), or Zulip for [working groups and part time communities](https://zulip.com/for/working-groups-and-communities/). -* **Running a Zulip server**. Use a preconfigured [DigitalOcean droplet](https://marketplace.digitalocean.com/apps/zulip), +- **Running a Zulip server**. Use a preconfigured [DigitalOcean droplet](https://marketplace.digitalocean.com/apps/zulip), [install Zulip](https://zulip.readthedocs.io/en/stable/production/install.html) directly, or use Zulip's experimental [Docker image](https://zulip.readthedocs.io/en/latest/production/deployment.html#zulip-in-docker). Commercial support is available; see for details. -* **Using Zulip without setting up a server**. +- **Using Zulip without setting up a server**. offers free and commercial hosting, including providing our paid plan for free to fellow open source projects. -* **Participating in [outreach +- **Participating in [outreach programs](https://zulip.readthedocs.io/en/latest/overview/contributing.html#outreach-programs)** like Google Summer of Code. diff --git a/docs/contributing/accessibility.md b/docs/contributing/accessibility.md index e960852d0c..783f43d791 100644 --- a/docs/contributing/accessibility.md +++ b/docs/contributing/accessibility.md @@ -6,14 +6,14 @@ In order to accommodate all users, Zulip strives to implement accessibility best practices in its user interface. There are many aspects to accessibility; here are some of the more important ones to keep in mind. -* All images should have alternative text attributes for the benefit of users +- All images should have alternative text attributes for the benefit of users who cannot see them (this includes users who are utilizing a voice interface to free up their eyes to look at something else instead). -* The entire application should be usable via a keyboard (many users are unable +- The entire application should be usable via a keyboard (many users are unable to use a mouse, and many accessibility aids emulate a keyboard). -* Text should have good enough contrast against the background to enable +- Text should have good enough contrast against the background to enable even users with moderate visual impairment to be able to read it. -* [ARIA](https://www.w3.org/WAI/intro/aria) (Accessible Rich Internet +- [ARIA](https://www.w3.org/WAI/intro/aria) (Accessible Rich Internet Application) attributes should be used appropriately to enable screen readers and other alternative interfaces to navigate the application effectively. @@ -32,19 +32,19 @@ as it is not possible for some content.) There are tools available to automatically audit a web page for compliance with many of the WCAG guidelines. Here are some of the more useful ones: -* [Accessibility Developer Tools][chrome-webstore] +- [Accessibility Developer Tools][chrome-webstore] This open source Chrome extension from Google adds an accessibility audit to the "Audits" tab of the Chrome Developer Tools. The audit is performed against the page's DOM via JavaScript, allowing it to identify some issues that a static HTML inspector would miss. -* [aXe](https://www.deque.com/products/axe/) An open source Chrome and Firefox +- [aXe](https://www.deque.com/products/axe/) An open source Chrome and Firefox extension which runs a somewhat different set of checks than Google's Chrome extension. -* [Wave](https://wave.webaim.org/) This web application takes a URL and loads +- [Wave](https://wave.webaim.org/) This web application takes a URL and loads it in a frame, reporting on all the issues it finds with links to more information. Has the advantage of not requiring installation, but requires a URL which can be directly accessed by an external site. -* [Web Developer](https://chrispederick.com/work/web-developer/) This browser +- [Web Developer](https://chrispederick.com/work/web-developer/) This browser extension has many useful features, including a convenient link for opening the current URL in Wave to get an accessibility report. @@ -70,14 +70,14 @@ If you want to help make Zulip more accessible, here is a list of the For more information about making Zulip accessible to as many users as possible, the following resources may be useful. -* [Font Awesome accessibility guide](https://fontawesome.com/how-to-use/on-the-web/other-topics/accessibility), +- [Font Awesome accessibility guide](https://fontawesome.com/how-to-use/on-the-web/other-topics/accessibility), which is especially helpful since Zulip uses Font Awesome for its icons. -* [Web Content Accessibility Guidelines (WCAG) 2.0](https://www.w3.org/TR/WCAG/) -* [WAI-ARIA](https://www.w3.org/WAI/intro/aria) - Web Accessibility Initiative +- [Web Content Accessibility Guidelines (WCAG) 2.0](https://www.w3.org/TR/WCAG/) +- [WAI-ARIA](https://www.w3.org/WAI/intro/aria) - Web Accessibility Initiative Accessible Rich Internet Application Suite -* [WebAIM](https://webaim.org/) - Web Accessibility in Mind -* The [MDN page on accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility) -* The [Open edX Accessibility Guidelines][openedx-guidelines] for developers +- [WebAIM](https://webaim.org/) - Web Accessibility in Mind +- The [MDN page on accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility) +- The [Open edX Accessibility Guidelines][openedx-guidelines] for developers [chrome-webstore]: https://chrome.google.com/webstore/detail/accessibility-developer-t/fpkknkljclfencbdbgkenhalefipecmb diff --git a/docs/contributing/bug-reports.md b/docs/contributing/bug-reports.md index 868b8ca9b3..9ec9f808eb 100644 --- a/docs/contributing/bug-reports.md +++ b/docs/contributing/bug-reports.md @@ -2,21 +2,21 @@ Please include these elements in your bug report to make it easier for us to help you. -* A brief title +- A brief title -* An explanation of what you were expecting vs. the actual result +- An explanation of what you were expecting vs. the actual result -* Steps to take in order to reproduce the buggy behavior +- Steps to take in order to reproduce the buggy behavior -* Whether you are using Zulip in production or in the development +- Whether you are using Zulip in production or in the development environment, and whether these are old versions -* Whether you are using the web app, a desktop app or a mobile device +- Whether you are using the web app, a desktop app or a mobile device to access Zulip -* Any additional information that would help: screenshots, GIFs, a +- Any additional information that would help: screenshots, GIFs, a pastebin of the error log Further reading: -* [How to write a bug report that will make your engineers love you](https://testlio.com/blog/the-ideal-bug-report/) -* [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html) +- [How to write a bug report that will make your engineers love you](https://testlio.com/blog/the-ideal-bug-report/) +- [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html) diff --git a/docs/contributing/chat-zulip-org.md b/docs/contributing/chat-zulip-org.md index a1e4c7906f..08f25d23b5 100644 --- a/docs/contributing/chat-zulip-org.md +++ b/docs/contributing/chat-zulip-org.md @@ -12,28 +12,28 @@ minutes to a few hours, depending on the time of day. ## Community norms -* Send test messages to +- Send test messages to [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) or as a PM to yourself to avoid disturbing others. -* When asking for help, provide the details needed for others to help +- When asking for help, provide the details needed for others to help you. E.g. include the full traceback in a code block (not a screenshot), a link to the code or a WIP PR you're having trouble debugging, etc. -* Ask questions on streams rather than PMing core contributors. +- Ask questions on streams rather than PMing core contributors. You'll get answers faster since other people can help, and it makes it possible for other developers to learn from reading the discussion. -* Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's +- Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it's usually easy to see which message you're replying to, so you don't need to mention your conversation partner in every reply. Mentioning other users is great for timely questions or making sure someone who is not online sees your message. -* Converse informally; there's no need to use titles like "Sir" or "Madam". -* Use +- Converse informally; there's no need to use titles like "Sir" or "Madam". +- Use [gender-neutral language](https://en.wikipedia.org/wiki/Gender-neutral_language). For example, avoid using a pronoun like her or his in sentences like "Every developer should clean [their] keyboard at least once a week." -* Follow the community [code of conduct](../code-of-conduct.md). -* Participate! Zulip is a friendly and welcoming community, and we +- Follow the community [code of conduct](../code-of-conduct.md). +- Participate! Zulip is a friendly and welcoming community, and we love meeting new people, hearing about what brought them to Zulip, and getting their feedback. If you're not sure where to start, introduce yourself and your interests in @@ -67,19 +67,19 @@ secret/embarrassing, etc. There are a few streams worth highlighting that are relevant for everyone, even non-developers: -* [#announce](https://chat.zulip.org/#narrow/stream/1-announce) is for +- [#announce](https://chat.zulip.org/#narrow/stream/1-announce) is for announcements and discussions thereof; we try to keep traffic there to a minimum. -* [#feedback](https://chat.zulip.org/#narrow/stream/137-feedback) is for +- [#feedback](https://chat.zulip.org/#narrow/stream/137-feedback) is for posting feedback on Zulip. -* [#design](https://chat.zulip.org/#narrow/stream/101-design) is where we +- [#design](https://chat.zulip.org/#narrow/stream/101-design) is where we discuss UI and feature design and collect feedback on potential design changes. We love feedback, so don't hesitate to speak up! -* [#user community](https://chat.zulip.org/#narrow/stream/138-user-community) is +- [#user community](https://chat.zulip.org/#narrow/stream/138-user-community) is for Zulip users to discuss their experiences using and adopting Zulip. -* [#production help](https://chat.zulip.org/#narrow/stream/31-production-help) +- [#production help](https://chat.zulip.org/#narrow/stream/31-production-help) is for production environment related discussions. -* [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) is +- [#test here](https://chat.zulip.org/#narrow/stream/7-test-here) is for sending test messages without inconveniencing other users :). We recommend muting this stream when not using it. @@ -88,25 +88,25 @@ community (e.g. one for each app, etc.); check out the [Streams page](https://chat.zulip.org/#streams/all) to see the descriptions for all of them. Relevant to almost everyone are these: -* [#checkins](https://chat.zulip.org/#narrow/stream/65-checkins) is for +- [#checkins](https://chat.zulip.org/#narrow/stream/65-checkins) is for progress updates on what you're working on and its status; usually folks post with their name as the topic. Everyone is welcome to participate! -* [#development help](https://chat.zulip.org/#narrow/stream/49-development-help) +- [#development help](https://chat.zulip.org/#narrow/stream/49-development-help) is for asking for help with any Zulip server/webapp development work (use the app streams for help working on one of the apps). -* [#code review](https://chat.zulip.org/#narrow/stream/91-code-review) +- [#code review](https://chat.zulip.org/#narrow/stream/91-code-review) is for getting feedback on your work. We encourage all developers to comment on work posted here, even if you're new to the Zulip project; reviewing other PRs is a great way to develop experience, and even just manually testing a proposed new feature and posting feedback is super helpful. -* [#documentation](https://chat.zulip.org/#narrow/stream/19-documentation) +- [#documentation](https://chat.zulip.org/#narrow/stream/19-documentation) is where we discuss improving Zulip's user, sysadmin, and developer documentation. -* [#translation](https://chat.zulip.org/#narrow/stream/58-translation) is +- [#translation](https://chat.zulip.org/#narrow/stream/58-translation) is for discussing Zulip's translations. -* [#learning](https://chat.zulip.org/#narrow/stream/92-learning) is for +- [#learning](https://chat.zulip.org/#narrow/stream/92-learning) is for posting great learning resources one comes across. There are also official private streams, including large ones for diff --git a/docs/contributing/code-reviewing.md b/docs/contributing/code-reviewing.md index 1e038de7c4..a1f294c39b 100644 --- a/docs/contributing/code-reviewing.md +++ b/docs/contributing/code-reviewing.md @@ -10,8 +10,8 @@ When you send a PR, try to think of a good person to review it -- outside of the handful of people who do a ton of reviews -- and `@`-mention them with something like "`@person`, would you review this?". Good choices include -* someone based in your timezone or a nearby timezone -* people working on similar things, or in a loosely related area +- someone based in your timezone or a nearby timezone +- people working on similar things, or in a loosely related area Alternatively, posting a message in [#code-review](https://chat.zulip.org/#narrow/stream/91-code-review) on [the Zulip @@ -28,14 +28,14 @@ to dive right into reviewing the PR's core functionality. Once you've received a review and resolved any feedback, it's critical to update the GitHub thread to reflect that. Best practices are to: -* Make sure that CI passes and the PR is rebased onto recent `main`. -* Post comments on each feedback thread explaining at least how you +- Make sure that CI passes and the PR is rebased onto recent `main`. +- Post comments on each feedback thread explaining at least how you resolved the feedback, as well as any other useful information (problems encountered, reasoning for why you picked one of several options, a test you added to make sure the bug won't recur, etc.). -* Mark any resolved threads as "resolved" in the GitHub UI, if +- Mark any resolved threads as "resolved" in the GitHub UI, if appropriate. -* Post a summary comment in the main feed for the PR, explaining that +- Post a summary comment in the main feed for the PR, explaining that this is ready for another review, and summarizing any changes from the previous version, details on how you tested the changes, new screenshots/etc. More detail is better than less, as long as you @@ -60,10 +60,10 @@ Anyone can do a code review -- you don't have to have a ton of experience, and you don't have to have the power to ultimately merge the PR. If you -* read the code, see if you understand what the change is +- read the code, see if you understand what the change is doing and why, and ask questions if you don't; or -* fetch the code (for Zulip server code, +- fetch the code (for Zulip server code, [tools/fetch-rebase-pull-request][git tool] is super handy), play around with it in your dev environment, and say what you think about how the feature works @@ -106,7 +106,7 @@ sooner is better. ## Things to look for -* *The CI build.* The tests need to pass. One can investigate +- *The CI build.* The tests need to pass. One can investigate any failures and figure out what to fix by clicking on a red X next to the commit hash or the Detail links on a pull request. (Example: in [#17584](https://github.com/zulip/zulip/pull/17584), @@ -120,25 +120,25 @@ sooner is better. See our docs on [continuous integration](../testing/continuous-integration.md) to learn more. -* *Technical design.* There are a lot of considerations here: +- *Technical design.* There are a lot of considerations here: security, migration paths/backwards compatibility, cost of new dependencies, interactions with features, speed of performance, API changes. Security is especially important and worth thinking about carefully with any changes to security-sensitive code like views. -* *User interface and visual design.* If frontend changes are +- *User interface and visual design.* If frontend changes are involved, the reviewer will check out the code, play with the new UI, and verify it for both quality and consistency with the rest of the Zulip UI. We highly encourage posting screenshots to save reviewers time in getting a feel for what the feature looks like -- you'll get a quicker response that way. -* *Error handling.* The code should always check for invalid user +- *Error handling.* The code should always check for invalid user input. User-facing error messages should be clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct the problem). -* *Testing.* The tests should validate that the feature works +- *Testing.* The tests should validate that the feature works correctly, and specifically test for common error conditions, bad user input, and potential bugs that are likely for the type of change being made. Tests that exclude whole classes of potential @@ -147,16 +147,16 @@ sooner is better. Markdown processors](../subsystems/markdown.md), or the `GetEventsTest` test for buggy race condition handling). -* *Translation.* Make sure that the strings are marked for +- *Translation.* Make sure that the strings are marked for [translation]. -* *Clear function, argument, variable, and test names.* Every new +- *Clear function, argument, variable, and test names.* Every new piece of Zulip code will be read many times by other developers, and future developers will grep for relevant terms when researching a problem, so it's important that variable names communicate clearly the purpose of each piece of the codebase. -* *Duplicated code.* Code duplication is a huge source of bugs in +- *Duplicated code.* Code duplication is a huge source of bugs in large projects and makes the codebase difficult to understand, so we avoid significant code duplication wherever possible. Sometimes avoiding code duplication involves some refactoring of existing @@ -169,27 +169,27 @@ sooner is better. conflicts) if there are still user experience issues under discussion for the feature itself. -* *Completeness.* For refactorings, verify that the changes are +- *Completeness.* For refactorings, verify that the changes are complete. Usually one can check that efficiently using `git grep`, and it's worth it, as we very frequently find issues by doing so. -* *Documentation updates.* If this changes how something works, does it +- *Documentation updates.* If this changes how something works, does it update the documentation in a corresponding way? If it's a new feature, is it documented, and documented in the right place? -* *Good comments.* It's often worth thinking about whether explanation +- *Good comments.* It's often worth thinking about whether explanation in a commit message or pull request discussion should be included in a comment, `/docs`, or other documentation. But it's better yet if verbose explanation isn't needed. We prefer writing code that is readable without explanation over a heavily commented codebase using lots of clever tricks. -* *Coding style.* See the Zulip [code-style] documentation for +- *Coding style.* See the Zulip [code-style] documentation for details. Our goal is to have as much of this as possible verified via the linters and tests, but there's always going to be unusual forms of Python/JavaScript style that our tools don't check for. -* *Clear commit messages.* See the [Zulip version +- *Clear commit messages.* See the [Zulip version control][commit-messages] documentation for details on what we look for. @@ -197,15 +197,15 @@ sooner is better. Some points specific to the Zulip server codebase: -* *Testing -- Backend.* We are trying to maintain ~100% test coverage +- *Testing -- Backend.* We are trying to maintain ~100% test coverage on the backend, so backend changes should have negative tests for the various error conditions. -* *Testing -- Frontend.* If the feature involves frontend changes, +- *Testing -- Frontend.* If the feature involves frontend changes, there should be frontend tests. See the [test writing][test-writing] documentation for more details. -* *mypy annotations.* New functions should be annotated using [mypy] +- *mypy annotations.* New functions should be annotated using [mypy] and existing annotations should be updated. Use of `Any`, `ignore`, and unparameterized containers should be limited to cases where a more precise type cannot be specified. @@ -226,14 +226,14 @@ the maintainer time and get the PR merged quicker. We also strongly recommend reviewers to go through the following resources. -* [The Gentle Art of Patch Review](https://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/) +- [The Gentle Art of Patch Review](https://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/) article by Sarah Sharp -* [Zulip & Good Code Review](https://www.harihareswara.net/sumana/2016/05/17/0) +- [Zulip & Good Code Review](https://www.harihareswara.net/sumana/2016/05/17/0) article by Sumana Harihareswara -* [Code Review - A consolidation of advice and stuff from the +- [Code Review - A consolidation of advice and stuff from the internet](https://gist.github.com/porterjamesj/002fb27dd70df003646df46f15e898de) article by James J. Porter -* [Zulip code of conduct](../code-of-conduct.md) +- [Zulip code of conduct](../code-of-conduct.md) [code-style]: ../contributing/code-style.md [commit-messages]: ../contributing/version-control.html#commit-messages diff --git a/docs/contributing/code-style.md b/docs/contributing/code-style.md index f20aeab1e0..83d7b66682 100644 --- a/docs/contributing/code-style.md +++ b/docs/contributing/code-style.md @@ -172,14 +172,14 @@ libraries as `from datetime import datetime, timezone` and `from django.utils.timezone import now as timezone_now`. Use: -* `timezone_now()` to get a datetime when Django is available, such as +- `timezone_now()` to get a datetime when Django is available, such as in `zerver/`. -* `datetime.now(tz=timezone.utc)` when Django is not available, such as +- `datetime.now(tz=timezone.utc)` when Django is not available, such as for bots and scripts. -* `datetime.fromtimestamp(timestamp, tz=timezone.utc)` if creating a +- `datetime.fromtimestamp(timestamp, tz=timezone.utc)` if creating a datetime from a timestamp. This is also available as `zerver.lib.timestamp.timestamp_to_datetime`. -* `datetime.strptime(date_string, format).replace(tzinfo=timezone.utc)` if +- `datetime.strptime(date_string, format).replace(tzinfo=timezone.utc)` if creating a datetime from a formatted string that is in UTC. Idioms that result in timezone-naive datetimes, and should be avoided, are @@ -189,10 +189,10 @@ parameter, `datetime.utcnow()` and `datetime.utcfromtimestamp()`, and the end. Additional notes: -* Especially in scripts and puppet configuration where Django is not +- Especially in scripts and puppet configuration where Django is not available, using `time.time()` to get timestamps can be cleaner than dealing with datetimes. -* All datetimes on the backend should be in UTC, unless there is a good +- All datetimes on the backend should be in UTC, unless there is a good reason to do otherwise. ### `x.attr('zid')` vs. `rows.id(x)` diff --git a/docs/contributing/gsoc-ideas.md b/docs/contributing/gsoc-ideas.md index 2aa2f63023..7f62aab16d 100644 --- a/docs/contributing/gsoc-ideas.md +++ b/docs/contributing/gsoc-ideas.md @@ -100,17 +100,17 @@ make contributions, and make a good proposal. Your application should include the following: -* Details on any experience you have related to the technologies that +- Details on any experience you have related to the technologies that Zulip has, or related to our product approach. -* Links to materials to help us evaluate your level of experience and +- Links to materials to help us evaluate your level of experience and how you work, such as personal projects of yours, including any existing open source or open culture contributions you've made and any bug reports you've submitted to open source projects. -* Some notes on what you are hoping to get out of your project. -* A description of the project you'd like to do, and why you're +- Some notes on what you are hoping to get out of your project. +- A description of the project you'd like to do, and why you're excited about it. -* Some notes on why you're excited about working on Zulip. -* A link to the initial contribution(s) you did. +- Some notes on why you're excited about working on Zulip. +- A link to the initial contribution(s) you did. We expect applicants to either have experience with the technologies relevant to their project or have strong general programming diff --git a/docs/contributing/version-control.md b/docs/contributing/version-control.md index 13e7bcfd85..2d5fa63f7b 100644 --- a/docs/contributing/version-control.md +++ b/docs/contributing/version-control.md @@ -91,12 +91,12 @@ First, check out of commits with good commit messages. The first line of the commit message is the **summary**. The summary: -* is written in the imperative (e.g., "Fix ...", "Add ...") -* is kept short (max 76 characters, ideally less), while concisely +- is written in the imperative (e.g., "Fix ...", "Add ...") +- is kept short (max 76 characters, ideally less), while concisely explaining what the commit does -* is clear about what part of the code is affected -- often by prefixing +- is clear about what part of the code is affected -- often by prefixing with the name of the subsystem and a colon, like "zjsunit: ..." or "docs: ..." -* is a complete sentence. +- is a complete sentence. ### Good summaries: @@ -124,13 +124,13 @@ Here are some more positive examples: Compare "*gather_subscriptions: Fix exception handling bad input.*" with: -* "*gather_subscriptions was broken*", which doesn't explain how +- "*gather_subscriptions was broken*", which doesn't explain how it was broken (and isn't in the imperative) -* "*Fix exception when given bad input*", in which it's impossible to +- "*Fix exception when given bad input*", in which it's impossible to tell from the summary what part of the codebase was changed -* "*gather_subscriptions: Fixing exception when given bad input.*", +- "*gather_subscriptions: Fixing exception when given bad input.*", not in the imperative -* "*gather_subscriptions: Fixed exception when given bad input.*", +- "*gather_subscriptions: Fixed exception when given bad input.*", not in the imperative The summary is followed by a blank line, and then the body of the diff --git a/docs/contributing/zulipbot-usage.md b/docs/contributing/zulipbot-usage.md index 7fed242524..d3df0d5657 100644 --- a/docs/contributing/zulipbot-usage.md +++ b/docs/contributing/zulipbot-usage.md @@ -13,47 +13,47 @@ model GitHub supports). ## Usage -* **Claim an issue** — Comment `@zulipbot claim` on the issue you want +- **Claim an issue** — Comment `@zulipbot claim` on the issue you want to claim; **@zulipbot** will assign you to the issue and label the issue as **in progress**. - * If you're a new contributor, **@zulipbot** will give you read-only + - If you're a new contributor, **@zulipbot** will give you read-only collaborator access to the repository and leave a welcome message on the issue you claimed. - * You can also claim an issue that you've opened by including + - You can also claim an issue that you've opened by including `@zulipbot claim` in the body of your issue. - * If you accidentally claim an issue you didn't want to claim, comment + - If you accidentally claim an issue you didn't want to claim, comment `@zulipbot abandon` to abandon an issue. -* **Label your issues** — Add appropriate labels to issues that you opened by +- **Label your issues** — Add appropriate labels to issues that you opened by including `@zulipbot add` in an issue comment or the body of your issue followed by the desired labels enclosed within double quotes (`""`). - * For example, to add the **bug** and **help wanted** labels to your + - For example, to add the **bug** and **help wanted** labels to your issue, comment or include `@zulipbot add "bug" "help wanted"` in the issue body. - * You'll receive an error message if you try to add any labels to your issue + - You'll receive an error message if you try to add any labels to your issue that don't exist in your repository. - * If you accidentally added the wrong labels, you can remove them by commenting + - If you accidentally added the wrong labels, you can remove them by commenting `@zulipbot remove` followed by the desired labels enclosed with double quotes (`""`). -* **Find unclaimed issues** — Use the [GitHub search +- **Find unclaimed issues** — Use the [GitHub search feature](https://help.github.com/en/articles/using-search-to-filter-issues-and-pull-requests) to find unclaimed issues by adding one of the following filters to your search: - * `-label: "in progress"` (excludes issues labeled with the **in progress** label) + - `-label: "in progress"` (excludes issues labeled with the **in progress** label) - * `no:assignee` (shows issues without assignees) + - `no:assignee` (shows issues without assignees) Issues labeled with the **in progress** label and/or assigned to other users have already been claimed. -* **Collaborate in area label teams** — Receive notifications on +- **Collaborate in area label teams** — Receive notifications on issues and pull requests within your fields of expertise on the [Zulip server repository](https://github.com/zulip/zulip) by joining the Zulip server @@ -73,7 +73,7 @@ team. Feel free to join as many area label teams as as you'd like! label as well as any pull requests that reference issues labeled with your team's area label. -* **Track inactive claimed issues** — If a claimed issue has not been updated +- **Track inactive claimed issues** — If a claimed issue has not been updated for a week, **@zulipbot** will post a comment on the inactive issue to ask the assignee(s) if they are still working on the issue. diff --git a/docs/development/authentication.md b/docs/development/authentication.md index 7f42c0f9d2..28014302c9 100644 --- a/docs/development/authentication.md +++ b/docs/development/authentication.md @@ -31,12 +31,12 @@ control over their email address, and then allowing them to set a password for their account. There are two development environment details worth understanding: -* All of our authentication flows in the development environment have +- All of our authentication flows in the development environment have special links to the `/emails` page (advertised in `/devtools`), which shows all emails that the Zulip server has "sent" (emails are not actually sent by the development environment), to make it convenient to click through the UI of signup, password reset, etc. -* There's a management command, +- There's a management command, `manage.py print_initial_password username@example.com`, that prints out **default** passwords for the development environment users. Note that if you change a user's password in the development @@ -45,89 +45,89 @@ details worth understanding: ### Google -* Visit [the Google developer +- Visit [the Google developer console](https://console.developers.google.com) and navigate to "APIs & services" > "Credentials". Create a "Project", which will correspond to your dev environment. -* Navigate to "APIs & services" > "Library", and find the "Identity +- Navigate to "APIs & services" > "Library", and find the "Identity Toolkit API". Choose "Enable". -* Return to "Credentials", and select "Create credentials". Choose +- Return to "Credentials", and select "Create credentials". Choose "OAuth client ID", and follow prompts to create a consent screen, etc. For "Authorized redirect URIs", fill in `http://zulipdev.com:9991/complete/google/` . -* You should get a client ID and a client secret. Copy them. In +- You should get a client ID and a client secret. Copy them. In `dev-secrets.conf`, set `social_auth_google_key` to the client ID and `social_auth_google_secret` to the client secret. ### GitHub -* Register an OAuth2 application with GitHub at one of +- Register an OAuth2 application with GitHub at one of or . Specify `http://zulipdev.com:9991/complete/github/` as the callback URL. -* You should get a page with settings for your new application, +- You should get a page with settings for your new application, showing a client ID and a client secret. In `dev-secrets.conf`, set `social_auth_github_key` to the client ID and `social_auth_github_secret` to the client secret. ### GitLab -* Register an OAuth application with GitLab at +- Register an OAuth application with GitLab at . Specify `http://zulipdev.com:9991/complete/gitlab` as the callback URL. -* You should get a page containing the Application ID and Secret for +- You should get a page containing the Application ID and Secret for your new application. In `dev-secrets.conf`, enter the Application ID as `social_auth_gitlab_key` and the Secret as `social_auth_gitlab_secret`. ### Apple -* Visit , +- Visit , Enable App ID and Create a Services ID with the instructions in . When prompted for a "Return URL", enter `http://zulipdev.com:9991/complete/apple/` . -* [Create a Sign in with Apple private key](https://help.apple.com/developer-account/?lang=en#/dev77c875b7e) +- [Create a Sign in with Apple private key](https://help.apple.com/developer-account/?lang=en#/dev77c875b7e) -* In `dev-secrets.conf`, set - * `social_auth_apple_services_id` to your +- In `dev-secrets.conf`, set + - `social_auth_apple_services_id` to your "Services ID" (eg. com.application.your). - * `social_auth_apple_app_id` to "App ID" or "Bundle ID". + - `social_auth_apple_app_id` to "App ID" or "Bundle ID". This is only required if you are testing Apple auth on iOS. - * `social_auth_apple_key` to your "Key ID". - * `social_auth_apple_team` to your "Team ID". -* Put the private key file you got from apple at the path + - `social_auth_apple_key` to your "Key ID". + - `social_auth_apple_team` to your "Team ID". +- Put the private key file you got from apple at the path `zproject/dev_apple.key`. ### SAML -* Sign up for a [developer Okta account](https://developer.okta.com/). -* Set up SAML authentication by following +- Sign up for a [developer Okta account](https://developer.okta.com/). +- Set up SAML authentication by following [Okta's documentation](https://developer.okta.com/docs/guides/saml-application-setup/overview/). Specify: - * `http://localhost:9991/complete/saml/` for the "Single sign on URL"`. - * `http://localhost:9991` for the "Audience URI (SP Entity ID)". - * Skip "Default RelayState". - * Skip "Name ID format". - * Set 'Email` for "Application username format". - * Provide "Attribute statements" of `email` to `user.email`, + - `http://localhost:9991/complete/saml/` for the "Single sign on URL"`. + - `http://localhost:9991` for the "Audience URI (SP Entity ID)". + - Skip "Default RelayState". + - Skip "Name ID format". + - Set 'Email` for "Application username format". + - Provide "Attribute statements" of `email` to `user.email`, `first_name` to `user.firstName`, and `last_name` to `user.lastName`. -* Assign at least one account in the "Assignments" tab. You'll use it for +- Assign at least one account in the "Assignments" tab. You'll use it for signing up / logging in to Zulip. -* Visit the big "Setup instructions" button on the "Sign on" tab. -* Edit `zproject/dev-secrets.conf` to add the two values provided: - * Set `saml_url = http...` from "Identity Provider Single Sign-On +- Visit the big "Setup instructions" button on the "Sign on" tab. +- Edit `zproject/dev-secrets.conf` to add the two values provided: + - Set `saml_url = http...` from "Identity Provider Single Sign-On URL". - * Set `saml_entity_id = http://...` from "Identity Provider Issuer". - * Download the certificate and put it at the path `zproject/dev_saml.cert`. -* Now you should have working SAML authentication! -* You can sign up to the target realm with the account that you've "assigned" + - Set `saml_entity_id = http://...` from "Identity Provider Issuer". + - Download the certificate and put it at the path `zproject/dev_saml.cert`. +- Now you should have working SAML authentication! +- You can sign up to the target realm with the account that you've "assigned" in the previous steps (if the account's email address is allowed in the realm, so you may have to change the realm settings to allow the appropriate email domain) and then you'll be able to log in freely. Alternatively, you can create an account @@ -188,11 +188,11 @@ We also have configured `AUTH_LDAP_USER_ATTR_MAP` in `zproject/dev_settings.py` to sync several of those fields. For example: -* Modes `a` and `b` will set the user's avatar on account creation and +- Modes `a` and `b` will set the user's avatar on account creation and update it when `manage.py sync_ldap_user_data` is run. -* Mode `b` is configured to automatically have the `birthday` and +- Mode `b` is configured to automatically have the `birthday` and `Phone number` custom profile fields populated/synced. -* Mode `a` is configured to deactivate/reactivate users whose accounts +- Mode `a` is configured to deactivate/reactivate users whose accounts are disabled in LDAP when `manage.py sync_ldap_user_data` is run. (Note that you'll likely need to edit `zerver/lib/dev_ldap_directory.py` to ensure there are some accounts diff --git a/docs/development/overview.md b/docs/development/overview.md index 3db8c727ac..d27be67f5e 100644 --- a/docs/development/overview.md +++ b/docs/development/overview.md @@ -28,8 +28,8 @@ performs well. Zulip also supports a wide range of ways to install the Zulip development environment: -* On Linux platforms, you can **[install directly][install-direct]**. -* On Windows, you can **[install directly][install-via-wsl]** via WSL 2. +- On Linux platforms, you can **[install directly][install-direct]**. +- On Windows, you can **[install directly][install-via-wsl]** via WSL 2. ## Slow internet connections @@ -65,8 +65,8 @@ need to. Once you've installed the Zulip development environment, you'll want to read these documents to learn how to use it: -* [Using the development environment][using-dev-env] -* [Testing][testing] (and [Configuring CI][ci]) +- [Using the development environment][using-dev-env] +- [Testing][testing] (and [Configuring CI][ci]) And if you've set up the Zulip development environment on a remote machine, take a look at our tips for diff --git a/docs/development/remote.md b/docs/development/remote.md index 458fdec5b1..b5751cf929 100644 --- a/docs/development/remote.md +++ b/docs/development/remote.md @@ -16,8 +16,8 @@ need to. The best way to connect to your server is using the command line tool `ssh`. -* On macOS and Linux/UNIX, `ssh` is a part of Terminal. -* On Windows, `ssh` comes with [Bash for Git][git-bash]. +- On macOS and Linux/UNIX, `ssh` is a part of Terminal. +- On Windows, `ssh` comes with [Bash for Git][git-bash]. Open *Terminal* or *Bash for Git*, and connect with the following: @@ -37,13 +37,13 @@ to the next section. You can create a new user with sudo privileges by running the following commands as root: -* You can create a `zulipdev` user by running the command +- You can create a `zulipdev` user by running the command `adduser zulipdev`. Run through the prompts to assign a password and user information. (You can pick any username you like for this user account.) -* You can add the user to the sudo group by running the command +- You can add the user to the sudo group by running the command `usermod -aG sudo zulipdev`. -* Finally, you can switch to the user by running the command +- Finally, you can switch to the user by running the command `su - zulipdev` (or just log in to that user using `ssh`). ## Setting up the development environment @@ -112,10 +112,10 @@ environment][rtd-using-dev-env]. To see changes on your remote development server, you need to do one of the following: -* [Edit locally](#editing-locally): Clone Zulip code to your computer and +- [Edit locally](#editing-locally): Clone Zulip code to your computer and then use your favorite editor to make changes. When you want to see changes on your remote Zulip development instance, sync with Git. -* [Edit remotely](#editing-remotely): Edit code directly on your remote +- [Edit remotely](#editing-remotely): Edit code directly on your remote Zulip development instance using a [Web-based IDE](#web-based-ide) (recommended for beginners) or a [command line editor](#command-line-editors), or a [desktop IDE](#desktop-gui-editors) using a plugin to sync your @@ -126,12 +126,12 @@ To see changes on your remote development server, you need to do one of the foll If you want to edit code locally install your favorite text editor. If you don't have a favorite, here are some suggestions: -* [atom](https://atom.io/) -* [emacs](https://www.gnu.org/software/emacs/) -* [vim](https://www.vim.org/) -* [spacemacs](https://github.com/syl20bnr/spacemacs) -* [sublime](https://www.sublimetext.com/) -* [PyCharm](https://www.jetbrains.com/pycharm/) +- [atom](https://atom.io/) +- [emacs](https://www.gnu.org/software/emacs/) +- [vim](https://www.vim.org/) +- [spacemacs](https://github.com/syl20bnr/spacemacs) +- [sublime](https://www.sublimetext.com/) +- [PyCharm](https://www.jetbrains.com/pycharm/) Next, follow our [Git and GitHub guide](../git/index.md) to clone and configure your fork of zulip on your local computer. @@ -172,10 +172,10 @@ from your remote development instance: There are a few good ways to edit code in your remote development environment: -* With a command-line editor like vim or emacs run over SSH. -* With a desktop GUI editor like VS Code or Atom and a plugin for +- With a command-line editor like vim or emacs run over SSH. +- With a desktop GUI editor like VS Code or Atom and a plugin for syncing your changes to the remote server. -* With a web-based IDE like CodeAnywhere. +- With a web-based IDE like CodeAnywhere. We document these options below; we recommend using whatever editor you prefer for development in general. @@ -226,20 +226,20 @@ a command line text editor on the remote machine. Two editors often available by default on Linux systems are: -* **Nano**: A very simple, beginner-friendly editor. However, it lacks a lot of +- **Nano**: A very simple, beginner-friendly editor. However, it lacks a lot of features useful for programming, such as syntax highlighting, so we only recommended it for quick edits to things like configuration files. Launch by running command `nano `. Exit by pressing *Ctrl-X*. -* **[Vim](https://www.vim.org/)**: A very powerful editor that can take a while +- **[Vim](https://www.vim.org/)**: A very powerful editor that can take a while to learn. Launch by running `vim `. Quit Vim by pressing *Esc*, typing `:q`, and then pressing *Enter*. Vim comes with a program to learn it called `vimtutor` (just run that command to start it). Other options include: -* [emacs](https://www.gnu.org/software/emacs/) -* [spacemacs](https://github.com/syl20bnr/spacemacs) +- [emacs](https://www.gnu.org/software/emacs/) +- [spacemacs](https://github.com/syl20bnr/spacemacs) ##### Web-based IDE @@ -264,9 +264,9 @@ Now your workspace should look similar this: Next, read the following to learn more about developing for Zulip: -* [Git & GitHub guide][rtd-git-guide] -* [Using the development environment][rtd-using-dev-env] -* [Testing][rtd-testing] +- [Git & GitHub guide][rtd-git-guide] +- [Using the development environment][rtd-using-dev-env] +- [Testing][rtd-testing] [install-direct]: ../development/setup-advanced.html#installing-directly-on-ubuntu-debian-centos-or-fedora [install-vagrant]: ../development/setup-vagrant.md diff --git a/docs/development/request-remote.md b/docs/development/request-remote.md index 169c69e53b..523ce0bf5c 100644 --- a/docs/development/request-remote.md +++ b/docs/development/request-remote.md @@ -70,14 +70,14 @@ Once your remote dev instance is ready: Once you've confirmed you can connect to your remote server, take a look at: -* [developing remotely](../development/remote.md) for tips on using the remote dev +- [developing remotely](../development/remote.md) for tips on using the remote dev instance, and -* our [Git & GitHub guide](../git/index.md) to learn how to use Git with Zulip. +- our [Git & GitHub guide](../git/index.md) to learn how to use Git with Zulip. Next, read the following to learn more about developing for Zulip: -* [Using the development environment](../development/using.md) -* [Testing](../testing/testing.md) +- [Using the development environment](../development/using.md) +- [Testing](../testing/testing.md) [github-join]: https://github.com/join [github-help-add-ssh-key]: https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account diff --git a/docs/development/setup-advanced.md b/docs/development/setup-advanced.md index b58ca7fe8e..863e1fdfd6 100644 --- a/docs/development/setup-advanced.md +++ b/docs/development/setup-advanced.md @@ -2,22 +2,22 @@ Contents: -* [Installing directly on Ubuntu, Debian, CentOS, or Fedora](#installing-directly-on-ubuntu-debian-centos-or-fedora) -* [Installing directly on Windows 10 with WSL 2](#installing-directly-on-windows-10-with-wsl-2) -* [Using the Vagrant Hyper-V provider on Windows](#using-the-vagrant-hyper-v-provider-on-windows-beta) -* [Newer versions of supported platforms](#newer-versions-of-supported-platforms) -* [Installing directly on cloud9](#installing-on-cloud9) +- [Installing directly on Ubuntu, Debian, CentOS, or Fedora](#installing-directly-on-ubuntu-debian-centos-or-fedora) +- [Installing directly on Windows 10 with WSL 2](#installing-directly-on-windows-10-with-wsl-2) +- [Using the Vagrant Hyper-V provider on Windows](#using-the-vagrant-hyper-v-provider-on-windows-beta) +- [Newer versions of supported platforms](#newer-versions-of-supported-platforms) +- [Installing directly on cloud9](#installing-on-cloud9) ## Installing directly on Ubuntu, Debian, CentOS, or Fedora If you'd like to install a Zulip development environment on a computer that's running one of: -* Ubuntu 20.04 Focal, 18.04 Bionic -* Debian 10 Buster, 11 Bullseye (beta) -* CentOS 7 (beta) -* Fedora 33 (beta) -* RHEL 7 (beta) +- Ubuntu 20.04 Focal, 18.04 Bionic +- Debian 10 Buster, 11 Bullseye (beta) +- CentOS 7 (beta) +- Fedora 33 (beta) +- RHEL 7 (beta) You can just run the Zulip provision script on your machine. @@ -294,16 +294,16 @@ This section documents how to set up the Zulip development environment in a Cloud9 workspace. If you don't have an existing Cloud9 account, you can sign up [here](https://aws.amazon.com/cloud9/). -* Create a Workspace, and select the blank template. -* Resize the workspace to be 1GB of memory and 4GB of disk +- Create a Workspace, and select the blank template. +- Resize the workspace to be 1GB of memory and 4GB of disk space. (This is under free limit for both the old Cloud9 and the AWS Free Tier). -* Clone the zulip repo: +- Clone the zulip repo: `git clone --config pull.rebase https://github.com//zulip.git` -* Restart rabbitmq-server since its broken on Cloud9: +- Restart rabbitmq-server since its broken on Cloud9: `sudo service rabbitmq-server restart`. -* And run provision `cd zulip && ./tools/provision`, once this is done. -* Activate the Zulip virtual environment by +- And run provision `cd zulip && ./tools/provision`, once this is done. +- Activate the Zulip virtual environment by `source /srv/zulip-py3-venv/bin/activate` or by opening a new terminal. diff --git a/docs/development/setup-vagrant.md b/docs/development/setup-vagrant.md index 3f92c0a97c..d631410d9e 100644 --- a/docs/development/setup-vagrant.md +++ b/docs/development/setup-vagrant.md @@ -10,16 +10,16 @@ or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Contents: -* [Requirements](#requirements) -* [Step 0: Set up Git & GitHub](#step-0-set-up-git-github) -* [Step 1: Install prerequisites](#step-1-install-prerequisites) -* [Step 2: Get Zulip code](#step-2-get-zulip-code) -* [Step 3: Start the development environment](#step-3-start-the-development-environment) -* [Step 4: Developing](#step-4-developing) -* [Troubleshooting and common errors](#troubleshooting-and-common-errors) -* [Specifying an Ubuntu mirror](#specifying-an-ubuntu-mirror) -* [Specifying a proxy](#specifying-a-proxy) -* [Customizing CPU and RAM allocation](#customizing-cpu-and-ram-allocation) +- [Requirements](#requirements) +- [Step 0: Set up Git & GitHub](#step-0-set-up-git-github) +- [Step 1: Install prerequisites](#step-1-install-prerequisites) +- [Step 2: Get Zulip code](#step-2-get-zulip-code) +- [Step 3: Start the development environment](#step-3-start-the-development-environment) +- [Step 4: Developing](#step-4-developing) +- [Troubleshooting and common errors](#troubleshooting-and-common-errors) +- [Specifying an Ubuntu mirror](#specifying-an-ubuntu-mirror) +- [Specifying a proxy](#specifying-a-proxy) +- [Customizing CPU and RAM allocation](#customizing-cpu-and-ram-allocation) **If you encounter errors installing the Zulip development environment,** check [troubleshooting and common @@ -32,10 +32,10 @@ server](../contributing/chat-zulip-org.md) for real-time help or When reporting your issue, please include the following information: -* host operating system -* installation method (Vagrant or direct) -* whether or not you are using a proxy -* a copy of Zulip's `vagrant` provisioning logs, available in +- host operating system +- installation method (Vagrant or direct) +- whether or not you are using a proxy +- a copy of Zulip's `vagrant` provisioning logs, available in `/var/log/provision.log` on your virtual machine ### Requirements @@ -73,10 +73,10 @@ GitHub account using Jump to: -* [macOS](#macos) -* [Ubuntu](#ubuntu) -* [Debian](#debian) -* [Windows](#windows-10) +- [macOS](#macos) +- [Ubuntu](#ubuntu) +- [Debian](#debian) +- [Windows](#windows-10) #### macOS @@ -534,9 +534,9 @@ $ ./tools/run-dev.py Next, read the following to learn more about developing for Zulip: -* [Git & GitHub guide][rtd-git-guide] -* [Using the development environment][rtd-using-dev-env] -* [Testing][rtd-testing] (and [Configuring CI][ci] to +- [Git & GitHub guide][rtd-git-guide] +- [Using the development environment][rtd-using-dev-env] +- [Testing][rtd-testing] (and [Configuring CI][ci] to run the full test suite against any branches you push to your fork, which can help you optimize your development workflow). @@ -550,16 +550,16 @@ equivalently `vagrant provision` from outside). If these solutions aren't working for you or you encounter an issue not documented below, there are a few ways to get further help: -* Ask in [#provision help](https://chat.zulip.org/#narrow/stream/21-provision-help) +- Ask in [#provision help](https://chat.zulip.org/#narrow/stream/21-provision-help) in the [Zulip development community server](../contributing/chat-zulip-org.md). -* [File an issue](https://github.com/zulip/zulip/issues). +- [File an issue](https://github.com/zulip/zulip/issues). When reporting your issue, please include the following information: -* host operating system -* installation method (Vagrant or direct) -* whether or not you are using a proxy -* a copy of Zulip's `vagrant` provisioning logs, available in +- host operating system +- installation method (Vagrant or direct) +- whether or not you are using a proxy +- a copy of Zulip's `vagrant` provisioning logs, available in `/var/log/provision.log` on your virtual machine. If you choose to post just the error output, please include the **beginning of the error output**, not just the last few lines. @@ -790,8 +790,8 @@ by rebooting the guest via `vagrant halt; vagrant up`. The `vagrant up` command basically does the following: -* Downloads an Ubuntu image and starts it using a Vagrant provider. -* Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs +- Downloads an Ubuntu image and starts it using a Vagrant provider. +- Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs `tools/provision`, which has a lot of subcommands that are executed via Python's `subprocess` module. These errors mean that one of those subcommands failed. diff --git a/docs/development/using.md b/docs/development/using.md index 5f840d30d0..e7b75590d3 100644 --- a/docs/development/using.md +++ b/docs/development/using.md @@ -13,80 +13,80 @@ the development environment][authentication-dev-server]. ## Common -* Zulip's `main` branch moves quickly, and you should rebase +- Zulip's `main` branch moves quickly, and you should rebase constantly with e.g. `git fetch upstream; git rebase upstream/main` to avoid developing on an old version of the Zulip codebase (leading to unnecessary merge conflicts). -* Remember to run `tools/provision` to update your development +- Remember to run `tools/provision` to update your development environment after switching branches; it will run in under a second if no changes are required. -* After making changes, you'll often want to run the +- After making changes, you'll often want to run the [linters](../testing/linters.md) and relevant [test suites](../testing/testing.md). Consider using our [Git pre-commit hook](../git/zulip-tools.html#set-up-git-repo-script) to automatically lint whenever you make a commit. -* All of our test suites are designed to support quickly testing just +- All of our test suites are designed to support quickly testing just a single file or test case, which you should take advantage of to save time. -* Many useful development tools, including tools for rebuilding the +- Many useful development tools, including tools for rebuilding the database with different test data, are documented in-app at `https://localhost:9991/devtools`. -* If you want to restore your development environment's database to a +- If you want to restore your development environment's database to a pristine state, you can use `./tools/rebuild-dev-database`. ## Server -* For changes that don't affect the database model, the Zulip +- For changes that don't affect the database model, the Zulip development environment will automatically detect changes and restart: - * The main Django/Tornado server processes are run on top of + - The main Django/Tornado server processes are run on top of Django's [manage.py runserver][django-runserver], which will automatically restart them when you save changes to Python code they use. You can watch this happen in the `run-dev.py` console to make sure the backend has reloaded. - * The Python queue workers will also automatically restart when you + - The Python queue workers will also automatically restart when you save changes, as long as they haven't crashed (which can happen if they reloaded into a version with a syntax error). -* If you change the database schema (`zerver/models.py`), you'll need +- If you change the database schema (`zerver/models.py`), you'll need to use the [Django migrations process](../subsystems/schema-migrations.md); see also the [new feature tutorial][new-feature-tutorial] for an example. -* While testing server changes, it's helpful to watch the `run-dev.py` +- While testing server changes, it's helpful to watch the `run-dev.py` console output, which will show tracebacks for any 500 errors your Zulip development server encounters (which are probably caused by bugs in your code). -* To manually query Zulip's database interactively, use +- To manually query Zulip's database interactively, use `./manage.py shell` or `manage.py dbshell`. -* The database(s) used for the automated tests are independent from +- The database(s) used for the automated tests are independent from the one you use for manual testing in the UI, so changes you make to the database manually will never affect the automated tests. ## Web -* Once the development server (`run-dev.py`) is running, you can visit +- Once the development server (`run-dev.py`) is running, you can visit in your browser. -* By default, the development server homepage just shows a list of the +- By default, the development server homepage just shows a list of the users that exist on the server and you can log in as any of them by just clicking on a user. - * This setup saves time for the common case where you want to test + - This setup saves time for the common case where you want to test something other than the login process. - * You can test the login or registration process by clicking the + - You can test the login or registration process by clicking the links for the normal login page. -* Most changes will take effect automatically. Details: - * If you change CSS files, your changes will appear immediately via +- Most changes will take effect automatically. Details: + - If you change CSS files, your changes will appear immediately via webpack hot module replacement. - * If you change JavaScript code (`static/js`) or Handlebars + - If you change JavaScript code (`static/js`) or Handlebars templates (`static/templates`), the browser window will be reloaded automatically. - * For Jinja2 backend templates (`templates/*`), you'll need to reload + - For Jinja2 backend templates (`templates/*`), you'll need to reload the browser window to see your changes. -* Any JavaScript exceptions encountered while using the webapp in a +- Any JavaScript exceptions encountered while using the webapp in a development environment will be displayed as a large notice, so you don't need to watch the JavaScript console for exceptions. -* Both Chrome and Firefox have great debuggers, inspectors, and +- Both Chrome and Firefox have great debuggers, inspectors, and profilers in their built-in developer tools. -* `debug.js` has some occasionally useful JavaScript profiling code. +- `debug.js` has some occasionally useful JavaScript profiling code. ## Mobile diff --git a/docs/documentation/api.md b/docs/documentation/api.md index 7f9e014343..c63d75fbfe 100644 --- a/docs/documentation/api.md +++ b/docs/documentation/api.md @@ -26,30 +26,30 @@ the validation Zulip has today. Our API documentation is defined by a few sets of files: -* Most data describing API endpoints and examples is stored in our +- Most data describing API endpoints and examples is stored in our [OpenAPI configuration](../documentation/openapi.md) at `zerver/openapi/zulip.yaml`. -* The top-level templates live under `templates/zerver/api/*`, and are +- The top-level templates live under `templates/zerver/api/*`, and are written using the Markdown framework that powers our [user docs](../documentation/user.md), with some special extensions for rendering nice code blocks and example responses. We expect to eventually remove most of these files where it is possible to fully generate the documentation from the OpenAPI files. -* The text for the Python examples comes from a test suite for the +- The text for the Python examples comes from a test suite for the Python API documentation (`zerver/openapi/python_examples.py`; run via `tools/test-api`). The `generate_code_example` macro will magically read content from that test suite and render it as the code example. This structure ensures that Zulip's API documentation is robust to a wide range of possible typos and other bugs in the API documentation. -* The JavaScript examples are similarly generated and tested using +- The JavaScript examples are similarly generated and tested using `zerver/openapi/javascript_examples.js`. -* The cURL examples are generated and tested using +- The cURL examples are generated and tested using `zerver/openapi/curl_param_value_generators.py`. -* The REST API index +- The REST API index (`templates/zerver/help/include/rest-endpoints.md`) in the broader /api left sidebar (`templates/zerver/api/sidebar_index.md`). -* We have an extensive set of tests designed to validate that the data +- We have an extensive set of tests designed to validate that the data in this file is correct, `zerver/tests/test_openapi.py` compares every endpoint's accepted parameters in `views` code with those declared in `zulip.yaml`. And [backend test @@ -74,10 +74,10 @@ We highly recommend looking at those resources while reading this page. If you look at the documentation for existing endpoints, you'll notice that a typical endpoint's documentation is divided into four sections: -* The top-level **Description** -* **Usage examples** -* **Arguments** -* **Responses** +- The top-level **Description** +- **Usage examples** +- **Arguments** +- **Responses** The rest of this guide describes how each of these sections works. @@ -215,11 +215,11 @@ above. declared using `REQ`. You can check your formatting using these helpful tools. - * `tools/check-openapi` will verify the syntax of `zerver/openapi/zulip.yaml`. - * `tools/test-backend zerver/tests/test_openapi.py`; this test compares + - `tools/check-openapi` will verify the syntax of `zerver/openapi/zulip.yaml`. + - `tools/test-backend zerver/tests/test_openapi.py`; this test compares your documentation against the code and can find many common mistakes in how arguments are declared. - * `test-backend`: The full Zulip backend test suite will fail if + - `test-backend`: The full Zulip backend test suite will fail if any actual API responses generated by the tests don't match your defined OpenAPI schema. Use `test-backend --rerun` for a fast edit/refresh cycle when debugging. @@ -293,14 +293,14 @@ Given that our documentation is written in large part using the OpenAPI format, why maintain a custom Markdown system for displaying it? There's several major benefits to this system: -* It is extremely common for API documentation to become out of date +- It is extremely common for API documentation to become out of date as an API evolves; this automated testing system helps make it possible for Zulip to maintain accurate documentation without a lot of manual management. -* Every Zulip server can host correct API documentation for its +- Every Zulip server can host correct API documentation for its version, with the key variables (like the Zulip server URL) already pre-substituted for the user. -* We're able to share implementation language and visual styling with +- We're able to share implementation language and visual styling with our Help Center, which is especially useful for the extensive non-REST API documentation pages (e.g. our bot framework). diff --git a/docs/documentation/integrations.md b/docs/documentation/integrations.md index d0cf4e1c4f..d7bf9295d0 100644 --- a/docs/documentation/integrations.md +++ b/docs/documentation/integrations.md @@ -10,14 +10,14 @@ integrations). Usually, this involves a few steps: -* Add text explaining all of the steps required to set up the +- Add text explaining all of the steps required to set up the integration, including what URLs to use, etc. See [Writing guidelines](#writing-guidelines) for detailed writing guidelines. Zulip's pre-defined Markdown macros can be used for some of these steps. See [Markdown macros](#markdown-macros) for further details. -* Make sure you've added your integration to +- Make sure you've added your integration to `zerver/lib/integrations.py` in both the `WEBHOOK_INTEGRATIONS` section (or `INTEGRATIONS` if not a webhook), and the `DOC_SCREENSHOT_CONFIG` sections. These registries configure your @@ -26,7 +26,7 @@ Usually, this involves a few steps: message (which is important for the screenshots to be updated as Zulip's design changes). -* You'll need to add an SVG graphic +- You'll need to add an SVG graphic of your integration's logo under the `static/images/integrations/logos/.svg`, where `` is the name of the integration, all in lower case; you can usually find them in the @@ -40,7 +40,7 @@ Usually, this involves a few steps: If you cannot find an SVG graphic of the logo, please find and include a PNG image of the logo instead. -* Finally, generate a message sent by the integration and take a screenshot of +- Finally, generate a message sent by the integration and take a screenshot of the message to provide an example message in the documentation. If your new integration is an incoming webhook integration, you can generate @@ -71,12 +71,12 @@ always create a new macro by adding a new file to that folder. Here are a few common macros used to document Zulip's integrations: -* `{!create-stream.md!}` macro - Recommends that users create a dedicated +- `{!create-stream.md!}` macro - Recommends that users create a dedicated stream for a given integration. Usually the first step is setting up an integration or incoming webhook. For an example rendering, see **Step 1** of [the docs for Zulip's GitHub integration][github-integration]. -* `{!create-bot-construct-url-indented.md!}` macro - Instructs users to create a bot +- `{!create-bot-construct-url-indented.md!}` macro - Instructs users to create a bot for a given integration and construct a webhook URL using the bot API key and stream name. The URL is generated automatically for every incoming webhook by using attributes in the `WebhookIntegration` class in @@ -91,49 +91,49 @@ Here are a few common macros used to document Zulip's integrations: deployed on. If special configuration is required to set the `SITE` variable, you should document that too. -* `{!append-stream-name.md!}` macro - Recommends appending `&stream=stream_name` +- `{!append-stream-name.md!}` macro - Recommends appending `&stream=stream_name` to a URL in cases where supplying a stream name in the URL is optional. Supplying a stream name is optional for most Zulip integrations. If you use `{!create-bot-construct-url-indented.md!}`, this macro need not be used. -* `{!append-topic.md!}` macro - Recommends appending `&topic=my_topic` to a URL +- `{!append-topic.md!}` macro - Recommends appending `&topic=my_topic` to a URL to supply a custom topic for webhook notification messages. Supplying a custom topic is optional for most Zulip integrations. If you use `{!create-bot-construct-url-indented.md!}`, this macro need not be used. -* `{!congrats.md!}` macro - Inserts congratulatory lines signifying the +- `{!congrats.md!}` macro - Inserts congratulatory lines signifying the successful setup of a given integration. This macro is usually used at the end of the documentation, right before the sample message screenshot. For an example rendering, see the end of [the docs for Zulip's GitHub integration][github-integration]. -* `{!download-python-bindings.md!}` macro - Links to Zulip's +- `{!download-python-bindings.md!}` macro - Links to Zulip's [API page](https://zulip.com/api/) to download and install Zulip's API bindings. This macro is usually used in non-webhook integration docs under `templates/zerver/integrations/.md`. For an example rendering, see **Step 2** of [the docs for Zulip's Codebase integration][codebase]. -* `{!change-zulip-config-file.md!}` macro - Instructs users to create a bot and +- `{!change-zulip-config-file.md!}` macro - Instructs users to create a bot and specify said bot's credentials in the config file for a given non-webhook integration. This macro is usually used in non-webhook integration docs under `templates/zerver/integrations/.md`. For an example rendering, see **Step 4** of [the docs for Zulip's Codebase integration][codebase]. -* `{!git-append-branches.md!}` and `{!git-webhook-url-with-branches.md!}` - +- `{!git-append-branches.md!}` and `{!git-webhook-url-with-branches.md!}` - These two macros explain how to specify a list of branches in the webhook URL to filter notifications in our Git-related webhooks. For an example rendering, see the last paragraph of **Step 2** in [the docs for Zulip's GitHub integration][github-integration]. -* `{!webhook-url.md!}` - Used internally by `{!create-bot-construct-url-indented.md!}` +- `{!webhook-url.md!}` - Used internally by `{!create-bot-construct-url-indented.md!}` to generate the webhook URL. -* `{!zulip-config.md!}` - Used internally by `{!change-zulip-config-file.md!}` +- `{!zulip-config.md!}` - Used internally by `{!change-zulip-config-file.md!}` to specify the lines in the config file for a non-webhook integration. -* `{!webhook-url-with-bot-email.md!}` - Used in certain non-webhook integrations +- `{!webhook-url-with-bot-email.md!}` - Used in certain non-webhook integrations to generate URLs of the form: ```text diff --git a/docs/documentation/openapi.md b/docs/documentation/openapi.md index 701b1e15de..22a195a4ca 100644 --- a/docs/documentation/openapi.md +++ b/docs/documentation/openapi.md @@ -144,7 +144,7 @@ You can find more examples, including GET requests and nested objects, in We're collecting decisions we've made on how our Swagger YAML files should be organized here: -* Use shared definitions and YAML anchors to avoid duplicating content +- Use shared definitions and YAML anchors to avoid duplicating content where possible. ## Tips for working with YAML: @@ -169,13 +169,13 @@ correct. ### Formatting help: -* Comments begin with a # character. +- Comments begin with a # character. -* Descriptions do not need to be in quotes, and may use common +- Descriptions do not need to be in quotes, and may use common Markdown format options like inline code \` (backtick) and `#` headings. -* A single `|` (pipe) character begins a multi-line description on the +- A single `|` (pipe) character begins a multi-line description on the next line. Single spaced lines (one newline at the end of each) are joined. Use an extra blank line for a paragraph break. We prefer to use this format for all descriptions because it doesn't require diff --git a/docs/documentation/overview.md b/docs/documentation/overview.md index 9fd89ae9ae..a35e0ee69b 100644 --- a/docs/documentation/overview.md +++ b/docs/documentation/overview.md @@ -2,26 +2,26 @@ Zulip has three major documentation systems: -* Developer and sysadmin documentation: Documentation for people +- Developer and sysadmin documentation: Documentation for people actually interacting with the Zulip codebase (either by developing it or installing it), and written in Markdown. -* Core website documentation: Complete webpages for complex topics, +- Core website documentation: Complete webpages for complex topics, written in HTML, JavaScript, and CSS (using the Django templating system). These roughly correspond to the documentation someone might look at when deciding whether to use Zulip. We don't expect to ever have more than about 10 pages written using this system. -* User-facing documentation: Our scalable system for documenting +- User-facing documentation: Our scalable system for documenting Zulip's huge collection of specific features without a lot of overhead or duplicated code/syntax, written in Markdown. We have several hundred pages written using this system. There are 3 branches of this documentation: - * User documentation (with a target audience of individual Zulip + - User documentation (with a target audience of individual Zulip users), - * Integrations documentation (with a target audience of IT folks + - Integrations documentation (with a target audience of IT folks setting up integrations), and - * API documentation (with a target audience of developers writing + - API documentation (with a target audience of developers writing code to extend Zulip). These three systems are documented in detail. @@ -123,12 +123,12 @@ details on how to contribute to this documentation. Zulip has several automated test suites that we run in CI and recommend running locally when making significant edits: -* `tools/lint` catches a number of common mistakes, and we highly +- `tools/lint` catches a number of common mistakes, and we highly recommend [using our linter pre-commit hook](../git/zulip-tools.html#set-up-git-repo-script). See the [main linter doc](../testing/linters.md) for more details. -* The ReadTheDocs docs are built and the links tested by +- The ReadTheDocs docs are built and the links tested by `tools/test-documentation`, which runs `build-docs` and then checks all the links. @@ -136,17 +136,17 @@ There's an exclude list for the link testing at this horrible path: `tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py`, which is relevant for flaky links. -* The API docs are tested by `tools/test-api`, which does some basic +- The API docs are tested by `tools/test-api`, which does some basic payload verification. Note that this test does not check for broken links (those are checked by `test-help-documentation`). -* `tools/test-help-documentation` checks `/help/`, `/api/`, +- `tools/test-help-documentation` checks `/help/`, `/api/`, `/integrations/`, and the core website ("portico") documentation for broken links. Note that the "portico" documentation check has a manually maintained whitelist of pages, so if you add a new page to this site, you will need to edit `PorticoDocumentationSpider` to add it. -* `tools/test-backend test_docs.py` tests various internal details of +- `tools/test-backend test_docs.py` tests various internal details of the variable substitution logic, as well as rendering. It's essential when editing the documentation framework, but not something you'll usually need to interact with when editing diff --git a/docs/documentation/user.md b/docs/documentation/user.md index 3fba5c31fb..e9847afc21 100644 --- a/docs/documentation/user.md +++ b/docs/documentation/user.md @@ -8,12 +8,12 @@ There are two types of documents: articles about specific features, and a handful of longer guides. The feature articles serve a few different purposes: -* Feature discovery, for someone browsing the `/help` page, and looking at +- Feature discovery, for someone browsing the `/help` page, and looking at the set of titles. -* Public documentation of our featureset, for someone googling "can zulip do .." -* Canned responses to support questions; if someone emails a Zulip admin +- Public documentation of our featureset, for someone googling "can zulip do .." +- Canned responses to support questions; if someone emails a Zulip admin asking "how do I change my name", they can reply with a link to the doc. -* Feature explanations for new Zulip users and admins, especially for +- Feature explanations for new Zulip users and admins, especially for organization settings. This system is designed to make writing and maintaining such documentation @@ -52,22 +52,22 @@ experience with. Tips for adding a new article: -* Find an existing article in the same section of the help documentation, +- Find an existing article in the same section of the help documentation, and copy the format, wording, style, etc as closely as you can. -* If the feature exists in other team chat products, check out their +- If the feature exists in other team chat products, check out their documentation for inspiration. -* Fewer words is better than more. Many Zulip users have English as a second +- Fewer words is better than more. Many Zulip users have English as a second language. -* Try to put yourself in the shoes of a new Zulip user. What would you want +- Try to put yourself in the shoes of a new Zulip user. What would you want to know? -* The goal of user-facing documentation is not to be comprehensive. The goal +- The goal of user-facing documentation is not to be comprehensive. The goal is to give the right bits of information for the intended audience. -* Real estate in the left sidebar is somewhat precious. Minor features +- Real estate in the left sidebar is somewhat precious. Minor features should rarely get their own article. An anti-pattern is trying to make up for bad UX by adding user @@ -98,17 +98,17 @@ allows .." rather than "we also allow ..". `You` is ok and used liberally. Zulip's Markdown processor allows you to include several special features in your documentation to help improve its readability: -* Since raw HTML is supported in Markdown, you can include arbitrary +- Since raw HTML is supported in Markdown, you can include arbitrary HTML/CSS in your documentation as needed. -* Code blocks allow you to highlight syntax, similar to Zulip's own Markdown. -* Anchor tags can be used to link to headers in other documents. -* [Images](#images) of Zulip UI can be added to documentation. -* Inline [icons](#icons) used to refer to features in the Zulip UI. -* You can utilize [macros](#macros) to limit repeated content in the +- Code blocks allow you to highlight syntax, similar to Zulip's own Markdown. +- Anchor tags can be used to link to headers in other documents. +- [Images](#images) of Zulip UI can be added to documentation. +- Inline [icons](#icons) used to refer to features in the Zulip UI. +- You can utilize [macros](#macros) to limit repeated content in the documentation. -* You can create special highlight warning blocks using +- You can create special highlight warning blocks using [tips and warnings](#tips-and-warnings). -* You can create tabs using [Markdown tab switcher](#tab-switcher). +- You can create tabs using [Markdown tab switcher](#tab-switcher). ### Images @@ -141,39 +141,39 @@ base class `icon-vector` and have dropped support for it. We now only support icons from [FontAwesome](https://fontawesome.com/v4.7.0/) (version 4.7.0) which make use of `fa` as a base class. -* cog () icon — +- cog () icon — `cog () icon` -* down chevron () icon — +- down chevron () icon — `down chevron () icon` -* eye () icon — +- eye () icon — `eye () icon` -* file () icon — +- file () icon — `file () icon` -* filled star () icon — +- filled star () icon — `filled star () icon` -* formatting () icon — +- formatting () icon — `formatting () icon` -* menu () icon — +- menu () icon — `menu () icon` -* overflow ( ) icon — +- overflow ( ) icon — `overflow ( ) icon` -* paperclip () icon — +- paperclip () icon — `paperclip () icon` -* pencil () icon — +- pencil () icon — `pencil () icon` -* pencil and paper () icon — +- pencil and paper () icon — `pencil and paper () icon` -* plus () icon — +- plus () icon — `plus () icon` -* smiley face () icon — +- smiley face () icon — `smiley face () icon` -* star () icon — +- star () icon — `star () icon` -* trash () icon — +- trash () icon — `trash () icon` -* video-camera () icon — +- video-camera () icon — `video-camera () icon` -* x () icon — +- x () icon — `x () icon` ### Macros @@ -186,22 +186,22 @@ The source for macros is the Markdown files under `templates/zerver/help/include` in the [main Zulip server repository](https://github.com/zulip/zulip). -* **Administrator only feature** `{!admin-only.md!}`: Notes that the feature +- **Administrator only feature** `{!admin-only.md!}`: Notes that the feature is only available to organization administrators. -* **Message actions** `{!message-actions.md!}`: First step to navigating to +- **Message actions** `{!message-actions.md!}`: First step to navigating to the on-hover message actions. -* **Message actions menu** `{!message-actions-menu.md!}`: Navigate to the +- **Message actions menu** `{!message-actions-menu.md!}`: Navigate to the message actions menu. -* **Save changes** `{!save-changes.md!}`: Save changes after modifying +- **Save changes** `{!save-changes.md!}`: Save changes after modifying organization settings. -* **Stream actions** `{!stream-actions.md!}`: Navigate to the stream actions +- **Stream actions** `{!stream-actions.md!}`: Navigate to the stream actions menu from the left sidebar. -* **Start composing** `{!start-composing.md!}`: Open the compose box. +- **Start composing** `{!start-composing.md!}`: Open the compose box. ### Tips and warnings diff --git a/docs/git/zulip-tools.md b/docs/git/zulip-tools.md index dbef75d7d6..95023dc5cc 100644 --- a/docs/git/zulip-tools.md +++ b/docs/git/zulip-tools.md @@ -123,13 +123,13 @@ are merging other users' commits into a Zulip repository. After doing changes, you can push a branch back to a pull request with e.g. `tools/push-to-pull-request 1234`. This is useful for a few things: -* Getting CI to run and enabling you to use the GitHub "Merge" buttons +- Getting CI to run and enabling you to use the GitHub "Merge" buttons to merge a PR after you make some corrections to a PR, without waiting for an extra round trip with the PR author. -* For commits that aren't ready to merge yet, communicating clearly +- For commits that aren't ready to merge yet, communicating clearly any changes you'd like to see happen that are easier for you to explain by just editing the code than in words. -* Saving a contributor from needing to duplicate any rebase work that +- Saving a contributor from needing to duplicate any rebase work that you did as part of integrating parts of the PR. You'll likely want to comment on the PR after doing so, to ensure that diff --git a/docs/overview/architecture-overview.md b/docs/overview/architecture-overview.md index a209f8c9a6..73b727cc42 100644 --- a/docs/overview/architecture-overview.md +++ b/docs/overview/architecture-overview.md @@ -196,12 +196,12 @@ perceived benefit for using Redis is usually to reduce memory consumption by running fewer services, and no such benefit would materialize: -* Our cache uses significant memory, but that memory usage would be +- Our cache uses significant memory, but that memory usage would be essentially the same with Redis as it is with memcached. -* All of these services have low minimum memory requirements, and in +- All of these services have low minimum memory requirements, and in fact our applications for Redis and RabbitMQ do not use significant memory even at scale. -* We would likely need to run multiple Redis services (with different +- We would likely need to run multiple Redis services (with different configurations) in order to ensure the pure LRU use case (memcached) doesn't push out data that we want to persist until expiry (Redis-based rate limiting) or until consumed (RabbitMQ-based @@ -271,35 +271,35 @@ in Zulip development conversations. In general, our goal is to minimize the set of terminology listed here by giving elements self-explanatory names. -* **bankruptcy**: When a user has been off Zulip for several days and +- **bankruptcy**: When a user has been off Zulip for several days and has hundreds of unread messages, they are prompted for whether they want to mark all their unread messages as read. This is called "declaring bankruptcy" (in reference to the concept in finance). -* **chevron**: A small downward-facing arrow next to a message's +- **chevron**: A small downward-facing arrow next to a message's timestamp, offering contextual options, e.g., "Reply", "Mute [this topic]", or "Link to this conversation". To avoid visual clutter, the chevron only appears in the web UI upon hover. -* **ellipsis**: A small vertical three dot icon (technically called +- **ellipsis**: A small vertical three dot icon (technically called as ellipsis-v), present in sidebars as a menu icon. It offers contextual options for global filters (All messages and Starred messages), stream filters and topics in left sidebar and users in right sidebar. To avoid visual clutter ellipsis only appears in the web UI upon hover. -* **huddle**: What the codebase calls a "group private message". +- **huddle**: What the codebase calls a "group private message". -* **message editing**: If the realm admin allows it, then after a user +- **message editing**: If the realm admin allows it, then after a user posts a message, the user has a few minutes to click "Edit" and change the content of their message. If they do, Zulip adds a marker such as "(EDITED)" at the top of the message, visible to anyone who can see the message. -* **realm**: What the codebase calls an "organization" in the UI. +- **realm**: What the codebase calls an "organization" in the UI. -* **recipient bar**: A visual indication of the context of a message +- **recipient bar**: A visual indication of the context of a message or group of messages, displaying the stream and topic or private message recipient list, at the top of a group of messages. A typical 1-line message to a new recipient shows to the user as @@ -308,7 +308,7 @@ self-explanatory names. the star and the chevron), and third the message content. The recipient bar is or contains hyperlinks to help the user narrow. -* **star**: Zulip allows a user to mark any message they can see, +- **star**: Zulip allows a user to mark any message they can see, public or private, as "starred". A user can easily access messages they've starred through the "Starred messages" link in the left sidebar, or use "is:starred" as a narrow or a search @@ -316,4 +316,4 @@ self-explanatory names. message is private; other users and realm admins don't know whether a message has been starred, or by whom. -* **subject**: What the codebase calls a "topic" in many places. +- **subject**: What the codebase calls a "topic" in many places. diff --git a/docs/overview/changelog.md b/docs/overview/changelog.md index 3f58cd4b7c..bea92e7e00 100644 --- a/docs/overview/changelog.md +++ b/docs/overview/changelog.md @@ -763,11 +763,11 @@ lose the setting and need to re-enable it. still work, but users should update the following setting names in their configuration as we will desupport the old names in a future release: - * In `/etc/zulip/zulip-secrets.conf`, `google_oauth2_client_secret` + - In `/etc/zulip/zulip-secrets.conf`, `google_oauth2_client_secret` is now called with `social_auth_google_secret`. - * In `/etc/zulip/settings.py`, `GOOGLE_OAUTH2_CLIENT_ID` should be + - In `/etc/zulip/settings.py`, `GOOGLE_OAUTH2_CLIENT_ID` should be replaced with `SOCIAL_AUTH_GOOGLE_KEY`. - * In `/etc/zulip/settings.py`, `GoogleMobileOauth2Backend` should + - In `/etc/zulip/settings.py`, `GoogleMobileOauth2Backend` should be replaced with called `GoogleAuthBackend`. - Installations using Zulip's LDAP integration without `LDAP_APPEND_DOMAIN` will need to configure two new settings telling @@ -1180,7 +1180,7 @@ Zulip installations; it has minimal changes for existing servers. #### Upgrade notes for 1.9.0 -* Zulip 1.9 contains a significant database migration that can take +- Zulip 1.9 contains a significant database migration that can take several minutes to run. The upgrade process automatically minimizes disruption by running this migration first, before beginning the user-facing downtime. However, if you'd like to watch the downtime @@ -1568,7 +1568,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1. #### Upgrade notes for 1.7.0 -* Zulip 1.7 contains some significant database migrations that can +- Zulip 1.7 contains some significant database migrations that can take several minutes to run. The upgrade process automatically minimizes disruption by running these first, before beginning the user-facing downtime. However, if you'd like to watch the downtime @@ -1577,7 +1577,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1. as the usual trick of doing an apt upgrade first. -* We've removed support for an uncommon legacy deployment model where +- We've removed support for an uncommon legacy deployment model where a Zulip server served multiple organizations on the same domain. Installs with multiple organizations now require each organization to have its own subdomain. @@ -1587,7 +1587,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1. that hosts multiple organizations, you'll want to read [our guide on multiple organizations](../production/multiple-organizations.md). -* We simplified the configuration for our password strength checker to +- We simplified the configuration for our password strength checker to be much more intuitive. If you were using the `PASSWORD_MIN_ZXCVBN_QUALITY` setting, [it has been replaced](https://github.com/zulip/zulip/commit/a116303604e362796afa54b5d923ea5312b2ea23) by @@ -1741,91 +1741,91 @@ Zulip apps. #### Full feature changelog -* Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and +- Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and WordPress webhook integrations. -* Added LaTeX support to the Markdown processor. -* Added support for filtering branches to all Git integrations. -* Added read-only access to organization-level settings for all users. -* Added UI for managing muted topics and uploaded files. -* Added UI for displaying message edit history. -* Added support for various features needed by new mobile app. -* Added deep links for settings/subscriptions interfaces. -* Added an animation when messages are edited. -* Added support for registration with GitHub auth (not just login). -* Added tracking of uploaded file quotas. -* Added option to display emoji as their alt codes. -* Added new audit log table, to eventually support an auditing UI. -* Added several new permissions-related organization settings. -* Added new endpoint for fetching presence data, useful in employee directories. -* Added typeahead for language for syntax highlighting in code blocks. -* Added support for basic Markdown in stream descriptions. -* Added email notifications on new Zulip logins. -* Added security hardening before serving uploaded files. -* Added new PRIVACY_POLICY setting to provide a Markdown privacy policy. -* Added an icon to distinguish bot users as message senders. -* Added a command-line Slack importer tool using the API. -* Added new announcement notifications on stream creation. -* Added support for some newer Unicode emoji code points. -* Added support for users deleting realm emoji they themselves uploaded. -* Added support for organization administrators deleting messages. -* Extended data available to mobile apps to cover the entire API. -* Redesigned bots UI. Now can change owners and reactivate bots. -* Redesigned the visuals of code blocks to be prettier. -* Changed right sidebar presence UI to only show recently active users +- Added LaTeX support to the Markdown processor. +- Added support for filtering branches to all Git integrations. +- Added read-only access to organization-level settings for all users. +- Added UI for managing muted topics and uploaded files. +- Added UI for displaying message edit history. +- Added support for various features needed by new mobile app. +- Added deep links for settings/subscriptions interfaces. +- Added an animation when messages are edited. +- Added support for registration with GitHub auth (not just login). +- Added tracking of uploaded file quotas. +- Added option to display emoji as their alt codes. +- Added new audit log table, to eventually support an auditing UI. +- Added several new permissions-related organization settings. +- Added new endpoint for fetching presence data, useful in employee directories. +- Added typeahead for language for syntax highlighting in code blocks. +- Added support for basic Markdown in stream descriptions. +- Added email notifications on new Zulip logins. +- Added security hardening before serving uploaded files. +- Added new PRIVACY_POLICY setting to provide a Markdown privacy policy. +- Added an icon to distinguish bot users as message senders. +- Added a command-line Slack importer tool using the API. +- Added new announcement notifications on stream creation. +- Added support for some newer Unicode emoji code points. +- Added support for users deleting realm emoji they themselves uploaded. +- Added support for organization administrators deleting messages. +- Extended data available to mobile apps to cover the entire API. +- Redesigned bots UI. Now can change owners and reactivate bots. +- Redesigned the visuals of code blocks to be prettier. +- Changed right sidebar presence UI to only show recently active users in large organizations. This has a huge performance benefit. -* Changed color for private messages to look better. -* Converted realm emoji to be uploaded, not links, for better robustness. -* Switched the default password hasher for new passwords to Argon2. -* Increased the paragraph spacing, making multi-paragraph. -* Improved formatting of all Git integrations. -* Improved the UI of the /stats analytics pages. -* Improved search typeahead to support group private messages. -* Improved logic for when the compose box should open/close. -* Improved lightbox to support scrolling through images. -* Improved Markdown support for bulleted lists. -* Improved copy-to-clipboard support in various places. -* Improved subject lines of missed message emails. -* Improved handling of users trying to log in with OAuth without an account. -* Improved UI of off-the-Internet errors to not be hidden in narrow windows. -* Improved rate-limiting errors to be more easily machine-readable. -* Parallelized the backend test suite; now runs 1600 tests in <30s. -* Fixed numerous bugs and performance issues with stream management. -* Fixed an issue with the fake emails assigned to bot users. -* Fixed a major performance issue in stream creation. -* Fixed numerous minor accessibility issues. -* Fixed a subtle interaction between click-to-reply and copy-paste. -* Fixed various formatting issues with /me messages. -* Fixed numerous real-time sync issues involving users changing their +- Changed color for private messages to look better. +- Converted realm emoji to be uploaded, not links, for better robustness. +- Switched the default password hasher for new passwords to Argon2. +- Increased the paragraph spacing, making multi-paragraph. +- Improved formatting of all Git integrations. +- Improved the UI of the /stats analytics pages. +- Improved search typeahead to support group private messages. +- Improved logic for when the compose box should open/close. +- Improved lightbox to support scrolling through images. +- Improved Markdown support for bulleted lists. +- Improved copy-to-clipboard support in various places. +- Improved subject lines of missed message emails. +- Improved handling of users trying to log in with OAuth without an account. +- Improved UI of off-the-Internet errors to not be hidden in narrow windows. +- Improved rate-limiting errors to be more easily machine-readable. +- Parallelized the backend test suite; now runs 1600 tests in <30s. +- Fixed numerous bugs and performance issues with stream management. +- Fixed an issue with the fake emails assigned to bot users. +- Fixed a major performance issue in stream creation. +- Fixed numerous minor accessibility issues. +- Fixed a subtle interaction between click-to-reply and copy-paste. +- Fixed various formatting issues with /me messages. +- Fixed numerous real-time sync issues involving users changing their name, avatar, or email address and streams being renamed. -* Fixed numerous performance issues across the project. -* Fixed various left sidebar ordering and live-updated bugs. -* Fixed numerous bugs with the message editing widget. -* Fixed missing logging / rate limiting on browser endpoints. -* Fixed regressions in Zulip's browser state preservation on reload logic. -* Fixed support for Unicode characters in the email mirror system. -* Fixed load spikes when email mirror is receiving a lot of traffic. -* Fixed the ugly grey flicker when scrolling fast on Macs. -* Fixed previews of GitHub image URLs. -* Fixed narrowing via clicking on desktop notifications. -* Fixed Subscribed/Unsubscribed bookends appearing incorrectly. -* Eliminated the idea of a realm having a canonical domain; now +- Fixed numerous performance issues across the project. +- Fixed various left sidebar ordering and live-updated bugs. +- Fixed numerous bugs with the message editing widget. +- Fixed missing logging / rate limiting on browser endpoints. +- Fixed regressions in Zulip's browser state preservation on reload logic. +- Fixed support for Unicode characters in the email mirror system. +- Fixed load spikes when email mirror is receiving a lot of traffic. +- Fixed the ugly grey flicker when scrolling fast on Macs. +- Fixed previews of GitHub image URLs. +- Fixed narrowing via clicking on desktop notifications. +- Fixed Subscribed/Unsubscribed bookends appearing incorrectly. +- Eliminated the idea of a realm having a canonical domain; now there's simply the list of allowed domains for new users. -* Migrated avatars to a user-id-based storage setup (not email-based). -* Trailing whitespace is now stripped in code blocks, avoiding +- Migrated avatars to a user-id-based storage setup (not email-based). +- Trailing whitespace is now stripped in code blocks, avoiding unnecessary scrollbars. -* Most API payloads now refer to users primarily by user ID, with +- Most API payloads now refer to users primarily by user ID, with email available for backwards-compatibility. In the future, we may remove email support. -* Cleaned up Zulip's supervisord configuration. A side effect is the +- Cleaned up Zulip's supervisord configuration. A side effect is the names of the log files have changed for all the queue workers. -* Refactored various endpoints to use a single code path for security +- Refactored various endpoints to use a single code path for security hardening. -* Removed support for the `MANDRILL_CLIENT` setting. It hadn't been +- Removed support for the `MANDRILL_CLIENT` setting. It hadn't been used in years. -* Changed `NOREPLY_EMAIL_ADDRESS` setting to `Name ` +- Changed `NOREPLY_EMAIL_ADDRESS` setting to `Name ` format. -* Disabled the web tutorial on mobile. -* Backend test coverage is now 93%, with 100% in views code. +- Disabled the web tutorial on mobile. +- Backend test coverage is now 93%, with 100% in views code. ### 1.5.2 -- 2017-06-01 @@ -2192,14 +2192,14 @@ Zulip apps. This section links to the upgrade notes from past releases, so you can easily read them all when upgrading across multiple releases. -* [Upgrade notes for 4.0](#upgrade-notes-for-4-0) -* [Upgrade notes for 3.0](#upgrade-notes-for-3-0) -* [Upgrade notes for 2.1.5](#upgrade-notes-for-2-1-5) -* [Upgrade notes for 2.1.0](#upgrade-notes-for-2-1-0) -* [Upgrade notes for 2.0.0](#upgrade-notes-for-2-0-0) -* [Upgrade notes for 1.9.0](#upgrade-notes-for-1-9-0) -* [Upgrade notes for 1.8.0](#upgrade-notes-for-1-8-0) -* [Upgrade notes for 1.7.0](#upgrade-notes-for-1-7-0) +- [Upgrade notes for 4.0](#upgrade-notes-for-4-0) +- [Upgrade notes for 3.0](#upgrade-notes-for-3-0) +- [Upgrade notes for 2.1.5](#upgrade-notes-for-2-1-5) +- [Upgrade notes for 2.1.0](#upgrade-notes-for-2-1-0) +- [Upgrade notes for 2.0.0](#upgrade-notes-for-2-0-0) +- [Upgrade notes for 1.9.0](#upgrade-notes-for-1-9-0) +- [Upgrade notes for 1.8.0](#upgrade-notes-for-1-8-0) +- [Upgrade notes for 1.7.0](#upgrade-notes-for-1-7-0) [docker-zulip]: https://github.com/zulip/docker-zulip [commit-log]: https://github.com/zulip/zulip/commits/main diff --git a/docs/overview/directory-structure.md b/docs/overview/directory-structure.md index 2c2e951571..d04f1706fb 100644 --- a/docs/overview/directory-structure.md +++ b/docs/overview/directory-structure.md @@ -13,33 +13,33 @@ Zulip uses the [Django web framework](https://docs.djangoproject.com/en/1.8/), so a lot of these paths will be familiar to Django developers. -* `zproject/urls.py` Main +- `zproject/urls.py` Main [Django routes file](https://docs.djangoproject.com/en/1.8/topics/http/urls/). Defines which URLs are handled by which view functions or templates. -* `zerver/models.py` Main +- `zerver/models.py` Main [Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/) file. Defines Zulip's database tables. -* `zerver/lib/*.py` Most library code. +- `zerver/lib/*.py` Most library code. -* `zerver/lib/actions.py` Most code doing writes to user-facing +- `zerver/lib/actions.py` Most code doing writes to user-facing database tables lives here. In particular, we have a policy that all code calling `send_event` to trigger [pushing data to clients](../subsystems/events-system.md) must live here. -* `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/). +- `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/). -* `zerver/webhooks/` Webhook views and tests for [Zulip's incoming webhook integrations]( +- `zerver/webhooks/` Webhook views and tests for [Zulip's incoming webhook integrations]( https://zulip.com/api/incoming-webhooks-overview). -* `zerver/tornado/views.py` Tornado views. +- `zerver/tornado/views.py` Tornado views. -* `zerver/worker/queue_processors.py` [Queue workers](../subsystems/queuing.md). +- `zerver/worker/queue_processors.py` [Queue workers](../subsystems/queuing.md). -* `zerver/lib/markdown/` [Backend Markdown processor](../subsystems/markdown.md). +- `zerver/lib/markdown/` [Backend Markdown processor](../subsystems/markdown.md). -* `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/). +- `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/). ------------------------------------------------------------------- @@ -48,39 +48,39 @@ paths will be familiar to Django developers. See [our docs](../subsystems/html-css.md) for details on Zulip's templating systems. -* `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates +- `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates for the backend (for zerver app; logged-in content is in `templates/zerver/app`). -* `static/templates/` [Handlebars](https://handlebarsjs.com/) templates for the frontend. +- `static/templates/` [Handlebars](https://handlebarsjs.com/) templates for the frontend. ---------------------------------------- ### JavaScript, TypeScript, and other static assets -* `static/js/` Zulip's own JavaScript and TypeScript sources. +- `static/js/` Zulip's own JavaScript and TypeScript sources. -* `static/styles/` Zulip's own CSS. +- `static/styles/` Zulip's own CSS. -* `static/images/` Zulip's images. +- `static/images/` Zulip's images. -* `static/third/` Third-party JavaScript and CSS that has been vendored. +- `static/third/` Third-party JavaScript and CSS that has been vendored. -* `node_modules/` Third-party JavaScript installed via `yarn`. +- `node_modules/` Third-party JavaScript installed via `yarn`. -* `static/assets/` For assets not to be served to the web (e.g. the system to +- `static/assets/` For assets not to be served to the web (e.g. the system to generate our favicons). ----------------------------------------------------------------------- ### Tests -* `zerver/tests/` Backend tests. +- `zerver/tests/` Backend tests. -* `frontend_tests/node_tests/` Node Frontend unit tests. +- `frontend_tests/node_tests/` Node Frontend unit tests. -* `frontend_tests/puppeteer_tests/` Puppeteer frontend integration tests. +- `frontend_tests/puppeteer_tests/` Puppeteer frontend integration tests. -* `tools/test-*` Developer-facing test runner scripts. +- `tools/test-*` Developer-facing test runner scripts. ----------------------------------------------------- @@ -89,12 +89,12 @@ templating systems. These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access). -* `zerver/management/commands/` +- `zerver/management/commands/` [Management commands](../subsystems/management-commands.md) one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly). -* `zilencer/management/commands/` includes some dev-specific +- `zilencer/management/commands/` includes some dev-specific commands such as `populate_db`, which are not included in the production distribution. @@ -102,25 +102,25 @@ Django context (i.e. with database access). ### Scripts -* `scripts/` Scripts that production deployments might run manually +- `scripts/` Scripts that production deployments might run manually (e.g., `restart-server`). -* `scripts/lib/` Scripts that are needed on production deployments but +- `scripts/lib/` Scripts that are needed on production deployments but humans should never run directly. -* `scripts/setup/` Scripts that production deployments will only run +- `scripts/setup/` Scripts that production deployments will only run once, during installation. -* `tools/` Scripts used only in a Zulip development environment. +- `tools/` Scripts used only in a Zulip development environment. These are not included in production release tarballs for Zulip, so that we can include scripts here one wouldn't want someone to run in production accidentally (e.g. things that delete the Zulip database without prompting). -* `tools/setup/` Subdirectory of `tools/` for things only used during +- `tools/setup/` Subdirectory of `tools/` for things only used during the development environment setup process. -* `tools/ci/` Subdirectory of `tools/` for things only used to +- `tools/ci/` Subdirectory of `tools/` for things only used to set up and run our tests in CI. Actual test suites should go in `tools/`. @@ -128,12 +128,12 @@ Django context (i.e. with database access). ### API and bots -* See the [Zulip API repository](https://github.com/zulip/python-zulip-api). +- See the [Zulip API repository](https://github.com/zulip/python-zulip-api). Zulip's Python API bindings, a number of Zulip integrations and bots, and a framework for running and testing Zulip bots, used to be developed in the main Zulip server repo but are now in their own repo. -* `templates/zerver/integrations/` (within `templates/zerver/`, above). +- `templates/zerver/integrations/` (within `templates/zerver/`, above). Documentation for these integrations. ------------------------------------------------------------------------- @@ -142,42 +142,42 @@ Django context (i.e. with database access). This is used to deploy essentially all configuration in production. -* `puppet/zulip/` For configuration for production deployments. +- `puppet/zulip/` For configuration for production deployments. -* `puppet/zulip/manifests/profile/standalone.pp` Main manifest for Zulip standalone deployments. +- `puppet/zulip/manifests/profile/standalone.pp` Main manifest for Zulip standalone deployments. ----------------------------------------------------------------------- ### Additional Django apps -* `confirmation` Email confirmation system. +- `confirmation` Email confirmation system. -* `analytics` Analytics for the Zulip server administrator (needs work to +- `analytics` Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). -* `corporate` The old Zulip.com website. Not included in production +- `corporate` The old Zulip.com website. Not included in production distribution. -* `zilencer` Primarily used to hold management commands that aren't +- `zilencer` Primarily used to hold management commands that aren't used in production. Not included in production distribution. ----------------------------------------------------------------------- ### Jinja2 compatibility files -* `zproject/jinja2/__init__.py` Jinja2 environment. +- `zproject/jinja2/__init__.py` Jinja2 environment. ----------------------------------------------------------------------- ### Translation files -* `locale/` Backend (Django) and frontend translation data files. +- `locale/` Backend (Django) and frontend translation data files. ----------------------------------------------------------------------- ### Documentation -* `docs/` Source for this documentation. +- `docs/` Source for this documentation. -------------------------------------------------------------- diff --git a/docs/overview/release-lifecycle.md b/docs/overview/release-lifecycle.md index e657eff0d0..811b7ae003 100644 --- a/docs/overview/release-lifecycle.md +++ b/docs/overview/release-lifecycle.md @@ -4,20 +4,20 @@ This page details the release lifecycle for the Zulip server and client-apps, well as our policies around backwards-compatibility and security support policies. In short: -* We recommend always running the latest releases of the Zulip clients +- We recommend always running the latest releases of the Zulip clients and servers. Server upgrades are designed to Just Work; mobile and desktop client apps update automatically. -* The server and client apps are backwards and forwards compatible +- The server and client apps are backwards and forwards compatible across a wide range of versions. So while it's important to upgrade the server to get security updates, bug fixes, and new features, the mobile and desktop apps will continue working for at least 18 months if you don't do so. -* New server releases are announced via the low-traffic +- New server releases are announced via the low-traffic [zulip-announce email list](https://groups.google.com/forum/#!forum/zulip-announce). We highly recommend subscribing so that you are notified about new security releases. -* Zulip Cloud runs the branch that will become the next major +- Zulip Cloud runs the branch that will become the next major server/webapp release, so it is always "newer" than the latest stable release. @@ -28,16 +28,16 @@ server repository][zulip-server]. ### Stable releases -* Zulip Server **stable releases**, such as Zulip 4.5. +- Zulip Server **stable releases**, such as Zulip 4.5. Organizations self-hosting Zulip primarily use stable releases. -* The numbering scheme is simple: the first digit indicates the major +- The numbering scheme is simple: the first digit indicates the major release series (which we'll refer to as "4.x"). (Before Zulip 3.0, Zulip versions had another digit, e.g. 1.9.2 was a bug fix release in the Zulip 1.9.x major release series). -* [New major releases][blog-major-releases], like Zulip 4.0, are +- [New major releases][blog-major-releases], like Zulip 4.0, are published every 3-6 months, and contain hundreds of features, bug fixes, and improvements to Zulip's internals. -* New maintenance releases, like 4.3, are published roughly once a +- New maintenance releases, like 4.3, are published roughly once a month. Maintenance releases are designed to have no risky changes and be easy to reverse, to minimize stress for administrators. When upgrading to a new major release series, We recommend always @@ -60,24 +60,24 @@ the Zulip server itself (E.g. `https://zulip.example.com/help/`). Many Zulip servers run versions from Git that have not been published in a stable release. -* [Zulip Cloud](https://zulip.com) essentially runs the `main` +- [Zulip Cloud](https://zulip.com) essentially runs the `main` branch. It is usually a few days behind `main` (with some cherry-picked bug fixes), but can fall up to 2 weeks behind when major UI or internals changes mean we'd like to bake changes longer on chat.zulip.org before exposing them to the full Zulip Cloud userbase. -* [chat.zulip.org][chat-zulip-org], the bleeding-edge server for the +- [chat.zulip.org][chat-zulip-org], the bleeding-edge server for the Zulip development community, is upgraded to `main` several times every week. We also often "test deploy" changes not yet in `main` to chat.zulip.org to facilitate design feedback. -* We maintain Git branches with names like `4.x` containing backported +- We maintain Git branches with names like `4.x` containing backported commits from `main` that we plan to include in the next maintenance release. Self hosters can [upgrade][upgrade-from-git] to these stable release branches to get bug fixes staged for the next stable release (which is very useful when you reported a bug whose fix we choose to backport). We support these branches as though they were a stable release. -* Self-hosters who want new features not yet present in a major +- Self-hosters who want new features not yet present in a major release can [upgrade to `main`][upgrading-to-main] or run [a fork of Zulip][fork-zulip]. @@ -91,13 +91,13 @@ process](../production/upgrade-or-modify.md) Just Works. The Zulip server and clients apps are all carefully engineered to ensure compatibility with old versions. In particular: -* The Zulip mobile and desktop apps maintain backwards-compatibility +- The Zulip mobile and desktop apps maintain backwards-compatibility code to support any Zulip server since 2.1.0. (They may also work with older versions, with a degraded experience). -* Zulip maintains an [API changelog](https://zulip.com/api/changelog) +- Zulip maintains an [API changelog](https://zulip.com/api/changelog) detailing all changes to the API to make it easy for client developers to do this correctly. -* The Zulip server preserves backwards-compatibility in its API to +- The Zulip server preserves backwards-compatibility in its API to support versions of the mobile and desktop apps released in roughly the last year. Because these clients auto-update, generally there are only a handful of active clients left by the time we desupport a @@ -130,9 +130,9 @@ Starting with Zulip 4.0, the Zulip webapp will display a banner warning users of a server running a Zulip release that is more than 18 months old. We do this for a few reasons: -* It is unlikely that a server of that age is not vulnerable to +- It is unlikely that a server of that age is not vulnerable to a security bug in Zulip or one of its dependencies. -* The Zulip mobile and desktop apps are only guaranteed to support +- The Zulip mobile and desktop apps are only guaranteed to support server versions less than 18 months old. The nag will appear only to organization administrators starting a @@ -163,11 +163,11 @@ releases, and do not support them in production. The Zulip server project uses several GitHub labels to structure communication within the project about priorities: -* The [high priority][label-high] label tags issues that we consider +- The [high priority][label-high] label tags issues that we consider important. This label is meant to be a determination of importance that can be done quickly and then used as an input to planning processes. -* The [release goal][label-release-goal] label is used for work that +- The [release goal][label-release-goal] label is used for work that we hope to include in the next major release. The related [post release][label-post-release] label is used to track work we want to focus on shortly after the next major release. @@ -192,12 +192,12 @@ Zulip's client apps officially support all Zulip server versions (and Git commits) released in the previous 18 months, matching the behavior of our [upgrade nag](#upgrade-nag). -* The Zulip mobile apps release new versions from the development +- The Zulip mobile apps release new versions from the development branch frequently (usually every couple weeks). Except when fixing a critical bug, releases are first published to our [beta channels][mobile-beta]. -* The Zulip desktop apps are implemented in [Electron][electron], the +- The Zulip desktop apps are implemented in [Electron][electron], the browser-based desktop application framework used by essentially all modern chat applications. The Zulip UI in these apps is served from the Zulip server (and thus can vary between tabs when it is diff --git a/docs/production/authentication-methods.md b/docs/production/authentication-methods.md index 29d1a0b7e7..1c72a2e1cf 100644 --- a/docs/production/authentication-methods.md +++ b/docs/production/authentication-methods.md @@ -26,10 +26,10 @@ creating the initial realm and user. You can disable it after that. With just a few lines of configuration, your Zulip server can authenticate users with any of several single-sign-on (SSO) authentication providers: -* Google accounts, with `GoogleAuthBackend` -* GitHub accounts, with `GitHubAuthBackend` -* GitLab accounts, with `GitLabAuthBackend` -* Microsoft Azure Active Directory, with `AzureADAuthBackend` +- Google accounts, with `GoogleAuthBackend` +- GitHub accounts, with `GitHubAuthBackend` +- GitLab accounts, with `GitLabAuthBackend` +- Microsoft Azure Active Directory, with `AzureADAuthBackend` Each of these requires one to a handful of lines of configuration in `settings.py`, as well as a secret in `zulip-secrets.conf`. Details @@ -51,9 +51,9 @@ In either configuration, you will need to do the following: organization using LDAP authentication. 1. Tell Zulip how to connect to your LDAP server: - * Fill out the section of your `/etc/zulip/settings.py` headed "LDAP + - Fill out the section of your `/etc/zulip/settings.py` headed "LDAP integration, part 1: Connecting to the LDAP server". - * If a password is required, put it in + - If a password is required, put it in `/etc/zulip/zulip-secrets.conf` by setting `auth_ldap_bind_password`. For example: `auth_ldap_bind_password = abcd1234`. @@ -61,10 +61,10 @@ In either configuration, you will need to do the following: 1. Decide how you want to map the information in your LDAP database to users' account data in Zulip. For each Zulip user, two closely related concepts are: - * their **email address**. Zulip needs this in order to send, for + - their **email address**. Zulip needs this in order to send, for example, a notification when they're offline and another user sends a PM. - * their **Zulip username**. This means the name the user types into the + - their **Zulip username**. This means the name the user types into the Zulip login form. You might choose for this to be the user's email address (`sam@example.com`), or look like a traditional "username" (`sam`), or be something else entirely, depending on @@ -79,30 +79,30 @@ In either configuration, you will need to do the following: (A) Using email addresses as Zulip usernames, if LDAP has each user's email address: - * Make `AUTH_LDAP_USER_SEARCH` a query by email address. - * Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to the same query with + - Make `AUTH_LDAP_USER_SEARCH` a query by email address. + - Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to the same query with `%(email)s` rather than `%(user)s` as the search parameter. - * Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP + - Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP attribute for the user's LDAP username in the search result for `AUTH_LDAP_REVERSE_EMAIL_SEARCH`. (B) Using LDAP usernames as Zulip usernames, with email addresses formed consistently like `sam` -> `sam@example.com`: - * Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username - * Set `LDAP_APPEND_DOMAIN = "example.com"`. + - Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username + - Set `LDAP_APPEND_DOMAIN = "example.com"`. (C) Using LDAP usernames as Zulip usernames, with email addresses taken from some other attribute in LDAP (for example, `mail`): - * Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username - * Set `LDAP_EMAIL_ATTR = "mail"`. - * Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find + - Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username + - Set `LDAP_EMAIL_ATTR = "mail"`. + - Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find an LDAP user given their email address (i.e. a search by `LDAP_EMAIL_ATTR`). For example: ```python AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(mail=%(email)s)") ``` - * Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP + - Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP attribute for the user's LDAP username in that search result. You can quickly test whether your configuration works by running: @@ -118,7 +118,7 @@ email address, if it isn't the same as the "Zulip username"). **Active Directory**: Most Active Directory installations will use one of the following configurations: -* To access by Active Directory username: +- To access by Active Directory username: ```python AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)") @@ -127,7 +127,7 @@ of the following configurations: AUTH_LDAP_USERNAME_ATTR = "sAMAccountName" ``` -* To access by Active Directory email address: +- To access by Active Directory email address: ```python AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(mail=%(user)s)") @@ -168,11 +168,11 @@ We recommend running this command in a **regular cron job**, to pick up changes made on your LDAP server. All of these data synchronization options have the same model: -* New users will be populated automatically with the +- New users will be populated automatically with the name/avatar/etc. from LDAP (as configured) on account creation. -* The `manage.py sync_ldap_user_data` cron job will automatically +- The `manage.py sync_ldap_user_data` cron job will automatically update existing users with any changes that were made in LDAP. -* You can easily test your configuration using `manage.py query_ldap`. +- You can easily test your configuration using `manage.py query_ldap`. Once you're happy with the configuration, remember to restart the Zulip server with `/home/zulip/deployments/current/scripts/restart-server` so that @@ -245,7 +245,7 @@ the next time your `manage.py sync_ldap_user_data` cron job runs. Other fields you may want to sync from LDAP include: -* Boolean flags; `is_realm_admin` (the organization's administrator +- Boolean flags; `is_realm_admin` (the organization's administrator permission) is the main one. You can use the [AUTH_LDAP_USER_FLAGS_BY_GROUP][django-auth-booleans] feature of `django-auth-ldap` to configure a group to get this permissions. @@ -253,9 +253,9 @@ Other fields you may want to sync from LDAP include: `is_active` because deactivating a user this way would not disable any active sessions the user might have; see the above discussion of automatic deactivation for how to do that properly). -* String fields like `default_language` (e.g. `en`) or `timezone`, if +- String fields like `default_language` (e.g. `en`) or `timezone`, if you have that data in the right format in your LDAP database. -* [Coming soon][custom-profile-fields-ldap]: Support for syncing +- [Coming soon][custom-profile-fields-ldap]: Support for syncing [custom profile fields](https://zulip.com/help/add-custom-profile-fields) from your LDAP database. @@ -334,16 +334,16 @@ Most issues with LDAP authentication are caused by misconfigurations of the user and email search settings. Some things you can try to get to the bottom of the problem: -* Review the instructions for the LDAP configuration type you're +- Review the instructions for the LDAP configuration type you're using: (A), (B) or (C) (described above), and that you have configured all of the required settings documented in the instructions for that configuration type. -* Use the `manage.py query_ldap` tool to verify your configuration. +- Use the `manage.py query_ldap` tool to verify your configuration. The output of the command will usually indicate the cause of any configuration problem. For the LDAP integration to work, this command should be able to successfully fetch a complete, correct set of data for the queried user. -* You can find LDAP-specific logs in `/var/log/zulip/ldap.log`. If +- You can find LDAP-specific logs in `/var/log/zulip/ldap.log`. If you're asking for help with your setup, please provide logs from this file (feel free to anonymize any email addresses to `username@example.com`) in your report. @@ -364,14 +364,14 @@ it as follows: 1. Tell your IdP how to find your Zulip server: - * **SP Entity ID**: `https://yourzulipdomain.example.com`. + - **SP Entity ID**: `https://yourzulipdomain.example.com`. The `Entity ID` should match the value of `SOCIAL_AUTH_SAML_SP_ENTITY_ID` computed in the Zulip settings. You can get the correct value by running the following: `/home/zulip/deployments/current/scripts/get-django-setting SOCIAL_AUTH_SAML_SP_ENTITY_ID`. - * **SSO URL**: + - **SSO URL**: `https://yourzulipdomain.example.com/complete/saml/`. This is the "SAML ACS url" in SAML terminology. @@ -384,10 +384,10 @@ it as follows: 2. Tell Zulip how to connect to your SAML provider(s) by filling out the section of `/etc/zulip/settings.py` on your Zulip server with the heading "SAML Authentication". - * You will need to update `SOCIAL_AUTH_SAML_ORG_INFO` with your + - You will need to update `SOCIAL_AUTH_SAML_ORG_INFO` with your organization name (`displayname` may appear in the IdP's authentication flow; `name` won't be displayed to humans). - * Fill out `SOCIAL_AUTH_SAML_ENABLED_IDPS` with data provided by + - Fill out `SOCIAL_AUTH_SAML_ENABLED_IDPS` with data provided by your identity provider. You may find [the python-social-auth SAML docs](https://python-social-auth.readthedocs.io/en/latest/backends/saml.html) @@ -514,10 +514,10 @@ straightforward way to deploy that SSO solution with Zulip. 1. In `/etc/zulip/settings.py`, configure two settings: - * `AUTHENTICATION_BACKENDS`: `'zproject.backends.ZulipRemoteUserBackend'`, + - `AUTHENTICATION_BACKENDS`: `'zproject.backends.ZulipRemoteUserBackend'`, and no other entries. - * `SSO_APPEND_DOMAIN`: see documentation in `settings.py`. + - `SSO_APPEND_DOMAIN`: see documentation in `settings.py`. Make sure that you've restarted the Zulip server since making this configuration change. @@ -564,23 +564,23 @@ Most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Suggestions for how to improve this SSO setup documentation are very welcome! -* For example, common issues have to do with `/etc/hosts` not mapping +- For example, common issues have to do with `/etc/hosts` not mapping `settings.EXTERNAL_HOST` to the Apache listening on `127.0.0.1`/`localhost`. -* While debugging, it can often help to temporarily change the Apache +- While debugging, it can often help to temporarily change the Apache config in `/etc/apache2/sites-available/zulip-sso` to listen on all interfaces rather than just `127.0.0.1`. -* While debugging, it can also be helpful to change `proxy_pass` in +- While debugging, it can also be helpful to change `proxy_pass` in `/etc/nginx/zulip-include/app.d/external-sso.conf` to point to a more explicit URL, possibly not over HTTPS. -* The following log files can be helpful when debugging this setup: +- The following log files can be helpful when debugging this setup: - * `/var/log/zulip/{errors.log,server.log}` (the usual places) - * `/var/log/nginx/access.log` (nginx access logs) - * `/var/log/apache2/zulip_auth_access.log` (from the + - `/var/log/zulip/{errors.log,server.log}` (the usual places) + - `/var/log/nginx/access.log` (nginx access logs) + - `/var/log/apache2/zulip_auth_access.log` (from the `zulip-sso.conf` Apache config file; you may want to change `LogLevel` in that file to "debug" to make this more verbose) @@ -591,7 +591,7 @@ assuming you're using the example configuration with HTTP basic auth. This summary should help with understanding what's going on as you try to debug. -* Since you've configured `/etc/zulip/settings.py` to only define the +- Since you've configured `/etc/zulip/settings.py` to only define the `zproject.backends.ZulipRemoteUserBackend`, `zproject/computed_settings.py` configures `/accounts/login/sso/` as `HOME_NOT_LOGGED_IN`. This makes `https://zulip.example.com/` @@ -599,12 +599,12 @@ to debug. nginx) redirect to `/accounts/login/sso/` for a user that isn't logged in. -* nginx proxies requests to `/accounts/login/sso/` to an Apache +- nginx proxies requests to `/accounts/login/sso/` to an Apache instance listening on `localhost:8888`, via the config in `/etc/nginx/zulip-include/app.d/external-sso.conf` (using the upstream `localhost_sso`, defined in `/etc/nginx/zulip-include/upstreams`). -* The Apache `zulip-sso` site which you've enabled listens on +- The Apache `zulip-sso` site which you've enabled listens on `localhost:8888` and (in the example config) presents the `htpasswd` dialogue. (In a real configuration, it takes the user through whatever more complex interaction your SSO solution performs.) The @@ -615,7 +615,7 @@ to debug. variable and either logs the user in or, if they don't have an account already, registers them. The login sets a cookie. -* After succeeding, that redirects the user back to `/` on port 443. +- After succeeding, that redirects the user back to `/` on port 443. This request is sent by nginx to the main Zulip Django app, which sees the cookie, treats them as logged in, and proceeds to serve them the main app page normally. @@ -642,17 +642,17 @@ domain for your server). ``` 1. Configure Apple authentication in `/etc/zulip/settings.py`: - * `SOCIAL_AUTH_APPLE_TEAM`: Your Team ID from Apple, which is a + - `SOCIAL_AUTH_APPLE_TEAM`: Your Team ID from Apple, which is a string like "A1B2C3D4E5". - * `SOCIAL_AUTH_APPLE_SERVICES_ID`: The Services ID you created in + - `SOCIAL_AUTH_APPLE_SERVICES_ID`: The Services ID you created in step 1, which might look like "com.example.services". - * `SOCIAL_AUTH_APPLE_APP_ID`: The App ID, or Bundle ID, of your + - `SOCIAL_AUTH_APPLE_APP_ID`: The App ID, or Bundle ID, of your app that you used in step 1 to configure your Services ID. This might look like "com.example.app". - * `SOCIAL_AUTH_APPLE_KEY`: Despite the name this is not a key, but + - `SOCIAL_AUTH_APPLE_KEY`: Despite the name this is not a key, but rather the Key ID of the key you created in step 2. This looks like "F6G7H8I9J0". - * `AUTHENTICATION_BACKENDS`: Uncomment (or add) a line like + - `AUTHENTICATION_BACKENDS`: Uncomment (or add) a line like `'zproject.backends.AppleAuthBackend',` to enable Apple auth using the created configuration. diff --git a/docs/production/deployment.md b/docs/production/deployment.md index 962bb620b0..22ff2b4b41 100644 --- a/docs/production/deployment.md +++ b/docs/production/deployment.md @@ -51,21 +51,21 @@ The Zulip installer supports the following advanced installer options as well as those mentioned in the [install](../production/install.html#installer-options) documentation: -* `--postgresql-version`: Sets the version of PostgreSQL that will be +- `--postgresql-version`: Sets the version of PostgreSQL that will be installed. We currently support PostgreSQL 10, 11, 12, and 13. -* `--postgresql-missing-dictionaries`: Set +- `--postgresql-missing-dictionaries`: Set `postgresql.missing_dictionaries` ([docs][doc-settings]) in the Zulip settings, which omits some configuration needed for full-text indexing. This should be used with [cloud managed databases like RDS](#using-zulip-with-amazon-rds-as-the-database). This option conflicts with `--no-overwrite-settings`. -* `--no-init-db`: This option instructs the installer to not do any +- `--no-init-db`: This option instructs the installer to not do any database initialization. This should be used when you already have a Zulip database. -* `--no-overwrite-settings`: This option preserves existing +- `--no-overwrite-settings`: This option preserves existing `/etc/zulip` configuration files. ## Running Zulip's service dependencies on different machines @@ -142,9 +142,9 @@ This complication will be removed in a future version. Access an administrative `psql` shell on your PostgreSQL database, and run the commands in `scripts/setup/create-db.sql` to: -* Create a database called `zulip`. -* Create a user called `zulip`. -* Now log in with the `zulip` user to create a schema called +- Create a database called `zulip`. +- Create a user called `zulip`. +- Now log in with the `zulip` user to create a schema called `zulip` in the `zulip` database. You might have to grant `create` privileges first for the `zulip` user to do this. @@ -159,9 +159,9 @@ In `/etc/zulip/settings.py` on your Zulip server, configure the following settings with details for how to connect to your PostgreSQL server. Your database provider should provide these details. -* `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server. -* `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server. -* `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server. +- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server. +- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server. +- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server. If you're using password authentication, you should specify the password of the `zulip` user in /etc/zulip/zulip-secrets.conf as @@ -296,11 +296,11 @@ instead of HTTPS. ### nginx configuration For `nginx` configuration, there's two things you need to set up: -* The root `nginx.conf` file. We recommend using +- The root `nginx.conf` file. We recommend using `/etc/nginx/nginx.conf` from your Zulip server for our recommended settings. E.g. if you don't set `client_max_body_size`, it won't be possible to upload large files to your Zulip server. -* The `nginx` site-specific configuration (in +- The `nginx` site-specific configuration (in `/etc/nginx/sites-available`) for the Zulip app. The following example is a good starting point: @@ -430,9 +430,9 @@ does this. The key configuration options are, for the `/json/events` and `/api/1/events` endpoints: -* `proxy_read_timeout 1200;`. It's critical that this be +- `proxy_read_timeout 1200;`. It's critical that this be significantly above 60s, but the precise value isn't important. -* `proxy_buffering off`. If you don't do this, your `nginx` proxy may +- `proxy_buffering off`. If you don't do this, your `nginx` proxy may return occasional 502 errors to clients using Zulip's events API. 3. The other tricky failure mode we've seen with `nginx` reverse diff --git a/docs/production/email-gateway.md b/docs/production/email-gateway.md index 007d8c2adf..02959f375e 100644 --- a/docs/production/email-gateway.md +++ b/docs/production/email-gateway.md @@ -4,10 +4,10 @@ Zulip's incoming email gateway integration makes it possible to send messages into Zulip by sending an email. It's highly recommended because it enables: -* When users reply to one of Zulip's message notification emails +- When users reply to one of Zulip's message notification emails from their email client, the reply can go directly into Zulip. -* Integrating third-party services that can send email notifications +- Integrating third-party services that can send email notifications into Zulip. See the [integration documentation](https://zulip.com/integrations/doc/email) for details. @@ -112,9 +112,9 @@ Congratulations! The integration should be fully operational. ([Here's how it works with Gmail](https://support.google.com/mail/answer/7126229?hl=en)) 1. Configure IMAP access in the appropriate Zulip settings: - * Login and server connection details in `/etc/zulip/settings.py` + - Login and server connection details in `/etc/zulip/settings.py` in the email gateway integration section (`EMAIL_GATEWAY_LOGIN` and others). - * Password in `/etc/zulip/zulip-secrets.conf` as `email_gateway_password`. + - Password in `/etc/zulip/zulip-secrets.conf` as `email_gateway_password`. 1. Install a cron job to poll the inbox every minute for new messages: ```bash diff --git a/docs/production/email.md b/docs/production/email.md index f919882906..a56a0da1d4 100644 --- a/docs/production/email.md +++ b/docs/production/email.md @@ -61,14 +61,14 @@ services) have free options. Once you've signed up, you'll want to find the service's provided "SMTP credentials", and configure Zulip as follows: -* The hostname like `EMAIL_HOST = 'smtp.mailgun.org'` in `/etc/zulip/settings.py` -* The username like `EMAIL_HOST_USER = 'username@example.com'` in +- The hostname like `EMAIL_HOST = 'smtp.mailgun.org'` in `/etc/zulip/settings.py` +- The username like `EMAIL_HOST_USER = 'username@example.com'` in `/etc/zulip/settings.py`. -* The TLS setting as `EMAIL_USE_TLS = True` in +- The TLS setting as `EMAIL_USE_TLS = True` in `/etc/zulip/settings.py`, for most providers -* The port as `EMAIL_PORT = 587` in `/etc/zulip/settings.py`, for most +- The port as `EMAIL_PORT = 587` in `/etc/zulip/settings.py`, for most providers -* The password like `email_password = abcd1234` in `/etc/zulip/zulip-secrets.conf`. +- The password like `email_password = abcd1234` in `/etc/zulip/zulip-secrets.conf`. ### Using system email @@ -100,16 +100,16 @@ We don't recommend using an inbox product like Gmail for outgoing email, because Gmail's anti-spam measures make this annoying. But if you want to use a Gmail account to send outgoing email anyway, here's how to make it work: -* Create a totally new Gmail account for your Zulip server; you don't +- Create a totally new Gmail account for your Zulip server; you don't want Zulip's automated emails to come from your personal email address. -* If you're using 2-factor authentication on the Gmail account, you'll +- If you're using 2-factor authentication on the Gmail account, you'll need to use an [app-specific password](https://support.google.com/accounts/answer/185833). -* If you're not using 2-factor authentication, read this Google +- If you're not using 2-factor authentication, read this Google support answer and configure that account as ["less secure"](https://support.google.com/accounts/answer/6010255); Gmail doesn't allow servers to send outgoing email by default. -* Note also that the rate limits for Gmail are also quite low +- Note also that the rate limits for Gmail are also quite low (e.g. 100 / day), so it's easy to get rate-limited if your server has significant traffic. For more active servers, we recommend moving to a free account on a transactional email service. @@ -148,11 +148,11 @@ default From address for your Zulip server, and one sent by the If it doesn't work, check these common failure causes: -* Your hosting provider may block outgoing SMTP traffic in its default +- Your hosting provider may block outgoing SMTP traffic in its default firewall rules. Check whether the port `EMAIL_PORT` is blocked in your hosting provider's firewall. -* Your SMTP server's permissions might not allow the email account +- Your SMTP server's permissions might not allow the email account you're using to send email from the `noreply` email addresses used by Zulip when sending confirmation emails. @@ -170,11 +170,11 @@ If it doesn't work, check these common failure causes: the security issue with helpdesk software that `ADD_TOKENS_TO_NOREPLY_ADDRESS` helps protect against. -* Make sure you set the password in `/etc/zulip/zulip-secrets.conf`. +- Make sure you set the password in `/etc/zulip/zulip-secrets.conf`. -* Check the username and password for typos. +- Check the username and password for typos. -* Be sure to restart your Zulip server after editing either +- Be sure to restart your Zulip server after editing either `settings.py` or `zulip-secrets.conf`, using `/home/zulip/deployments/current/scripts/restart-server` . Note that the `manage.py` command above will read the latest @@ -186,24 +186,24 @@ If it doesn't work, check these common failure causes: Here are a few final notes on what to look at when debugging why you aren't receiving emails from Zulip: -* Most transactional email services have an "outgoing email" log where +- Most transactional email services have an "outgoing email" log where you can inspect the emails that reached the service, whether an email was flagged as spam, etc. -* Starting with Zulip 1.7, Zulip logs an entry in +- Starting with Zulip 1.7, Zulip logs an entry in `/var/log/zulip/send_email.log` whenever it attempts to send an email. The log entry includes whether the request succeeded or failed. -* If attempting to send an email throws an exception, a traceback +- If attempting to send an email throws an exception, a traceback should be in `/var/log/zulip/errors.log`, along with any other exceptions Zulip encounters. -* If your SMTP provider uses SSL on port 465 (and not TLS on port +- If your SMTP provider uses SSL on port 465 (and not TLS on port 587), you need to set `EMAIL_PORT = 465` as well as replacing `EMAIL_USE_TLS = True` with `EMAIL_USE_SSL = True`; otherwise, Zulip will try to use the TLS protocol on port 465, which won't work. -* Zulip's email sending configuration is based on the standard Django +- Zulip's email sending configuration is based on the standard Django [SMTP backend](https://docs.djangoproject.com/en/2.0/topics/email/#smtp-backend) configuration. So if you're having trouble getting your email provider working, you may want to search for documentation related diff --git a/docs/production/export-and-import.md b/docs/production/export-and-import.md index 2803a08729..72f0685322 100644 --- a/docs/production/export-and-import.md +++ b/docs/production/export-and-import.md @@ -5,16 +5,16 @@ move data from one Zulip server to another, do backups, compliance work, or migrate from your own servers to the hosted Zulip Cloud service (or back): -* The [Backup](#backups) tool is designed for exact restoration of a +- The [Backup](#backups) tool is designed for exact restoration of a Zulip server's state, for disaster recovery, testing with production data, or hardware migration. This tool has a few limitations: - * Backups must be restored on a server running the same Zulip + - Backups must be restored on a server running the same Zulip version (most precisely, one where `manage.py showmigrations` has the same output). - * Backups must be restored on a server running the same PostgreSQL + - Backups must be restored on a server running the same PostgreSQL version. - * Backups aren't useful for migrating organizations between + - Backups aren't useful for migrating organizations between self-hosting and Zulip Cloud (which may require renumbering all the users/messages/etc.). @@ -24,7 +24,7 @@ service (or back): document [backup details](#backup-details) for users managing backups manually. -* The logical [Data export](#data-export) tool is designed for +- The logical [Data export](#data-export) tool is designed for migrating data between Zulip Cloud and other Zulip servers, as well as various auditing purposes. The logical export tool produces a `.tar.gz` archive with most of the Zulip database data encoded in @@ -41,13 +41,13 @@ service (or back): tool isn't applicable, including situations where an easily machine-parsable export format is desired. -* Zulip also has an [HTML archive +- Zulip also has an [HTML archive tool](https://github.com/zulip/zulip-archive), which is primarily intended for public archives, but can also be useful to inexpensively preserve public stream conversations when decommissioning a Zulip organization. -* It's possible to set up [PostgreSQL streaming +- It's possible to set up [PostgreSQL streaming replication](#postgresql-streaming-replication) and the [S3 file upload backend](../production/upload-backends.html#s3-backend-configuration) @@ -128,11 +128,11 @@ server, including the database, settings, secrets from The following data is not included in these backup archives, and you may want to back up separately: -* The server access/error logs from `/var/log/zulip`. The Zulip +- The server access/error logs from `/var/log/zulip`. The Zulip server only appends to logs, and they can be very large compared to the rest of the data for a Zulip server. -* Files uploaded with the Zulip +- Files uploaded with the Zulip [S3 file upload backend](../production/upload-backends.md). We don't include these for two reasons. First, the uploaded file data in S3 can easily be many times larger than the rest of the backup, @@ -140,7 +140,7 @@ and you may want to back up separately: exceed its disk capacity. Additionally, S3 is a reliable persistent storage system with its own high-quality tools for doing backups. -* SSL certificates. These are not included because they are +- SSL certificates. These are not included because they are particularly security-sensitive and are either trivially replaced (if generated via Certbot) or provided by the system administrator. @@ -159,7 +159,7 @@ backup strategy), and also serves as documentation for what is included in the backups generated by Zulip's standard tools. The data includes: -* The PostgreSQL database. You can back this up with any standard +- The PostgreSQL database. You can back this up with any standard database export or backup tool. Zulip has built-in support for taking daily incremental backups using [wal-g](https://github.com/wal-g/wal-g); these backups are stored for @@ -179,12 +179,12 @@ PostgreSQL server to add: the Nagios plugin installed into `/usr/lib/nagios/plugins/zulip_postgresql_backups/check_postgresql_backup`. -* Any user-uploaded files. If you're using S3 as storage for file +- Any user-uploaded files. If you're using S3 as storage for file uploads, this is backed up in S3. But if you have instead set `LOCAL_UPLOADS_DIR`, any files uploaded by users (including avatars) will be stored in that directory and you'll want to back it up. -* Your Zulip configuration including secrets from `/etc/zulip/`. +- Your Zulip configuration including secrets from `/etc/zulip/`. E.g. if you lose the value of `secret_key`, all users will need to log in again when you set up a replacement server since you won't be able to verify their cookies. If you lose `avatar_salt`, any @@ -198,27 +198,27 @@ email), etc. To restore from a manual backup, the process is basically the reverse of the above: -* Install new server as normal by downloading a Zulip release tarball +- Install new server as normal by downloading a Zulip release tarball and then using `scripts/setup/install`. You don't need to run the `initialize-database` second stage which puts default data into the database. -* Unpack to `/etc/zulip` the `settings.py` and `zulip-secrets.conf` files +- Unpack to `/etc/zulip` the `settings.py` and `zulip-secrets.conf` files from your backups. -* If you ran `initialize-database` anyway above, you'll want to run +- If you ran `initialize-database` anyway above, you'll want to run `scripts/setup/postgresql-init-db` to drop the initial database first. -* Restore your database from the backup. +- Restore your database from the backup. -* Reconfigure rabbitmq to use the password from `secrets.conf` +- Reconfigure rabbitmq to use the password from `secrets.conf` by running, as root, `scripts/setup/configure-rabbitmq`. -* If you're using local file uploads, restore those files to the path +- If you're using local file uploads, restore those files to the path specified by `settings.LOCAL_UPLOADS_DIR` and (if appropriate) any logs. -* Start the server using `scripts/restart-server`. +- Start the server using `scripts/restart-server`. This restoration process can also be used to migrate a Zulip installation from one server to another. @@ -233,8 +233,8 @@ that they are up to date using the Nagios plugin at: Zulip has database configuration for using PostgreSQL streaming replication. You can see the configuration in these files: -* `puppet/zulip_ops/manifests/profile/postgresql.pp` -* `puppet/zulip_ops/files/postgresql/*` +- `puppet/zulip_ops/manifests/profile/postgresql.pp` +- `puppet/zulip_ops/files/postgresql/*` We use this configuration for Zulip Cloud, and it works well in production, but it's not fully generic. Contributions to make it a @@ -300,10 +300,10 @@ archive of all the organization's uploaded files. 1. [Install a new Zulip server](../production/install.md), **skipping Step 3** (you'll create your Zulip organization via the data import tool instead). - * Ensure that the Zulip server you're importing into is running the same + - Ensure that the Zulip server you're importing into is running the same version of Zulip as the server you're exporting from. - * For exports from Zulip Cloud (zulip.com), you need to [upgrade to + - For exports from Zulip Cloud (zulip.com), you need to [upgrade to `main`][upgrade-zulip-from-git], since we run run `main` on Zulip Cloud: @@ -315,7 +315,7 @@ archive of all the organization's uploaded files. zulip.com runs pre-release versions of Zulip that are often several months of development ahead of the latest release. - * Note that if your server has limited free RAM, you'll want to + - Note that if your server has limited free RAM, you'll want to shut down the Zulip server with `./scripts/stop-server` while you run the import, since our minimal system requirements do not budget extra RAM for running the data import tool. @@ -326,10 +326,10 @@ new server to reuse the server-level configuration and secret keys from your old server. There are a few important details to understand about doing so: - * Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is + - Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is safe and recommended. Care is required when copying secrets from `/etc/zulip/zulip-secrets.conf` (details below). - * If you copy `zulip_org_id` and `zulip_org_key` (the credentials + - If you copy `zulip_org_id` and `zulip_org_key` (the credentials for the [mobile push notifications service](../production/mobile-push-notifications.md)), you should be very careful to make sure the no users had their IDs @@ -345,15 +345,15 @@ about doing so: simply to not copy these settings and re-register your server for mobile push notifications if any users had their IDs renumbered during the logical export/import process. - * If you copy the `rabbitmq_password` secret from + - If you copy the `rabbitmq_password` secret from `zulip-secrets.conf`, you'll need to run `scripts/setup/configure-rabbitmq` to update your local RabbitMQ installation to use the password in your Zulip secrets file. - * You will likely want to copy `camo_key` (required to avoid + - You will likely want to copy `camo_key` (required to avoid breaking certain links) and any settings you added related to authentication and email delivery so that those work on your new server. - * Copying `avatar_salt` is not recommended, due to similar issues + - Copying `avatar_salt` is not recommended, due to similar issues to the mobile push notifications service. Zulip will automatically rewrite avatars at URLs appropriate for the new user IDs, and using the same avatar salt (and same server URL) @@ -421,8 +421,8 @@ delete the test import data from your Zulip server before doing a final import. You can **permanently delete** all data from a Zulip organization using the following procedure: -* Start a [Zulip management shell](../production/management-commands.html#manage-py-shell) -* In the management shell, run the following commands, replacing `""` +- Start a [Zulip management shell](../production/management-commands.html#manage-py-shell) +- In the management shell, run the following commands, replacing `""` with the subdomain if [you are hosting the organization on a subdomain](../production/multiple-organizations.md): diff --git a/docs/production/install.md b/docs/production/install.md index afb46b9a61..3d47602eb4 100644 --- a/docs/production/install.md +++ b/docs/production/install.md @@ -24,9 +24,9 @@ wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz tar -xf zulip-server-latest.tar.gz ``` -* If you'd like to verify the download, we +- If you'd like to verify the download, we [publish the sha256sums of our release tarballs](https://www.zulip.org/dist/releases/SHA256SUMS.txt). -* You can also +- You can also [install a pre-release version of Zulip](../production/deployment.html#installing-zulip-from-git) using code from our [repository on GitHub](https://github.com/zulip/zulip/). @@ -48,21 +48,21 @@ If the script gives an error, consult [Troubleshooting](#troubleshooting) below. #### Installer options -* `--email=you@example.com`: The email address of the person or team +- `--email=you@example.com`: The email address of the person or team who should get support and error emails from this Zulip server. This becomes `ZULIP_ADMINISTRATOR` ([docs][doc-settings]) in the Zulip settings. -* `--hostname=zulip.example.com`: The user-accessible domain name for +- `--hostname=zulip.example.com`: The user-accessible domain name for this Zulip server, i.e., what users will type in their web browser. This becomes `EXTERNAL_HOST` ([docs][doc-settings]) in the Zulip settings. -* `--self-signed-cert`: With this option, the Zulip installer +- `--self-signed-cert`: With this option, the Zulip installer generates a self-signed SSL certificate for the server. This isn't suitable for production use, but may be convenient for testing. -* `--certbot`: With this option, the Zulip installer automatically +- `--certbot`: With this option, the Zulip installer automatically obtains an SSL certificate for the server [using Certbot][doc-certbot]. If you'd prefer to acquire an SSL certificate yourself in any other way, it's easy to [provide it to Zulip][doc-ssl-manual]. @@ -100,23 +100,23 @@ do. To really see Zulip in action, you'll need to get the people you work together with using it with you. -* [Set up outgoing email](email.md) so Zulip can confirm new users' +- [Set up outgoing email](email.md) so Zulip can confirm new users' email addresses and send notifications. -* Learn how to [get your organization started][realm-admin-docs] using +- Learn how to [get your organization started][realm-admin-docs] using Zulip at its best. Learning more: -* Subscribe to the [Zulip announcements email +- Subscribe to the [Zulip announcements email list](https://groups.google.com/forum/#!forum/zulip-announce) for server administrators. This extremely low-traffic list is for important announcements, including [new releases](../overview/release-lifecycle.md) and security issues. You can also use the [RSS feed](https://groups.google.com/forum/#!aboutgroup/zulip-announce). -* Follow [Zulip on Twitter](https://twitter.com/zulip). -* Learn how to [configure your Zulip server settings](settings.md). -* Learn about [Backups, export and import](../production/export-and-import.md) +- Follow [Zulip on Twitter](https://twitter.com/zulip). +- Learn how to [configure your Zulip server settings](settings.md). +- Learn about [Backups, export and import](../production/export-and-import.md) and [upgrading](../production/upgrade-or-modify.md) a production Zulip server. @@ -128,17 +128,17 @@ server. ## Details: What the installer does The install script does several things: -* Creates the `zulip` user, which the various Zulip servers will run as. -* Creates `/home/zulip/deployments/`, which the Zulip code for this +- Creates the `zulip` user, which the various Zulip servers will run as. +- Creates `/home/zulip/deployments/`, which the Zulip code for this deployment (and future deployments when you upgrade) goes into. At the very end of the install process, the script moves the Zulip code tree it's running from (which you unpacked from a tarball above) to a directory there, and makes `/home/zulip/deployments/current` as a symbolic link to it. -* Installs Zulip's various dependencies. -* Configures the various third-party services Zulip uses, including +- Installs Zulip's various dependencies. +- Configures the various third-party services Zulip uses, including PostgreSQL, RabbitMQ, Memcached and Redis. -* Initializes Zulip's database. +- Initializes Zulip's database. If you'd like to deploy Zulip with these services on different machines, check out our [deployment options documentation](deployment.md). diff --git a/docs/production/management-commands.md b/docs/production/management-commands.md index 76874e9c1f..e464ef29b9 100644 --- a/docs/production/management-commands.md +++ b/docs/production/management-commands.md @@ -104,35 +104,35 @@ access other functions, you'll need to import them yourself. There are dozens of useful management commands under `zerver/management/commands/`. We detail a few here: -* `./manage.py help`: Lists all available management commands. -* `./manage.py dbshell`: If you're more comfortable with raw SQL than +- `./manage.py help`: Lists all available management commands. +- `./manage.py dbshell`: If you're more comfortable with raw SQL than Python, this will open a PostgreSQL SQL shell connected to the Zulip server's database. Beware of changing data; editing data directly with SQL will often not behave correctly because PostgreSQL doesn't know to flush Zulip's caches or notify browsers of changes. -* `./manage.py send_custom_email`: Can be used to send an email to a set +- `./manage.py send_custom_email`: Can be used to send an email to a set of users. The `--help` documents how to run it from a `manage.py shell` for use with more complex programmatically computed sets of users. -* `./manage.py send_password_reset_email`: Sends password reset email(s) +- `./manage.py send_password_reset_email`: Sends password reset email(s) to one or more users. -* `./manage.py change_realm_subdomain`: Change subdomain of a realm. -* `./manage.py change_user_email`: Change a user's email address. -* `./manage.py change_user_role`: Can change are user's role +- `./manage.py change_realm_subdomain`: Change subdomain of a realm. +- `./manage.py change_user_email`: Change a user's email address. +- `./manage.py change_user_role`: Can change are user's role (easier done [via the UI](https://zulip.com/help/change-a-users-role)) or give bots the `can_forge_sender` permission, which is needed for certain special API features. -* `./manage.py export_single_user`: does a limited version of the [main +- `./manage.py export_single_user`: does a limited version of the [main export tools](../production/export-and-import.md) containing just the messages accessible by a single user. -* `./manage.py reactivate_realm`: Reactivates a realm. -* `./manage.py deactivate_user`: Deactivates a user. This can be done +- `./manage.py reactivate_realm`: Reactivates a realm. +- `./manage.py deactivate_user`: Deactivates a user. This can be done more easily in Zulip's organization administrator UI. -* `./manage.py delete_user`: Completely delete a user from the database. +- `./manage.py delete_user`: Completely delete a user from the database. For most purposes, deactivating users is preferred, since that does not alter message history for other users. See the `./manage.py delete_user --help` documentation for details. -* `./manage.py clear_auth_rate_limit_history`: If a user failed authenticaton +- `./manage.py clear_auth_rate_limit_history`: If a user failed authenticaton attempts too many times and further attempts are disallowed by the rate limiter, this can be used to reset the limit. @@ -144,13 +144,13 @@ via `manage.py command_name --help`. Zulip supports several mechanisms for running custom code on a self-hosted Zulip server: -* Using an existing [integration][integrations] or writing your own +- Using an existing [integration][integrations] or writing your own [webhook integration][webhook-integrations] or [bot][writing-bots]. -* Writing a program using the [Zulip API][zulip-api]. -* [Modifying the Zulip server][modifying-zulip]. -* Using the interactive [management shell](#manage-py-shell), +- Writing a program using the [Zulip API][zulip-api]. +- [Modifying the Zulip server][modifying-zulip]. +- Using the interactive [management shell](#manage-py-shell), documented above, for one-time work or prototyping. -* Writing a custom management command, detailed here. +- Writing a custom management command, detailed here. Custom management commands are Python 3 programs that run inside Zulip's context, so that they can access its libraries, database, and diff --git a/docs/production/mobile-push-notifications.md b/docs/production/mobile-push-notifications.md index ab60bc4211..4075af1ab4 100644 --- a/docs/production/mobile-push-notifications.md +++ b/docs/production/mobile-push-notifications.md @@ -58,15 +58,15 @@ Congratulations! You've successfully set up the service. If you'd like to verify that everything is working, you can do the following. Please follow the instructions carefully: -* [Configure mobile push notifications to always be sent][mobile-notifications-always] +- [Configure mobile push notifications to always be sent][mobile-notifications-always] (normally they're only sent if you're idle, which isn't ideal for this sort of testing). -* On an Android device, download and log in to the +- On an Android device, download and log in to the [Zulip Android app](https://play.google.com/store/apps/details?id=com.zulipmobile). If you were already logged in before configuring the server, you'll need to log out first, since the app only registers for push notifications on login. -* Hit the home button, so Zulip is running in the background, and then +- Hit the home button, so Zulip is running in the background, and then have **another user** send you a **private message** (By default, Zulip only sends push notifications for private messages sent by other users and messages mentioning you). A push notification should appear @@ -113,27 +113,27 @@ push notifications, you agree to those terms. We've designed this push notification bouncer service with security and privacy in mind: -* A central design goal of the the Push Notification Service is to +- A central design goal of the the Push Notification Service is to avoid any message content being stored or logged by the service, even in error cases. -* The Push Notification Service only stores the necessary metadata for +- The Push Notification Service only stores the necessary metadata for delivering the notifications to the appropriate devices, and nothing else: - * The APNS/FCM tokens needed to securely send mobile push + - The APNS/FCM tokens needed to securely send mobile push notifications to iOS and Android devices, one per device registered to be notified by your Zulip server. - * User ID numbers generated by your Zulip server, needed to route + - User ID numbers generated by your Zulip server, needed to route a given notification to the appropriate set of mobile devices. These user ID numbers are are opaque to the Push Notification Service and Kandra Labs. -* The Push Notification Service receives (but does not store) the +- The Push Notification Service receives (but does not store) the contents of individual mobile push notifications: - * The numeric message ID generated by your Zulip server. - * Metadata on the message's sender (name and avatar URL). - * Metadata on the message's recipient (stream name + ID, topic, + - The numeric message ID generated by your Zulip server. + - Metadata on the message's sender (name and avatar URL). + - Metadata on the message's recipient (stream name + ID, topic, private message recipients, etc.). - * A timestamp. - * The message's content. + - A timestamp. + - The message's content. There's a `PUSH_NOTIFICATION_REDACT_CONTENT` setting available to disable any message content being sent via the push notification @@ -146,14 +146,14 @@ and privacy in mind: which would eliminate that usability tradeoff and additionally allow us to not have any access to the other details mentioned in this section. -* All of the network requests (both from Zulip servers to the Push +- All of the network requests (both from Zulip servers to the Push Notification Service and from the Push Notification Service to the relevant Google and Apple services) are encrypted over the wire with SSL/TLS. -* The code for the push notification forwarding service is 100% open +- The code for the push notification forwarding service is 100% open source and available as part of the [Zulip server project on GitHub](https://github.com/zulip/zulip). -* The push notification forwarding servers are professionally managed +- The push notification forwarding servers are professionally managed by a small team of security expert engineers. If you have any questions about the security model, contact @@ -217,17 +217,17 @@ the app stores yourself. If you've done that work, the Zulip server configuration for sending push notifications through the new app is quite straightforward: -* Create a +- Create a [FCM push notifications](https://firebase.google.com/docs/cloud-messaging) key in the Google Developer console and set `android_gcm_api_key` in `/etc/zulip/zulip-secrets.conf` to that key. -* Register for a +- Register for a [mobile push notification certificate][apple-docs] from Apple's developer console. Set `APNS_SANDBOX=False` and `APNS_CERT_FILE` to be the path of your APNS certificate file in `/etc/zulip/settings.py`. -* Set the `APNS_TOPIC` and `ZULIP_IOS_APP_ID` settings to the ID for +- Set the `APNS_TOPIC` and `ZULIP_IOS_APP_ID` settings to the ID for your app (for the official Zulip apps, they are both `org.zulip.Zulip`). -* Restart the Zulip server. +- Restart the Zulip server. [apple-docs]: https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html diff --git a/docs/production/multiple-organizations.md b/docs/production/multiple-organizations.md index 5b2ff2856a..d1acf7f1fa 100644 --- a/docs/production/multiple-organizations.md +++ b/docs/production/multiple-organizations.md @@ -28,20 +28,20 @@ server at the same time. When you want to create a new organization, you need to do a few things: -* If you're using Zulip older than 1.7, you'll need to set +- If you're using Zulip older than 1.7, you'll need to set `REALMS_HAVE_SUBDOMAINS=True` in your `/etc/zulip/settings.py` file. That setting is the default in 1.7 and later. -* Make sure you have SSL certificates for all of the subdomains you're +- Make sure you have SSL certificates for all of the subdomains you're going to use. If you're using [our Let's Encrypt instructions](ssl-certificates.md), it's easy to just specify multiple subdomains in your certificate request. -* If necessary, modify your `nginx` configuration to use your new +- If necessary, modify your `nginx` configuration to use your new certificates. -* Use `./manage.py generate_realm_creation_link` again to create your +- Use `./manage.py generate_realm_creation_link` again to create your new organization. Review [the install instructions](install.md) if you need a refresher on how this works. -* If you're planning on using GitHub auth or another social +- If you're planning on using GitHub auth or another social authentication method, review [the notes on `SOCIAL_AUTH_SUBDOMAIN` below](#authentication). diff --git a/docs/production/postgresql.md b/docs/production/postgresql.md index 0e24733205..3fab9893bf 100644 --- a/docs/production/postgresql.md +++ b/docs/production/postgresql.md @@ -32,21 +32,21 @@ called "zulip" in your database server. You can configure these options in `/etc/zulip/settings.py` (the below descriptions are from the PostgreSQL documentation): -* `REMOTE_POSTGRES_HOST`: Name or IP address of the remote host -* `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server, +- `REMOTE_POSTGRES_HOST`: Name or IP address of the remote host +- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server, different options you can use are: - * disable: I don't care about security, and I don't want to pay the + - disable: I don't care about security, and I don't want to pay the overhead of encryption. - * allow: I don't care about security, but I will pay the overhead of + - allow: I don't care about security, but I will pay the overhead of encryption if the server insists on it. - * prefer: I don't care about encryption, but I wish to pay the + - prefer: I don't care about encryption, but I wish to pay the overhead of encryption if the server supports it. - * require: I want my data to be encrypted, and I accept the + - require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want. - * verify-ca: I want my data encrypted, and I accept the overhead. I + - verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust. - * verify-full: I want my data encrypted, and I accept the + - verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it's the one I specify. diff --git a/docs/production/requirements.md b/docs/production/requirements.md index 0011a5fe17..aa8aa3ca40 100644 --- a/docs/production/requirements.md +++ b/docs/production/requirements.md @@ -1,15 +1,15 @@ # Requirements and scalability To run a Zulip server, you will need: -* A dedicated machine or VM -* A supported OS: - * Ubuntu 20.04 Focal - * Ubuntu 18.04 Bionic - * Debian 10 Buster -* At least 2GB RAM, and 10GB disk space - * If you expect 100+ users: 4GB RAM, and 2 CPUs -* A hostname in DNS -* Credentials for sending email +- A dedicated machine or VM +- A supported OS: + - Ubuntu 20.04 Focal + - Ubuntu 18.04 Bionic + - Debian 10 Buster +- At least 2GB RAM, and 10GB disk space + - If you expect 100+ users: 4GB RAM, and 2 CPUs +- A hostname in DNS +- Credentials for sending email For details on each of these requirements, see below. @@ -50,7 +50,7 @@ https://help.ubuntu.com/community/Repositories/Ubuntu #### Hardware specifications -* CPU and memory: For installations with 100+ users you'll need a +- CPU and memory: For installations with 100+ users you'll need a minimum of **2 CPUs** and **4GB RAM**. For installations with fewer users, 1 CPU and 2GB RAM is sufficient. We strongly recommend against installing with less than 2GB of RAM, as you will likely experience @@ -58,7 +58,7 @@ https://help.ubuntu.com/community/Repositories/Ubuntu using highly CPU-limited servers like the AWS `t2` style instances for organizations with hundreds of users (active or no). -* Disk space: You'll need at least 10GB of free disk space for a +- Disk space: You'll need at least 10GB of free disk space for a server with dozens of users. We recommend using an SSD and avoiding cloud storage backends that limit the IOPS per second, since the disk is primarily used for the Zulip database. @@ -68,30 +68,30 @@ on hardware requirements for larger organizations. #### Network and security specifications -* Incoming HTTPS access (usually port 443, though this is +- Incoming HTTPS access (usually port 443, though this is [configurable](../production/deployment.html#using-an-alternate-port)) from the networks where your users are (usually, the public Internet). -* Incoming port 80 access (optional). Zulip only serves content over +- Incoming port 80 access (optional). Zulip only serves content over HTTPS, and will redirect HTTP requests to HTTPS. -* Incoming port 25 if you plan to enable Zulip's [incoming email +- Incoming port 25 if you plan to enable Zulip's [incoming email integration](../production/email-gateway.md). -* Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so +- Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage image and website previews and mobile push notifications. Outgoing Internet access is not required if you [disable those features](https://zulip.com/help/allow-image-link-previews). -* Outgoing SMTP access (usually port 587) to your [SMTP +- Outgoing SMTP access (usually port 587) to your [SMTP server](../production/email.md) so that Zulip can send emails. -* A domain name (e.g. `zulip.example.com`) that your users will use to +- A domain name (e.g. `zulip.example.com`) that your users will use to access the Zulip server. In order to generate valid SSL certificates [with Certbot][doc-certbot], and to enable other services such as Google authentication, public DNS name is simpler, but Zulip can be configured to use a non-public domain or even an IP address as its external hostname (though we don't recommend that configuration). -* Zulip supports [running behind a reverse proxy][reverse-proxy]. -* Zulip servers running inside a private network should configure the +- Zulip supports [running behind a reverse proxy][reverse-proxy]. +- Zulip servers running inside a private network should configure the [Smokescreen integration][smokescreen-proxy] to protect against [SSRF attacks][SSRF], where users could make the Zulip server make requests to private resources. @@ -123,7 +123,7 @@ certificate documentation](ssl-certificates.md). #### Outgoing email -* Outgoing email (SMTP) credentials that Zulip can use to send +- Outgoing email (SMTP) credentials that Zulip can use to send outgoing emails to users (e.g. email address confirmation emails during the signup process, message notification emails, password reset, etc.). If you don't have an existing outgoing SMTP solution, @@ -140,10 +140,10 @@ Zulip in production](../production/install.md). This section details some basic guidelines for running a Zulip server for larger organizations (especially >1000 users or 500+ daily active users). Zulip's resource needs depend mainly on 3 parameters: -* daily active users (e.g. number of employees if everyone's an +- daily active users (e.g. number of employees if everyone's an employee) -* total user accounts (can be much larger) -* message volume. +- total user accounts (can be much larger) +- message volume. In the following, we discuss a configuration with at most two types of servers: application servers (running Django, Tornado, RabbitMQ, @@ -155,20 +155,20 @@ most use cases, there's little scalability benefit to doing so. See [deployment options](../production/deployment.md) for details on installing Zulip with a dedicated database server. -* **Dedicated database**. For installations with hundreds of daily +- **Dedicated database**. For installations with hundreds of daily active users, we recommend using a [remote PostgreSQL database](postgresql.md), but it's not required. -* **RAM:** We recommended more RAM for larger installations: - * With 25+ daily active users, 4GB of RAM. - * With 100+ daily active users, 8GB of RAM. - * With 400+ daily active users, 16GB of RAM for the Zulip +- **RAM:** We recommended more RAM for larger installations: + - With 25+ daily active users, 4GB of RAM. + - With 100+ daily active users, 8GB of RAM. + - With 400+ daily active users, 16GB of RAM for the Zulip application server, plus 16GB for the database. - * With 2000+ daily active users 32GB of RAM, plus 32GB for the + - With 2000+ daily active users 32GB of RAM, plus 32GB for the database. - * Roughly linear scaling beyond that. + - Roughly linear scaling beyond that. -* **CPU:** The Zulip application server's CPU usage is heavily +- **CPU:** The Zulip application server's CPU usage is heavily optimized due to extensive work on optimizing the performance of requests for latency reasons. Because most servers with sufficient RAM have sufficient CPU resources, CPU requirements are rarely an @@ -177,34 +177,34 @@ installing Zulip with a dedicated database server. database-optimized (usually low CPU, high memory) instance for the database. -* **Disk for application server:** We recommend using [the S3 file +- **Disk for application server:** We recommend using [the S3 file uploads backend][s3-uploads] to store uploaded files at scale. With the S3 backend configuration, we recommend 50GB of disk for the OS, Zulip software, logs and scratch/free space. Disk needs when storing uploads locally -* **Disk for database:** SSD disk is highly recommended. For +- **Disk for database:** SSD disk is highly recommended. For installations where most messages have <100 recipients, 10GB per 1M messages of history is sufficient plus 1GB per 1000 users is sufficient. If most messages are to public streams with 10K+ users subscribed (like on chat.zulip.org), add 20GB per (1000 user accounts) per (1M messages to public streams). -* **Example:** When the +- **Example:** When the [chat.zulip.org](../contributing/chat-zulip-org.md) community server had 12K user accounts (~300 daily actives) and 800K messages of history (400K to public streams), it was a default configuration single-server installation with 16GB of RAM, 4 cores (essentially always idle), and its database was using about 100GB of disk. -* **Disaster recovery:** One can easily run a hot spare application +- **Disaster recovery:** One can easily run a hot spare application server and a hot spare database (using [PostgreSQL streaming replication][streaming-replication]). Make sure the hot spare application server has copies of `/etc/zulip` and you're either syncing `LOCAL_UPLOADS_DIR` or using the [S3 file uploads backend][s3-uploads]. -* **Sharding:** Zulip releases do not fully support dividing Tornado +- **Sharding:** Zulip releases do not fully support dividing Tornado traffic for a single Zulip realm/organization between multiple application servers, which is why we recommend a hot spare over load-balancing. We don't have an easily deployed configuration for @@ -212,7 +212,7 @@ installing Zulip with a dedicated database server. can't currently offer this model outside of enterprise support contracts. - * Zulip 2.0 and later supports running multiple Tornado servers + - Zulip 2.0 and later supports running multiple Tornado servers sharded by realm/organization, which is how we scale Zulip Cloud. [Contact us][contact-support] for help implementing the sharding policy. diff --git a/docs/production/security-model.md b/docs/production/security-model.md index 5cd4fbe789..7411e5d388 100644 --- a/docs/production/security-model.md +++ b/docs/production/security-model.md @@ -11,7 +11,7 @@ announcement). ## Secure your Zulip server like your email server -* It's reasonable to think about security for a Zulip server like you +- It's reasonable to think about security for a Zulip server like you do security for a team email server -- only trusted individuals within an organization should have shell access to the server. @@ -27,15 +27,15 @@ announcement). ## Encryption and authentication -* Traffic between clients (web, desktop and mobile) and the Zulip +- Traffic between clients (web, desktop and mobile) and the Zulip server is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. -* Zulip requires CSRF tokens in all interactions with the web API to +- Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks. -* The preferred way to log in to Zulip is using an SSO solution like +- The preferred way to log in to Zulip is using an SSO solution like Google auth, LDAP, or similar, but Zulip also supports password authentication. See [the authentication methods documentation](../production/authentication-methods.md) @@ -51,11 +51,11 @@ are rejected, and strong passwords encouraged. The minimum password strength allowed is controlled by two settings in `/etc/zulip/settings.py`: -* `PASSWORD_MIN_LENGTH`: The minimum acceptable length, in characters. +- `PASSWORD_MIN_LENGTH`: The minimum acceptable length, in characters. Shorter passwords are rejected even if they pass the `zxcvbn` test controlled by `PASSWORD_MIN_GUESSES`. -* `PASSWORD_MIN_GUESSES`: The minimum acceptable strength of the +- `PASSWORD_MIN_GUESSES`: The minimum acceptable strength of the password, in terms of the estimated number of passwords an attacker is likely to guess before trying this one. If the user attempts to set a password that `zxcvbn` estimates to be guessable in less than @@ -92,53 +92,53 @@ strength allowed is controlled by two settings in ## Messages and history -* Zulip message content is rendered using a specialized Markdown +- Zulip message content is rendered using a specialized Markdown parser which escapes content to protect against cross-site scripting attacks. -* Zulip supports both public streams and private streams. - * Any non-guest user can join any public stream in the organization, +- Zulip supports both public streams and private streams. + - Any non-guest user can join any public stream in the organization, and can view the complete message history of any public stream without joining the stream. Guests can only access streams that another user adds them to. - * Organization owners and administrators can see and modify most + - Organization owners and administrators can see and modify most aspects of a private stream, including the membership and estimated traffic. Owners and administrators generally cannot see messages sent to private streams or do things that would indirectly give them access to those messages, like adding members or changing the stream privacy settings. - * Non-admins cannot easily see which private streams exist, or interact + - Non-admins cannot easily see which private streams exist, or interact with them in any way until they are added. Given a stream name, they can figure out whether a stream with that name exists, but cannot see any other details about the stream. - * See [Stream permissions](https://zulip.com/help/stream-permissions) for more details. + - See [Stream permissions](https://zulip.com/help/stream-permissions) for more details. -* Zulip supports editing the content and topics of messages that have +- Zulip supports editing the content and topics of messages that have already been sent. As a general philosophy, our policies provide hard limits on the ways in which message content can be changed or undone. In contrast, our policies around message topics favor usefulness (e.g. for conversational organization) over faithfulness to the original. In all configurations: - * Message content can only ever be modified by the original author. + - Message content can only ever be modified by the original author. - * Any message visible to an organization owner or administrator can + - Any message visible to an organization owner or administrator can be deleted at any time by that administrator. - * See + - See [Configuring message editing and deletion](https://zulip.com/help/configure-message-editing-and-deletion) for more details. ## Users and bots -* There are several types of users in a Zulip organization: organization +- There are several types of users in a Zulip organization: organization owners, organization administrators, members (normal users), guests, and bots. -* Owners and administrators have the ability to deactivate and +- Owners and administrators have the ability to deactivate and reactivate other human and bot users, archive streams, add/remove administrator privileges, as well as change configuration for the organization. @@ -148,21 +148,21 @@ strength allowed is controlled by two settings in streams to which the administrator is not subscribed. There are two exceptions: - * Organization owners may get access to private messages via some types of + - Organization owners may get access to private messages via some types of [data export](https://zulip.com/help/export-your-organization). - * Administrators can change the ownership of a bot. If a bot is subscribed + - Administrators can change the ownership of a bot. If a bot is subscribed to a private stream, then an administrator can indirectly get access to stream messages by taking control of the bot, though the access will be limited to what the bot can do. (E.g. incoming webhook bots cannot read messages.) -* Every Zulip user has an API key, available on the settings page. +- Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. -* To properly remove a user's access to a Zulip team, it does not +- To properly remove a user's access to a Zulip team, it does not suffice to change their password or deactivate their account in a SSO system, since neither of those prevents authenticating with the user's API key or those of bots the user has created. Instead, you @@ -170,23 +170,23 @@ strength allowed is controlled by two settings in [deactivate the user's account](https://zulip.com/help/deactivate-or-reactivate-a-user) via Zulip's "Organization settings" interface. -* The Zulip mobile apps authenticate to the server by sending the +- The Zulip mobile apps authenticate to the server by sending the user's password and retrieving the user's API key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user's phone is lost, in addition to changing passwords, you should rotate the user's Zulip API key. -* Guest users are like Members, but they do not have automatic access +- Guest users are like Members, but they do not have automatic access to public streams. -* Zulip supports several kinds of bots with different capabilities. +- Zulip supports several kinds of bots with different capabilities. - * Incoming webhook bots can only send messages into Zulip. - * Outgoing webhook bots and Generic bots can essentially do anything a + - Incoming webhook bots can only send messages into Zulip. + - Outgoing webhook bots and Generic bots can essentially do anything a non-administrator user can, with a few exceptions (e.g. a bot cannot log in to the web application, register for mobile push notifications, or create other bots). - * Bots with the `can_forge_sender` permission can send messages that appear to have been sent by + - Bots with the `can_forge_sender` permission can send messages that appear to have been sent by another user. They also have the ability to see the names of all streams, including private streams. This is important for implementing integrations like the Jabber, IRC, and Zephyr mirrors. @@ -197,7 +197,7 @@ strength allowed is controlled by two settings in ## User-uploaded content and user-generated requests -* Zulip supports user-uploaded files. Ideally they should be hosted +- Zulip supports user-uploaded files. Ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com). @@ -235,26 +235,26 @@ strength allowed is controlled by two settings in browser is logged into a Zulip account that has received the uploaded file in question). -* Zulip supports using the Camo image proxy to proxy content like +- Zulip supports using the Camo image proxy to proxy content like inline image previews, that can be inserted into the Zulip message feed by other users. This ensures that clients do not make requests to external servers to fetch images, improving privacy. -* By default, Zulip will provide image previews inline in the body of +- By default, Zulip will provide image previews inline in the body of messages when a message contains a link to an image. You can control this using the `INLINE_IMAGE_PREVIEW` setting. -* Zulip may make outgoing HTTP connections to other servers in a +- Zulip may make outgoing HTTP connections to other servers in a number of cases: - * Outgoing webhook bots (creation of which can be restricted) - * Inline image previews in messages (enabled by default, but can be disabled) - * Inline webpage previews and embeds (must be configured to be enabled) - * Twitter message previews (must be configured to be enabled) - * BigBlueButton and Zoom API requests (must be configured to be enabled) - * Mobile push notifications (must be configured to be enabled) + - Outgoing webhook bots (creation of which can be restricted) + - Inline image previews in messages (enabled by default, but can be disabled) + - Inline webpage previews and embeds (must be configured to be enabled) + - Twitter message previews (must be configured to be enabled) + - BigBlueButton and Zoom API requests (must be configured to be enabled) + - Mobile push notifications (must be configured to be enabled) -* Notably, these first 3 features give end users (limited) control to cause +- Notably, these first 3 features give end users (limited) control to cause the Zulip server to make HTTP requests on their behalf. As a result, Zulip supports routing all outgoing outgoing HTTP requests [through Smokescreen][smokescreen-setup] to ensure that Zulip cannot be diff --git a/docs/production/settings.md b/docs/production/settings.md index acfa4a84d2..6a046b7d3d 100644 --- a/docs/production/settings.md +++ b/docs/production/settings.md @@ -95,11 +95,11 @@ configuration along with your other Zulip server configuration. ### Miscellaneous server settings Some popular settings in `/etc/zulip/settings.py` include: -* The Twitter integration, which provides pretty inline previews of +- The Twitter integration, which provides pretty inline previews of tweets. -* The [email gateway](../production/email-gateway.md), which lets +- The [email gateway](../production/email-gateway.md), which lets users send emails into Zulip. -* The [Video call integrations](../production/video-calls.md). +- The [Video call integrations](../production/video-calls.md). ## Zulip announcement list diff --git a/docs/production/ssl-certificates.md b/docs/production/ssl-certificates.md index 45cebbe218..e02a4472e4 100644 --- a/docs/production/ssl-certificates.md +++ b/docs/production/ssl-certificates.md @@ -11,8 +11,8 @@ chore (nor expense) that it used to be. If you already have an SSL certificate, just install (or symlink) its files into place at the following paths: -* `/etc/ssl/private/zulip.key` for the private key -* `/etc/ssl/certs/zulip.combined-chain.crt` for the certificate. +- `/etc/ssl/private/zulip.key` for the private key +- `/etc/ssl/certs/zulip.combined-chain.crt` for the certificate. Your certificate file should contain not only your own certificate but its **full chain, including any intermediate certificates** used by @@ -32,11 +32,11 @@ browsers ignore errors that others don't. Two good tests include: -* If your server is accessible from the public Internet, use the [SSL +- If your server is accessible from the public Internet, use the [SSL Labs tester][ssllabs-tester]. Be sure to check for "Chain issues"; if any, your certificate file is missing intermediate certificates. -* Alternatively, run a command like `curl -SsI https://zulip.example.com` +- Alternatively, run a command like `curl -SsI https://zulip.example.com` (using your server's URL) from a machine that can reach your server. Make sure that on the same machine, `curl -SsI https://incomplete-chain.badssl.com` gives an error; @@ -55,11 +55,11 @@ SSL certificates from Let's Encrypt and renew them automatically. We recommend most Zulip servers use Certbot. You'll want something else if: -* you have an existing workflow for managing SSL certificates +- you have an existing workflow for managing SSL certificates that you prefer; -* you need wildcard certificates (support from Let's Encrypt released +- you need wildcard certificates (support from Let's Encrypt released in [March 2018][letsencrypt-wildcard]); or -* your Zulip server is not on the public Internet. (In this case you +- your Zulip server is not on the public Internet. (In this case you can [still use Certbot][certbot-manual-mode], but it's less convenient; and you'll want to ignore Zulip's automation.) @@ -203,10 +203,10 @@ details. Two signs for diagnosing this issue in contrast to some other root cause: -* This issue affects only Android 7.0; it's fixed in Android 7.1.1 and +- This issue affects only Android 7.0; it's fixed in Android 7.1.1 and later. -* If your server is reachable from the public Internet, use the [SSL +- If your server is reachable from the public Internet, use the [SSL Labs tester][ssllabs-tester]. Under "Cipher Suites" you may see lines beginning with `TLS_ECDHE`, for cipher suites which use elliptic-curve cryptography. These lines will have further text diff --git a/docs/production/troubleshooting.md b/docs/production/troubleshooting.md index edc2ededa4..6bfc271b5a 100644 --- a/docs/production/troubleshooting.md +++ b/docs/production/troubleshooting.md @@ -96,17 +96,17 @@ The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise support the Zulip application: -* PostgreSQL -* RabbitMQ -* Nginx -* Redis -* memcached +- PostgreSQL +- RabbitMQ +- Nginx +- Redis +- memcached If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common configuration problems and how to resolve them: -* If your browser reports no webserver is running, that is likely +- If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To fix this, configure them properly @@ -115,7 +115,7 @@ problems and how to resolve them: service nginx restart ``` -* If your host is being port scanned by unauthorized users, you may see +- If your host is being port scanned by unauthorized users, you may see messages in `/var/log/zulip/server.log` like ```text 2017-02-22 14:11:33,537 ERROR Invalid HTTP_HOST header: '10.2.3.4'. You may need to add u'10.2.3.4' to ALLOWED_HOSTS. @@ -126,7 +126,7 @@ problems and how to resolve them: attempt. For more on this issue, see the [Django release notes on Host header poisoning](https://www.djangoproject.com/weblog/2013/feb/19/security/#s-issue-host-header-poisoning) -* An AMQPConnectionError traceback or error running rabbitmqctl +- An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running; to fix this, try: ```bash service rabbitmq-server restart @@ -202,14 +202,14 @@ reporting/investigating any that you do see. Beyond that, the most important monitoring for a Zulip server is standard stuff: -* Basic host health monitoring for issues running out of disk space, +- Basic host health monitoring for issues running out of disk space, especially for the database and where uploads are stored. -* Service uptime and standard monitoring for the [services Zulip +- Service uptime and standard monitoring for the [services Zulip depends on](#troubleshooting-services). Most monitoring software has standard plugins for Nginx, PostgreSQL, Redis, RabbitMQ, and memcached, and those will work well with Zulip. -* `supervisorctl status` showing all services `RUNNING`. -* Checking for processes being OOM killed. +- `supervisorctl status` showing all services `RUNNING`. +- Checking for processes being OOM killed. Beyond that, Zulip ships a few application-specific end-to-end health checks. The Nagios plugins `check_send_receive_time`, @@ -231,30 +231,30 @@ useful Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: -* `check_send_receive_time`: Sends a test message through the system +- `check_send_receive_time`: Sends a test message through the system between two bot users to check that end-to-end message sending works. An effective end-to-end check for Zulip's Django and Tornado systems being healthy. -* `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective +- `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective checks for Zulip's RabbitMQ-based queuing systems being healthy. -* `check_worker_memory`: Monitors for memory leaks in queue workers. -* `check_email_deliverer_backlog` and `check_email_deliverer_process`: +- `check_worker_memory`: Monitors for memory leaks in queue workers. +- `check_email_deliverer_backlog` and `check_email_deliverer_process`: Monitors for whether scheduled outgoing emails (e.g. invitation reminders) are being sent properly. Database monitoring: -* `check_fts_update_log`: Checks whether full-text search updates are +- `check_fts_update_log`: Checks whether full-text search updates are being processed properly or getting backlogged. -* `check_postgres`: General checks for database health. -* `check_postgresql_backup`: Checks status of PostgreSQL backups. -* `check_postgresql_replication_lag`: Checks whether PostgreSQL streaming +- `check_postgres`: General checks for database health. +- `check_postgresql_backup`: Checks status of PostgreSQL backups. +- `check_postgresql_replication_lag`: Checks whether PostgreSQL streaming replication is up to date. Standard server monitoring: -* `check_website_response.sh`: Basic HTTP check. -* `check_debian_packages`: Checks whether the system is behind on +- `check_website_response.sh`: Basic HTTP check. +- `check_debian_packages`: Checks whether the system is behind on `apt upgrade`. If you're using these plugins, bug reports and pull requests to make diff --git a/docs/production/upgrade-or-modify.md b/docs/production/upgrade-or-modify.md index ba8ce68eb7..d95885e729 100644 --- a/docs/production/upgrade-or-modify.md +++ b/docs/production/upgrade-or-modify.md @@ -44,13 +44,13 @@ to a new Zulip release: ``` The upgrade process will: - * Run `apt-get upgrade` - * Install new versions of Zulip's dependencies (mainly Python packages). - * (`upgrade-zulip-from-git` only) Build Zulip's frontend assets using `webpack`. - * Shut down the Zulip service - * Run a `puppet apply` - * Run any database migrations - * Bring the Zulip service back up on the new version. + - Run `apt-get upgrade` + - Install new versions of Zulip's dependencies (mainly Python packages). + - (`upgrade-zulip-from-git` only) Build Zulip's frontend assets using `webpack`. + - Shut down the Zulip service + - Run a `puppet apply` + - Run any database migrations + - Bring the Zulip service back up on the new version. Upgrading will result in brief downtime for the service, which should be under 30 seconds unless there is an expensive database migration @@ -165,10 +165,10 @@ guide](../production/troubleshooting.md). The upgrade scripts are idempotent, so there's no harm in trying again after resolving an issue. The most common causes of errors are: -* Networking issues (e.g. your Zulip server doesn't have reliable +- Networking issues (e.g. your Zulip server doesn't have reliable Internet access or needs a proxy set up). Fix the networking issue and try again. -* Especially when using `upgrade-zulip-from-git`, systems with the +- Especially when using `upgrade-zulip-from-git`, systems with the minimal RAM for running Zulip can run into out-of-memory issues during the upgrade process (generally `tools/webpack` is the step that fails). You can get past this by shutting down the Zulip @@ -176,9 +176,9 @@ after resolving an issue. The most common causes of errors are: the upgrade process. Useful logs are available in a few places: -* The Zulip upgrade scripts log all output to +- The Zulip upgrade scripts log all output to `/var/log/zulip/upgrade.log`. -* The Zulip server logs all Internal Server Errors to +- The Zulip server logs all Internal Server Errors to `/var/log/zulip/errors.log`. If you need help and don't have a support contract, you can visit @@ -525,10 +525,10 @@ If you do modify Zulip and then report an issue you see in your modified version of Zulip, please be responsible about communicating that fact: -* Ideally, you'd reproduce the issue in an unmodified version (e.g. on +- Ideally, you'd reproduce the issue in an unmodified version (e.g. on [chat.zulip.org](../contributing/chat-zulip-org.md) or [zulip.com](https://zulip.com)). -* Where that is difficult or you think it's very unlikely your changes +- Where that is difficult or you think it's very unlikely your changes are related to the issue, just mention your changes in the issue report. If you're looking to modify Zulip by applying changes developed by the @@ -542,41 +542,41 @@ One way to modify Zulip is to just edit files under can work OK for testing small changes to Python code or shell scripts. But we don't recommend this approach for maintaining changes because: -* You cannot modify JavaScript, CSS, or other frontend files this way, +- You cannot modify JavaScript, CSS, or other frontend files this way, because we don't include them in editable form in our production release tarballs (doing so would make our release tarballs much larger without any runtime benefit). -* You will need to redo your changes after you next upgrade your Zulip +- You will need to redo your changes after you next upgrade your Zulip server (or they will be lost). -* You need to remember to restart the server or your changes won't +- You need to remember to restart the server or your changes won't have effect. -* Your changes aren't tracked, so mistakes can be hard to debug. +- Your changes aren't tracked, so mistakes can be hard to debug. Instead, we recommend the following GitHub-based workflow (see [our Git guide][git-guide] if you need a primer): -* Decide where you're going to edit Zulip's code. We recommend [using +- Decide where you're going to edit Zulip's code. We recommend [using the Zulip development environment](../development/overview.md) on a desktop or laptop as it will make it extremely convenient for you to test your changes without deploying them in production. But if your changes are small or you're OK with risking downtime, you don't strictly need it; you just need an environment with Git installed. -* **Important**. Determine what Zulip version you're running on your +- **Important**. Determine what Zulip version you're running on your server. You can check by inspecting `ZULIP_VERSION` in `/home/zulip/deployments/current/version.py` (we'll use `2.0.4` below). If you apply your changes to the wrong version of Zulip, it's likely to fail and potentially cause downtime. -* [Fork and clone][fork-clone] the [zulip/zulip][] repository on +- [Fork and clone][fork-clone] the [zulip/zulip][] repository on [GitHub](https://github.com). -* Create a branch (named `acme-branch` below) containing your changes: +- Create a branch (named `acme-branch` below) containing your changes: ```bash cd zulip git checkout -b acme-branch 2.0.4 ``` -* Use your favorite code editor to modify Zulip. -* Commit your changes and push them to GitHub: +- Use your favorite code editor to modify Zulip. +- Commit your changes and push them to GitHub: ```bash git commit -a @@ -588,7 +588,7 @@ git diff 2.0.4 acme-branch git push origin +acme-branch ``` -* Log in to your Zulip server and configure and use +- Log in to your Zulip server and configure and use [upgrade-zulip-from-git][] to install the changes; remember to configure `git_repo_url` to point to your fork on GitHub and run it as `upgrade-zulip-from-git acme-branch`. @@ -635,10 +635,10 @@ branch, as before. If you are using [docker-zulip][], there are two things that are different from the above: -* Because of how container images work, editing files directly is even +- Because of how container images work, editing files directly is even more precarious, because Docker is designed for working with container images and may lose your changes. -* Instead of running `upgrade-zulip-from-git`, you will need to use +- Instead of running `upgrade-zulip-from-git`, you will need to use the [docker upgrade workflow][docker-zulip-upgrade] to build a container image based on your modified version of Zulip. @@ -691,11 +691,11 @@ so, it's important to understand how to happily run a server based on For background, it's backporting arbitrary patches from `main` to an older version requires some care. Common issues include: -* Changes containing database migrations (new files under +- Changes containing database migrations (new files under `*/migrations/`), which includes most new features. We don't support applying database migrations out of order. -* Changes that are stacked on top of other changes to the same system. -* Essentially any patch with hundreds of lines of changes will have +- Changes that are stacked on top of other changes to the same system. +- Essentially any patch with hundreds of lines of changes will have merge conflicts and require extra work to apply. While it's possible to backport these sorts of changes, you're @@ -706,10 +706,10 @@ If you need an unreleased feature, the best path is usually to upgrade to Zulip `main` using [upgrade-zulip-from-git][]. Before upgrading to `main`, make sure you understand: -* In Zulip's version numbering scheme, `main` will always be "newer" +- In Zulip's version numbering scheme, `main` will always be "newer" than the latest maintenance release (E.g. `3.1` or `2.1.6`) and "older" than the next major release (E.g. `3.0` or `4.0`). -* The `main` branch is under very active development; dozens of new +- The `main` branch is under very active development; dozens of new changes are integrated into it on most days. The `main` branch can have thousands of changes not present in the latest release (all of which will be included in our next major release). On average @@ -717,30 +717,30 @@ upgrading to `main`, make sure you understand: (because we fix hundreds of bugs in every major release) but it might have some bugs that are more severe than we would consider acceptable for a release. -* We deploy `main` to chat.zulip.org and zulip.com on a regular +- We deploy `main` to chat.zulip.org and zulip.com on a regular basis (often daily), so it's very important to the project that it be stable. Most regressions will be minor UX issues or be fixed quickly, because we need them to be fixed for Zulip Cloud. -* The development community is very interested in helping debug issues +- The development community is very interested in helping debug issues that arise when upgrading from the latest release to `main`, since they provide us an opportunity to fix that category of issue before our next major release. (Much more so than we are in helping folks debug other custom changes). That said, we cannot make any guarantees about how quickly we'll resolve an issue to folks without a formal support contract. -* We do not support downgrading from `main` to earlier versions, so +- We do not support downgrading from `main` to earlier versions, so if downtime for your Zulip server is unacceptable, make sure you have a current [backup](../production/export-and-import.html#backups) in case the upgrade fails. -* Our changelog contains [draft release +- Our changelog contains [draft release notes](../overview/changelog.md) available listing major changes since the last release. The **Upgrade notes** section will always be current, even if some new features aren't documented. -* Whenever we push a security or maintenance release, the changes in +- Whenever we push a security or maintenance release, the changes in that release will always be merged to `main`; so you can get the security fixes by upgrading to `main`. -* You can always upgrade from `main` to the next major release when it +- You can always upgrade from `main` to the next major release when it comes out, using either [upgrade-zulip-from-git][] or the release tarball. So there's no risk of upgrading to `main` resulting in a system that's not upgradeable back to a normal release. diff --git a/docs/production/video-calls.md b/docs/production/video-calls.md index c7009f0a56..00c25c293c 100644 --- a/docs/production/video-calls.md +++ b/docs/production/video-calls.md @@ -15,18 +15,18 @@ installation, you'll need to register a custom Zoom app as follows: 1. Create an app with the **OAuth** type. - * Choose an app name such as "ExampleCorp Zulip". - * Select **User-managed app**. - * Disable the option to publish the app on the Marketplace. - * Click **Create**. + - Choose an app name such as "ExampleCorp Zulip". + - Select **User-managed app**. + - Disable the option to publish the app on the Marketplace. + - Click **Create**. 1. Inside of the Zoom app management page: - * On the **App Credentials** tab, set both the **Redirect URL for + - On the **App Credentials** tab, set both the **Redirect URL for OAuth** and the **Whitelist URL** to `https://zulip.example.com/calls/zoom/complete` (replacing `zulip.example.com` by your main Zulip hostname). - * On the **Scopes** tab, add the `meeting:write` scope. + - On the **Scopes** tab, add the `meeting:write` scope. You can then configure your Zulip server to use that Zoom app as follows: diff --git a/docs/subsystems/billing.md b/docs/subsystems/billing.md index 1d9c743e10..70831b87d6 100644 --- a/docs/subsystems/billing.md +++ b/docs/subsystems/billing.md @@ -4,8 +4,8 @@ Zulip uses a third party (Stripe) for billing, so working on the billing system requires a little bit of setup. To set up the development environment to work on the billing code: -* Create a Stripe account -* Go to , and add the +- Create a Stripe account +- Go to , and add the publishable key and secret key as `stripe_publishable_key` and `stripe_secret_key` to `zproject/dev-secrets.conf`. @@ -15,13 +15,13 @@ Nearly all the billing-relevant code lives in `corporate/`. Stripe makes pretty regular updates to their API. The process for upgrading our code is: -* Go to in your Stripe account. -* Upgrade the API version. -* Run `tools/test-backend --generate-stripe-fixtures` -* Fix any failing tests, and manually look through `git diff` to understand +- Go to in your Stripe account. +- Upgrade the API version. +- Run `tools/test-backend --generate-stripe-fixtures` +- Fix any failing tests, and manually look through `git diff` to understand the changes. -* If there are no material changes, commit the diff, and open a PR. -* Ask Rishi or Tim to go to in the +- If there are no material changes, commit the diff, and open a PR. +- Ask Rishi or Tim to go to in the zulipchat Stripe account, and upgrade the API version there. We currently aren't set up to do version upgrades where there are breaking diff --git a/docs/subsystems/caching.md b/docs/subsystems/caching.md index caf45ab15e..4ea951bf72 100644 --- a/docs/subsystems/caching.md +++ b/docs/subsystems/caching.md @@ -73,7 +73,7 @@ def get_user(email: str, realm: Realm) -> UserProfile: ``` This decorator implements a pretty classic caching paradigm: -* The `user_profile_cache_key` function defines a unique map from a +- The `user_profile_cache_key` function defines a unique map from a canonical form of its arguments to a string. These strings are namespaced (the `user_profile:` part) so that they won't overlap with other caches, and encode the arguments so that two uses of this @@ -82,13 +82,13 @@ This decorator implements a pretty classic caching paradigm: is important to ensure we don't send special characters to memcached). And we have two versions, depending whether the caller has access to a `Realm` or just a `realm_id`. -* When `get_user` is called, `cache_with_key` will compute the key, +- When `get_user` is called, `cache_with_key` will compute the key, and do a Django `cache_get` query for the key (which goes to memcached). If the key is in the cache, it just returns the value. Otherwise, it fetches the value from the database (using the actual code in the body of `get_user`), and then stores the value back to that memcached key before returning the result to the caller. -* Cache entries expire after the timeout; in this case, a week. +- Cache entries expire after the timeout; in this case, a week. Though in frequently deployed environments like chat.zulip.org, often cache entries will stop being used long before that, because `KEY_PREFIX` is rotated every time we deploy to production; see @@ -104,7 +104,7 @@ that if an item is in the cache, the body of `get_user` (above) is never called. This means some things that might seem like clever code reuse are actually a really bad idea. For example: -* Don't add a `get_active_user` function that uses the same cache key +- Don't add a `get_active_user` function that uses the same cache key function as `get_user` (but with a different query that filters our deactivated users). If one called `get_active_user` to access a deactivated user, the right thing would happen, but if you called @@ -237,11 +237,11 @@ objects to minimize data transfer between Django and memcached). We generally try to avoid in-process backend caching in Zulip's Django codebase, because every Zulip production installation involves multiple servers. We do have a few, however: -* `per_request_display_recipient_cache`: A cache flushed at the start +- `per_request_display_recipient_cache`: A cache flushed at the start of every request; this simplifies correctly implementing our goal of not repeatedly fetching the "display recipient" (e.g. stream name) for each message in the `GET /messages` codebase. -* Caches of various data, like the `SourceMap` object, that are +- Caches of various data, like the `SourceMap` object, that are expensive to construct, not needed for most requests, and don't change once a Zulip server has been deployed in production. diff --git a/docs/subsystems/dependencies.md b/docs/subsystems/dependencies.md index 6db77ce5f1..8b0c37f05e 100644 --- a/docs/subsystems/dependencies.md +++ b/docs/subsystems/dependencies.md @@ -6,15 +6,15 @@ this document, we discuss the various classes of dependencies that Zulip has, and how we manage them. Zulip's dependency management has some really nice properties: -* **Fast provisioning**. When switching to a different commit in the +- **Fast provisioning**. When switching to a different commit in the Zulip project with the same dependencies, it takes under 5 seconds to re-provision a working Zulip development environment after switching. If there are new dependencies, one only needs to wait to download the new ones, not all the pre-existing dependencies. -* **Consistent provisioning**. Every time a Zulip development or +- **Consistent provisioning**. Every time a Zulip development or production environment is provisioned/installed, it should end up using the exactly correct versions of all major dependencies. -* **Low maintenance burden**. To the extent possible, we want to +- **Low maintenance burden**. To the extent possible, we want to avoid manual work and keeping track of things that could be automated. This makes it easy to keep running the latest versions of our various dependencies. @@ -103,10 +103,10 @@ on specific versions of these packages wherever possible. The exact lists of `apt` packages needed by Zulip are maintained in a few places: -* For production, in our Puppet configuration, `puppet/zulip/`, using +- For production, in our Puppet configuration, `puppet/zulip/`, using the `Package` and `SafePackage` directives. -* For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`. -* The packages needed to build a Zulip virtualenv, in +- For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`. +- The packages needed to build a Zulip virtualenv, in `VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are separate from the rest because (1) we may need to install a virtualenv before running the more complex scripts that, in turn, @@ -124,7 +124,7 @@ about how Zulip makes this system work well for us that are worth highlighting. The system is largely managed by the code in `scripts/lib/setup_venv.py` -* **Using `pip` to manage dependencies**. This is standard in the +- **Using `pip` to manage dependencies**. This is standard in the Python ecosystem, and means we only need to record a list of versions in a `requirements.txt` file to declare what we're using. Since we have a few different installation targets, we maintain @@ -134,7 +134,7 @@ highlighting. The system is largely managed by the code in majority of packages common to prod and development, etc.). We use `pip install --no-deps` to ensure we only install the packages we explicitly declare as dependencies. -* **virtualenv with pinned versions**. For a large application like +- **virtualenv with pinned versions**. For a large application like Zulip, it is important to ensure that we're always using consistent, predictable versions of all of our Python dependencies. To ensure this, we install our dependencies in a [virtualenv][] that contains @@ -145,7 +145,7 @@ highlighting. The system is largely managed by the code in effect is that it's easy to debug problems caused by dependency upgrades, since we're always doing those upgrades with an explicit commit updating the `requirements/` directory. -* **Pinning versions of indirect dependencies**. We "pin" or "lock" +- **Pinning versions of indirect dependencies**. We "pin" or "lock" the versions of our indirect dependencies files with `tools/update-locked-requirements` (powered by `pip-compile`). What this means is that we have some "source" requirements files, like @@ -162,7 +162,7 @@ highlighting. The system is largely managed by the code in direct dependency (or dependencies) needed that indirect dependency. The process for using this system is documented in more detail in `requirements/README.md`. -* **Caching of virtualenvs and packages**. To make updating the +- **Caching of virtualenvs and packages**. To make updating the dependencies of a Zulip installation efficient, we maintain a cache of virtualenvs named by the hash of the relevant `requirements.txt` file (`scripts/lib/hash_reqs.py`). These caches live under @@ -175,21 +175,21 @@ highlighting. The system is largely managed by the code in needed, making small version upgrades extremely efficient. And finally, we use `pip`'s built-in caching to ensure that a specific version of a specific package is only downloaded once. -* **Garbage-collecting caches**. We have a tool, +- **Garbage-collecting caches**. We have a tool, `scripts/lib/clean_venv_cache.py`, which will clean old cached virtualenvs that are no longer in use. In production, the algorithm preserves recent virtualenvs as well as those in use by any current production deployment directory under `/home/zulip/deployments/`. This helps ensure that a Zulip installation doesn't leak large amounts of disk over time. -* **Scripts**. Often, we want a script running in production to use +- **Scripts**. Often, we want a script running in production to use the Zulip virtualenv. To make that work without a lot of duplicated code, we have a helpful function, `scripts.lib.setup_path.setup_path`, which on import will put the currently running Python script into the Zulip virtualenv. This is called by `./manage.py` to ensure that our Django code always uses the correct virtualenv as well. -* **Mypy type checker**. Because we're using mypy in a strict mode, +- **Mypy type checker**. Because we're using mypy in a strict mode, when you add use of a new Python dependency, you usually need to either adds stubs to the `stubs/` directory for the library, or edit `mypy.ini` in the root of the Zulip project to configure @@ -212,19 +212,19 @@ We use the same set of strategies described for Python dependencies for most of our JavaScript dependencies, so we won't repeat the reasoning here. -* In a fashion very analogous to the Python codebase, +- In a fashion very analogous to the Python codebase, `scripts/lib/node_cache.py` manages cached `node_modules` directories in `/srv/zulip-npm-cache`. Each is named by its hash, computed by the `generate_sha1sum_node_modules` function. `scripts/lib/clean_node_cache.py` handles garbage-collection. -* We use [yarn][], a `pip`-like tool for JavaScript, to download most +- We use [yarn][], a `pip`-like tool for JavaScript, to download most JavaScript dependencies. Yarn talks to standard the [npm][] repository. We use the standard `package.json` file to declare our direct dependencies, with sections for development and production. Yarn takes care of pinning the versions of indirect dependencies in the `yarn.lock` file; `yarn install` updates the `yarn.lock` files. -* `tools/update-prod-static`. This process is discussed in detail in +- `tools/update-prod-static`. This process is discussed in detail in the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline) article, but we don't use the `node_modules` directories directly in production. Instead, static assets are compiled using our static @@ -233,7 +233,7 @@ reasoning here. directory in a Zulip production release tarball, which is a good thing, because doing so would more than double the size of a Zulip release tarball. -* **Checked-in packages**. In contrast with Python, we have a few +- **Checked-in packages**. In contrast with Python, we have a few JavaScript dependencies that we have copied into the main Zulip repository under `static/third`, often with patches. These date from an era before `npm` existed. It is a project goal to eliminate @@ -248,11 +248,11 @@ its version) and `scripts/lib/third/install-yarn.sh` (the standard installer for `yarn`, modified to support installing to a path that is not the current user's home directory). -* `nvm` has its own system for installing each version of `node` at +- `nvm` has its own system for installing each version of `node` at its own path, which we use, though we install a `/usr/local/bin/node` wrapper to access the desired version conveniently and efficiently (`nvm` has a lot of startup overhead). -* `install-yarn.sh` is configured to install `yarn` at +- `install-yarn.sh` is configured to install `yarn` at `/srv/zulip-yarn`. We don't do anything special to try to manage multiple versions of `yarn`. @@ -305,12 +305,12 @@ our JavaScript Markdown processor has access to the supported list. When making changes to Zulip's provisioning process or dependencies, usually one needs to think about making changes in 3 places: -* `tools/lib/provision.py`. This is the main provisioning script, +- `tools/lib/provision.py`. This is the main provisioning script, used by most developers to maintain their development environment. -* `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation" +- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation" documentation. Strategically, we'd like to move the support for more versions of Linux from here into `tools/lib/provision.py`. -* Production. Our tools for compiling/generating static assets need +- Production. Our tools for compiling/generating static assets need to be called from `tools/update-prod-static`, which is called by `tools/build-release-tarball` (for doing Zulip releases) as well as `tools/upgrade-zulip-from-git` (for deploying a Zulip server off of diff --git a/docs/subsystems/django-upgrades.md b/docs/subsystems/django-upgrades.md index 01004f7d80..55fe806816 100644 --- a/docs/subsystems/django-upgrades.md +++ b/docs/subsystems/django-upgrades.md @@ -3,30 +3,30 @@ This article documents notes on the process for upgrading Zulip to new major versions of Django. Here are the steps: -* Carefully read the Django upstream changelog, and `git grep` to +- Carefully read the Django upstream changelog, and `git grep` to check if we're using anything deprecated or significantly modified and put them in an issue (and then starting working through them). Also, note any new features we might want to use after the upgrade, and open an issue listing them; [example](https://github.com/zulip/zulip/issues/2564). -* Start submitting PRs to do any deprecation-type migrations that work +- Start submitting PRs to do any deprecation-type migrations that work on both the old and new version of Django. The goal here is to have the actual cutover commit be as small as possible, and to test as much of the changes for the migration as we can independently from the big cutover. -* Check the version support of the third-party Django packages we use +- Check the version support of the third-party Django packages we use (`git grep django requirements/` to see a list), upgrade any as needed and file bugs upstream for any that lack support. Look into fixing said bugs. -* Look at the pieces of Django code that we've copied and then +- Look at the pieces of Django code that we've copied and then adapted, and confirm whether Django has any updates to the modified code we should apply. Partial list: - * `CursorDebugWrapper`, which we have a modified version of in + - `CursorDebugWrapper`, which we have a modified version of in `zerver/lib/db.py`. See [the issue for contributing this upstream](https://github.com/zulip/zulip/issues/974) - * `PasswordResetForm` and any other forms we import from + - `PasswordResetForm` and any other forms we import from `django.contrib.auth.forms` in `zerver/forms.py` (which has all of our Django forms). - * Our AsyncDjangoHandler class has some code copied from the core + - Our AsyncDjangoHandler class has some code copied from the core Django handlers code; look at whether that code was changed in Django upstream. diff --git a/docs/subsystems/email.md b/docs/subsystems/email.md index 0d7fd1754c..490e1b7b0a 100644 --- a/docs/subsystems/email.md +++ b/docs/subsystems/email.md @@ -11,32 +11,32 @@ our instructions for On to the documentation. Zulip's email system is fairly straightforward, with only a few things you need to know to get started. -* All email templates are in `templates/zerver/emails/`. Each email has three +- All email templates are in `templates/zerver/emails/`. Each email has three template files: `.subject.txt`, `.txt`, and `.source.html`. Email templates, along with all other templates in the `templates/` directory, are Jinja2 templates. -* Most of the CSS and HTML layout for emails is in `email_base.html`. Note +- Most of the CSS and HTML layout for emails is in `email_base.html`. Note that email has to ship with all of its CSS and HTML, so nothing in `static/` is useful for an email. If you're adding new CSS or HTML for an email, there's a decent chance it should go in `email_base.html`. -* All email is eventually sent by `zerver.lib.send_email.send_email`. There +- All email is eventually sent by `zerver.lib.send_email.send_email`. There are several other functions in `zerver.lib.send_email`, but all of them eventually call the `send_email` function. The most interesting one is `send_future_email`. The `ScheduledEmail` entries are eventually processed by a supervisor job that runs `zerver/management/commands/deliver_scheduled_emails.py`. -* Always use `user_profile.delivery_email`, not `user_profile.email`, +- Always use `user_profile.delivery_email`, not `user_profile.email`, when passing data into the `send_email` library. The `user_profile.email` field may not always be valid. -* A good way to find a bunch of example email pathways is to `git grep` for +- A good way to find a bunch of example email pathways is to `git grep` for `zerver/emails` in the `zerver/` directory. One slightly complicated decision you may have to make when adding an email is figuring out how to schedule it. There are 3 ways to schedule email. -* Send it immediately, in the current Django process, e.g. by calling +- Send it immediately, in the current Django process, e.g. by calling `send_email` directly. An example of this is the `confirm_registration` email. -* Add it to a queue. An example is the `invitation` email. -* Send it (approximately) at a specified time in the future, using +- Add it to a queue. An example is the `invitation` email. +- Send it (approximately) at a specified time in the future, using `send_future_email`. An example is the `followup_day2` email. Email takes about a quarter second per email to process and send. Generally @@ -53,10 +53,10 @@ we've set the email backend (aka what happens when you call the email `.send()` method in Django) in the development environment to be our custom backend, `EmailLogBackEnd`. It does the following: -* Logs any sent emails to `var/log/email_content.log`. This log is +- Logs any sent emails to `var/log/email_content.log`. This log is displayed by the `/emails` endpoint (e.g. http://zulip.zulipdev.com:9991/emails). -* Print a friendly message on console advertising `/emails` to make +- Print a friendly message on console advertising `/emails` to make this nice and discoverable. ### Testing in a real email client @@ -81,11 +81,11 @@ Once you have the login credentials of the SMTP provider, since there is not `/etc/zulip/settings.py` in development, configure it using the following keys in `zproject/dev-secrets.conf` -* `email_host` - SMTP hostname. -* `email_port` - SMTP port. -* `email_host_user` - Username of the SMTP user -* `email_password` - Password of the SMTP user. -* `email_use_tls` - Set to `true` for most providers. Else, don't set any value. +- `email_host` - SMTP hostname. +- `email_port` - SMTP port. +- `email_host_user` - Username of the SMTP user +- `email_password` - Password of the SMTP user. +- `email_use_tls` - Set to `true` for most providers. Else, don't set any value. Here is an example of how `zproject/dev-secrets.conf` might look if you are using Gmail. @@ -103,18 +103,18 @@ email_password = gmail_password ### Notes -* After changing any HTML email or `email_base.html`, you need to run +- After changing any HTML email or `email_base.html`, you need to run `scripts/setup/inline_email_css.py` for the changes to be reflected in the development environment. The script generates files like `templates/zerver/emails/compiled/.html`. -* Images won't be displayed in a real email client unless you change +- Images won't be displayed in a real email client unless you change the `base_image_uri` used for emails to a public URL such as `https://chat.zulip.org/static/images/emails` (image links to `localhost:9991` aren't allowed by modern email providers). See `zproject/email_backends.py` for more details. -* While running the backend test suite, we use +- While running the backend test suite, we use `django.core.mail.backends.locmem.EmailBackend` as the email backend. The `locmem` backend stores messages in a special attribute of the django.core.mail module, "outbox". The outbox attribute is diff --git a/docs/subsystems/emoji.md b/docs/subsystems/emoji.md index a0cae95d9c..cbcf4c029e 100644 --- a/docs/subsystems/emoji.md +++ b/docs/subsystems/emoji.md @@ -6,10 +6,10 @@ document discusses a number of these issues. Currently, Zulip supports these four display formats for emoji: -* Google modern -* Google classic -* Twitter -* Plain text +- Google modern +- Google classic +- Twitter +- Plain text ## Emoji codes @@ -68,15 +68,15 @@ emoji tooling). See [our dependencies document](../subsystems/dependencies.md) for more details on this strategy. The emoji tree generated by this process contains several import elements: -* `emoji_codes.json`: A set of mappings used by the Zulip frontend to +- `emoji_codes.json`: A set of mappings used by the Zulip frontend to understand what Unicode emoji exist and what their shortnames are, used for autocomplete, emoji pickers, etc. This has been deduplicated using the logic in `tools/setup/emoji/emoji_setup_utils.py` to generally only have `:angry:` and not also `:angry_face:`, since having both is ugly and pointless for purposes like autocomplete and emoji pickers. -* `images/emoji/unicode/*.png`: A farm of emoji -* `images/emoji/*.png`: A farm of symlinks from emoji names to the +- `images/emoji/unicode/*.png`: A farm of emoji +- `images/emoji/*.png`: A farm of symlinks from emoji names to the `images/emoji/unicode/` tree. This is used to serve individual emoji images, as well as for the [backend Markdown processor](../subsystems/markdown.md) to know which emoji @@ -84,7 +84,7 @@ The emoji tree generated by this process contains several import elements: tree, we currently include all of the emoji in `emoji-map.json`; this means that if you send `:angry_face:`, it won't autocomplete, but will still work (but not in previews). -* Some CSS and PNGs for the emoji spritesheets, used in Zulip for +- Some CSS and PNGs for the emoji spritesheets, used in Zulip for emoji pickers where we would otherwise need to download over 1000 of individual emoji images (which would cause a browser performance problem). We have multiple spritesheets: one for each emoji @@ -102,36 +102,36 @@ The following set of considerations is not comprehensive, but has a few principles that were applied to the current set of names. We use (strong), (medium), and (weak) denote how strong a consideration it is. -* Even with over 1000 symbols, emoji feels surprisingly sparse as a language, +- Even with over 1000 symbols, emoji feels surprisingly sparse as a language, and more often than not, if you search for something, you don't find an appropriate emoji for it. So a primary goal for our set of names is to maximize the number of situations in which the user finds an emoji that feels appropriate. (strong) -* Conversely, we remove generic words that will gum up the typeahead. So +- Conversely, we remove generic words that will gum up the typeahead. So `:outbox:` instead of `:outbox_tray:`. Each word should count. (medium) -* We aim for the set of names to be as widely culturally applicable as +- We aim for the set of names to be as widely culturally applicable as possible, even if the glyphs are not. So `:statue:` instead of `:new_york:` for the statue of liberty, and `:tower:` instead of `:tokyo_tower:`. (strong) -* We remove unnecessary gender descriptions. So `:ok_signal:` instead of +- We remove unnecessary gender descriptions. So `:ok_signal:` instead of `:ok_woman:`. (strong) -* We don't add names that could be inappropriate in school or work +- We don't add names that could be inappropriate in school or work environments, even if the use is common on the internet. For example, we have not added `:butt:` for `:peach:`, or `:cheers:` for `:beers:`. (strong) -* Names should be compatible with the four emoji sets we support, but don't +- Names should be compatible with the four emoji sets we support, but don't have to be compatible with any other emoji set. (medium) -* We try not to use a creative canonical_name for emoji that are likely to +- We try not to use a creative canonical_name for emoji that are likely to be familiar to a large subset of users. This largely applies to certain faces. (medium) -* The set of names should be compatible with the iamcal, gemoji, and Unicode +- The set of names should be compatible with the iamcal, gemoji, and Unicode names. Compatible here means that if there is an emoji name a user knows from one of those sets, and the user searches for the key word of that name, they will get an emoji in our set. It is okay if this emoji has a @@ -142,26 +142,26 @@ Much of the work of picking names went into the first bullet above: making the emoji language less sparse. Some tricks and heuristics that were used for that: -* There are many near duplicates, like `:dog:` and `:dog_face:`, or +- There are many near duplicates, like `:dog:` and `:dog_face:`, or `:mailbox:`, `:mailbox_with_mail:`, and `:mailbox_with_no_mail:`. In these cases we repurpose the duplicates to be as useful as we can, like `:dog:` and `:puppy:`, and `:mailbox:`, `:unread_mail:`, `:inbox_zero:` for the ones above. There isn't a ton of flexibility, since we can't change the glyphs. But in most cases we have been able to come up with something. -* Many emoji have commonly understood meanings among people that use emoji a +- Many emoji have commonly understood meanings among people that use emoji a lot, and there are websites and articles that document some of these meanings. A commonly understood meaning can be a great thing to add as an alternate name, since often it is a sign that the meaning is addressing a real gap in the emoji system. -* Many emoji names are unnecessarily specific in iamcal/etc, like +- Many emoji names are unnecessarily specific in iamcal/etc, like `:flower_playing_cards:`, `:izakaya_lantern:`, or `:amphora:`. Renaming them to `:playing_cards:`, `:lantern:`, and `:vase:` makes them more widely usable. In such cases we often keep the specific name as an alternate. -* If there are natural things someone might type, like `:happy:`, we try to +- If there are natural things someone might type, like `:happy:`, we try to find an emoji to match. This extends to things that someone might not think to type, but as soon as someone in the organization discovers it it could get wide use, like `:working_on_it:`. Good future work would be to @@ -171,7 +171,7 @@ for that: Other notes -* Occasionally there are near duplicates where we don't have ideas for +- Occasionally there are near duplicates where we don't have ideas for useful names for the second one. In that case we sometimes remove the emoji rather than have two nearly identical glyphs in the emoji picker and typeahead. For instance, we kept `:spiral_notepad:` and dropped @@ -179,7 +179,7 @@ Other notes of glyphs look very different, we'll find two names that allow them both to stay. -* We removed many of the moons and clocks, to make the typeahead experience +- We removed many of the moons and clocks, to make the typeahead experience better when searching for something that catches all the moons or all the clocks. We kept all the squares and diamonds and other shapes, even though they have the same problem, since they are commonly used to make emoji art diff --git a/docs/subsystems/events-system.md b/docs/subsystems/events-system.md index 3cb76e20b1..f1b5f271fb 100644 --- a/docs/subsystems/events-system.md +++ b/docs/subsystems/events-system.md @@ -34,11 +34,11 @@ little notification that the operation succeeded). Architecturally, there are a few things needed to make a successful real-time sync system work: -* **Generation**. Generating events when changes happen to data, and +- **Generation**. Generating events when changes happen to data, and determining which users should receive each event. -* **Delivery**. Efficiently delivering those events to interested +- **Delivery**. Efficiently delivering those events to interested clients, ideally in an exactly-once fashion. -* **UI updates**. Updating the UI in the client once it has received +- **UI updates**. Updating the UI in the client once it has received events from the server. Reactive JavaScript libraries like React and Vue can help simplify the @@ -64,10 +64,10 @@ to be consumed by the delivery system. Usually, this list of users is one of 3 things: -* A single user (e.g. for user-level settings changes). -* Everyone in the realm (e.g. for organization-level settings changes, +- A single user (e.g. for user-level settings changes). +- Everyone in the realm (e.g. for organization-level settings changes, like new realm emoji). -* Everyone who would receive a given message (for messages, emoji +- Everyone who would receive a given message (for messages, emoji reactions, message editing, etc.); i.e. the subscribers to a stream or the people on a private message thread. @@ -175,10 +175,10 @@ anyway). When a client starts up, it usually wants to get 2 things from the server: -* The "current state" of various pieces of data, e.g. the current +- The "current state" of various pieces of data, e.g. the current settings, set of users in the organization (for typeahead), stream, messages, etc. (aka the "initial state"). -* A subscription to receive updates to those data when they are +- A subscription to receive updates to those data when they are changed by a client (aka an event queue). Ideally, one would get those two things atomically, i.e. if some other @@ -203,15 +203,15 @@ subroutines. Here's how it works when you make a `register` API request; the logic is in `zerver/views/events_register.py` and `zerver/lib/events.py`. The request is directly handled by Django: -* Django makes an HTTP request to Tornado, requesting that a new event +- Django makes an HTTP request to Tornado, requesting that a new event queue be created, and records its queue ID. -* Django does all the various database/cache/etc. queries to fetch the +- Django does all the various database/cache/etc. queries to fetch the data, non-atomically, from the various data sources (see the `fetch_initial_state_data` function). -* Django makes a second HTTP request to Tornado, requesting any events +- Django makes a second HTTP request to Tornado, requesting any events that had been added to the Tornado event queue since it was created. -* Finally, Django "applies" the events (see the `apply_events` +- Finally, Django "applies" the events (see the `apply_events` function) to the initial state that it fetched. E.g. for a name change event, it finds the user data in the `realm_user` data structure, and updates it to have the new name. @@ -245,9 +245,9 @@ The real trick is debugging these tests. The test example above has three things going on: -* Set up some data (`get_stream`) -* Call `verify_action` with an action function (`do_add_default_stream`) -* Use a schema checker to validate data (`check_default_streams`) +- Set up some data (`get_stream`) +- Call `verify_action` with an action function (`do_add_default_stream`) +- Use a schema checker to validate data (`check_default_streams`) #### verify_action @@ -265,15 +265,15 @@ action and then fetching a fresh copy of the state. In particular, `verify_action` does the following: -* Call `fetch_initial_state_data` to get the current state. -* Call the action function (e.g. `do_add_default_stream`). -* Capture the events generated by the action function. -* Check the events generated are documented in the [OpenAPI +- Call `fetch_initial_state_data` to get the current state. +- Call the action function (e.g. `do_add_default_stream`). +- Capture the events generated by the action function. +- Check the events generated are documented in the [OpenAPI schema](../documentation/api.md) defined in `zerver/openapi/zulip.yaml`. -* Call `apply_events(state, events)`, to get the resulting "hybrid state". -* Call `fetch_initial_state_data` again to get the "normal state". -* Compare the two results. +- Call `apply_events(state, events)`, to get the resulting "hybrid state". +- Call `fetch_initial_state_data` again to get the "normal state". +- Compare the two results. In the event that you wrote the `apply_events` logic correctly the first time, then the two states will be identical, and the @@ -300,16 +300,16 @@ events = self.verify_action(lambda: do_add_default_stream(stream)) There are some notable optional parameters for `verify_action`: -* `state_change_expected` must be set to `False` if your action +- `state_change_expected` must be set to `False` if your action doesn't actually require state changes for some reason; otherwise, `verify_action` will complain that your test doesn't really exercise any `apply_events` logic. Typing notifications (which are ephemereal) are a common place where we use this. -* `num_events` will tell `verify_action` how many events the +- `num_events` will tell `verify_action` how many events the `hamlet` user will receive after the action (the default is 1). -* parameters such as `client_gravatar` and `slim_presence` get +- parameters such as `client_gravatar` and `slim_presence` get passed along to `fetch_initial_state_data` (and it's important to test both boolean values of these parameters for relevant actions). diff --git a/docs/subsystems/hashchange-system.md b/docs/subsystems/hashchange-system.md index f82b029772..3a67c183fe 100644 --- a/docs/subsystems/hashchange-system.md +++ b/docs/subsystems/hashchange-system.md @@ -7,12 +7,12 @@ be used to deep-link into the application and allow the browser's "back" functionality to let the user navigate between parts of the UI. Some examples are: -* `/#settings/your-bots`: Bots section of the settings overlay. -* `/#streams`: Streams overlay, where the user manages streams +- `/#settings/your-bots`: Bots section of the settings overlay. +- `/#streams`: Streams overlay, where the user manages streams (subscription etc.) -* `/#streams/11/announce`: Streams overlay with stream ID 11 (called +- `/#streams/11/announce`: Streams overlay with stream ID 11 (called "announce") selected. -* `/#narrow/stream/42-android/topic/fun`: Message feed showing stream +- `/#narrow/stream/42-android/topic/fun`: Message feed showing stream "android" and topic "fun". (The `42` represents the id of the stream. @@ -22,27 +22,27 @@ code), which is unfortunately one of our thorniest modules. Part of the reason that it's thorny is that it needs to support a lot of different flows: -* The user clicking on an in-app link, which in turn opens an overlay. +- The user clicking on an in-app link, which in turn opens an overlay. For example the streams overlay opens when the user clicks the small cog symbol on the left sidebar, which is in fact a link to `/#streams`. This makes it easy to have simple links around the app without custom click handlers for each one. -* The user uses the "back" button in their browser (basically +- The user uses the "back" button in their browser (basically equivalent to the previous one, as a *link* out of the browser history will be visited). -* The user clicking some in-app click handler (e.g. "Stream settings" +- The user clicking some in-app click handler (e.g. "Stream settings" for an individual stream), that potentially does several UI-manipulating things including e.g. loading the streams overlay, and needs to update the hash without re-triggering the open animation (etc.). -* Within an overlay like the streams overlay, the user clicks to +- Within an overlay like the streams overlay, the user clicks to another part of the overlay, which should update the hash but not re-trigger loading the overlay (which would result in a confusing animation experience). -* The user is in a part of the webapp, and reloads their browser window. +- The user is in a part of the webapp, and reloads their browser window. Ideally the reloaded browser window should return them to their original state. -* A server-initiated browser reload (done after a new version is +- A server-initiated browser reload (done after a new version is deployed, or when a user comes back after being idle for a while, see [notes below][self-server-reloads]), where we try to preserve extra state (e.g. content of compose box, scroll position within a @@ -56,21 +56,21 @@ that it's easy to accidentally break something. The main external API lives in `static/js/browser_history.js`: -* `browser_history.update` is used to update the browser +- `browser_history.update` is used to update the browser history, and it should be called when the app code is taking care of updating the UI directly -* `browser_history.go_to_location` is used when you want the `hashchange` +- `browser_history.go_to_location` is used when you want the `hashchange` module to actually dispatch building the next page Internally you have these functions: -* `hashchange.hashchanged` is the function used to handle the hash, +- `hashchange.hashchanged` is the function used to handle the hash, whether it's changed by the browser (e.g. by clicking on a link to a hash or using the back button) or triggered internally. -* `hashchange.do_hashchange_normal` handles most cases, like loading the main +- `hashchange.do_hashchange_normal` handles most cases, like loading the main page (but maybe with a specific URL if you are narrowed to a stream or topic or PMs, etc.). -* `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have +- `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have some minor complexity related to remembering the page from which the overlay was launched, as well as optimizing in-page transitions (i.e. don't close/re-open the overlay if you can @@ -81,7 +81,7 @@ Internally you have these functions: There are a few circumstances when the Zulip browser window needs to reload itself: -* If the browser has been offline for more than 10 minutes, the +- If the browser has been offline for more than 10 minutes, the browser's [event queue][events-system] will have been garbage-collected by the server, meaning the browser can no longer get real-time updates altogether. In this case, the browser @@ -89,7 +89,7 @@ reload itself: unsuspend callback (based on some clever time logic) that ensures we check immediately when a client unsuspends; grep for `watchdog` to see the code. -* If a new version of the server has been deployed, we want to reload +- If a new version of the server has been deployed, we want to reload the browser so that it will start running the latest code. However, we don't want server deploys to be disruptive. So, the backend preserves user-side event queues (etc.) and just pushes a special @@ -106,10 +106,10 @@ reload itself: Here are some key functions in the reload system: -* `reload.preserve_state` is called when a server-initiated browser +- `reload.preserve_state` is called when a server-initiated browser reload happens, and encodes a bunch of data like the current scroll position into the hash. -* `reload.initialize` handles restoring the preserved state after a +- `reload.initialize` handles restoring the preserved state after a reload where the hash starts with `/#reload`. ## All reloads diff --git a/docs/subsystems/html-css.md b/docs/subsystems/html-css.md index 8149eff968..8863e5cc39 100644 --- a/docs/subsystems/html-css.md +++ b/docs/subsystems/html-css.md @@ -62,12 +62,12 @@ browsers to make sure things look the same. ### Behavior -* Templates are automatically recompiled in development when the file +- Templates are automatically recompiled in development when the file is saved; a refresh of the page should be enough to display the latest version. You might need to do a hard refresh, as some browsers cache webpages. -* Variables can be used in templates. The variables available to the +- Variables can be used in templates. The variables available to the template are called the **context**. Passing the context to the HTML template sets the values of those variables to the value they were given in the context. The sections below contain specifics on how the @@ -84,10 +84,10 @@ found [here][jconditionals]. The context for Jinja2 templates is assembled from a few places: -* `zulip_default_context` in `zerver/context_processors.py`. This is +- `zulip_default_context` in `zerver/context_processors.py`. This is the default context available to all Jinja2 templates. -* As an argument in the `render` call in the relevant function that +- As an argument in the `render` call in the relevant function that renders the template. For example, if you want to find the context passed to `index.html`, you can do: @@ -98,7 +98,7 @@ zerver/views/home.py: response = render(request, 'zerver/app/index.html', The next line in the code being the context definition. -* `zproject/urls.py` for some fairly static pages that are rendered +- `zproject/urls.py` for some fairly static pages that are rendered using `TemplateView`, for example: ```python @@ -203,14 +203,14 @@ For your asset to be included in a development/production bundle, it needs to be accessible from one of the entry points defined either in `tools/webpack.assets.json` or `tools/webpack.dev-assets.json`. -* If you plan to only use the file within the app proper, and not on the login +- If you plan to only use the file within the app proper, and not on the login page or other standalone pages, put it in the `app` bundle by importing it in `static/js/bundles/app.js`. -* If it needs to be available both in the app and all +- If it needs to be available both in the app and all logged-out/portico pages, import it to `static/js/bundles/common.js` which itself is imported to the `app` and `common` bundles. -* If it's just used on a single standalone page which is only used in +- If it's just used on a single standalone page which is only used in a development environment (e.g. `/devlogin`) create a new entry point in `tools/webpack.dev-assets.json` or it's used in both production and development (e.g. `/stats`) create a new entry point @@ -224,10 +224,10 @@ If you want to test minified files in development, look for the ### How it works in production A few useful notes are: -* Zulip installs static assets in production in +- Zulip installs static assets in production in `/home/zulip/prod-static`. When a new version is deployed, before the server is restarted, files are copied into that directory. -* We use the VFL (versioned file layout) strategy, where each file in +- We use the VFL (versioned file layout) strategy, where each file in the codebase (e.g. `favicon.ico`) gets a new name (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each deployment, has a manifest file @@ -239,7 +239,7 @@ server is restarted, files are copied into that directory. deployment can't find their static assets. It also is necessary for any incremental rollout strategy where different clients get different versions of the site. -* Some paths for files (e.g. emoji) are stored in the +- Some paths for files (e.g. emoji) are stored in the `rendered_content` of past messages, and thus cannot be removed without breaking the rendering of old messages (or doing a mass-rerender of old messages). diff --git a/docs/subsystems/logging.md b/docs/subsystems/logging.md index 7e60b5f2c2..2c7859456f 100644 --- a/docs/subsystems/logging.md +++ b/docs/subsystems/logging.md @@ -20,19 +20,19 @@ is great for small installations. The [Django][django-errors] framework provides much of the infrastructure needed by our error reporting system: -* The ability to send emails to the server's administrators with any +- The ability to send emails to the server's administrators with any 500 errors, using the `mail_admins` function. We enhance these data with extra details (like what user was involved in the error) in `zerver/logging_handlers.py`, and then send them to the administrator in `zerver/lib/error_notify.py` (which also supports sending Zulips to a stream about production errors). -* The ability to rate-limit certain errors to avoid sending hundreds +- The ability to rate-limit certain errors to avoid sending hundreds of emails in an outage (see `_RateLimitFilter` in `zerver/lib/logging_util.py`) -* A nice framework for filtering passwords and other important user +- A nice framework for filtering passwords and other important user data from the exception details, which we use in `zerver/filters.py`. -* Middleware for handling `JsonableError`, our system for allowing +- Middleware for handling `JsonableError`, our system for allowing code anywhere in Django to report an API-facing `json_error` from anywhere in a view code path. @@ -84,18 +84,18 @@ can be extremely valuable for investigating performance problems. ``` The format of this output is: -* Timestamp -* Log level -* Logger name, abbreviated as "zr" for these Zulip request logs -* IP address -* HTTP method -* HTTP status code -* Time to process -* (Optional perf data details, e.g. database time/queries, memcached +- Timestamp +- Log level +- Logger name, abbreviated as "zr" for these Zulip request logs +- IP address +- HTTP method +- HTTP status code +- Time to process +- (Optional perf data details, e.g. database time/queries, memcached time/queries, Django process startup time, Markdown processing time, etc.) -* Endpoint/URL from zproject/urls.py -* "email via client" showing user account involved (if logged in) and +- Endpoint/URL from zproject/urls.py +- "email via client" showing user account involved (if logged in) and the type of client they used ("web", "Android", etc.). The performance data details are particularly useful for investigating @@ -124,33 +124,33 @@ might use). In development, this means displaying a highly visible overlay over the message view area, to make exceptions in testing a new feature hard to miss. -* Blueslip is implemented in `static/js/blueslip.js`. -* In order to capture essentially any error occurring in the browser, +- Blueslip is implemented in `static/js/blueslip.js`. +- In order to capture essentially any error occurring in the browser, Blueslip listens for the `error` event on `window`, and has methods for being manually triggered by Zulip JavaScript code for warnings and assertion failures. -* Blueslip keeps a log of all the notices it has received during a +- Blueslip keeps a log of all the notices it has received during a browser session, and includes them in reports to the server, so that one can see cases where exceptions chained together. You can print this log from the browser console using `blueslip = require("./static/js/blueslip"); blueslip.get_log()`. Blueslip supports several error levels: -* `throw new Error(…)`: For fatal errors that cannot be easily +- `throw new Error(…)`: For fatal errors that cannot be easily recovered from. We try to avoid using it, since it kills the current JS thread, rather than returning execution to the caller. -* `blueslip.error`: For logging of events that are definitely caused +- `blueslip.error`: For logging of events that are definitely caused by a bug and thus sufficiently important to be reported, but where we can handle the error without creating major user-facing problems (e.g. an exception when handling a presence update). -* `blueslip.warn`: For logging of events that are a problem but not +- `blueslip.warn`: For logging of events that are a problem but not important enough to send an email about in production. They are, however, highlighted in the JS console in development. -* `blueslip.log` (and `blueslip.info`): Logged to the JS console in +- `blueslip.log` (and `blueslip.info`): Logged to the JS console in development and also in the blueslip log in production. Useful for data that might help discern what state the browser was in during an error (e.g. whether the user was in a narrow). -* `blueslip.debug`: Similar to `blueslip.log`, but are not printed to +- `blueslip.debug`: Similar to `blueslip.log`, but are not printed to the JS console in development. ## Frontend performance reporting @@ -159,12 +159,12 @@ In order to make it easier to debug potential performance problems in the critically latency-sensitive message sending code pathway, we log and report to the server the following whenever a message is sent: -* The time the user triggered the message (aka the start time). -* The time the `send_message` response returned from the server. -* The time the message was received by the browser from the +- The time the user triggered the message (aka the start time). +- The time the `send_message` response returned from the server. +- The time the message was received by the browser from the `get_events` protocol (these last two race with each other). -* Whether the message was locally echoed. -* If so, whether there was a disparity between the echoed content and +- Whether the message was locally echoed. +- If so, whether there was a disparity between the echoed content and the server-rendered content, which can be used for statistics on how effective our [local echo system](../subsystems/markdown.md) is. @@ -173,9 +173,9 @@ The code is all in `zerver/lib/report.py` and `static/js/sent_messages.js`. We have similar reporting for the time it takes to narrow / switch to a new view: -* The time the action was initiated -* The time when the updated message feed was visible to the user -* The time when the browser was idle again after switching views +- The time the action was initiated +- The time when the updated message feed was visible to the user +- The time when the browser was idle again after switching views (intended to catch issues where we generate a lot of deferred work). [django-errors]: https://docs.djangoproject.com/en/2.2/howto/error-reporting/ diff --git a/docs/subsystems/management-commands.md b/docs/subsystems/management-commands.md index e9de1ce512..516412be21 100644 --- a/docs/subsystems/management-commands.md +++ b/docs/subsystems/management-commands.md @@ -13,20 +13,20 @@ While Zulip takes advantage of built-in Django management commands for things like managing Django migrations, we also have dozens that we've written for a range of purposes: -* Cron jobs to do regular updates, e.g. `update_analytics_counts.py`, +- Cron jobs to do regular updates, e.g. `update_analytics_counts.py`, `sync_ldap_user_data`, etc. -* Useful parts of provisioning or upgrading a Zulip development +- Useful parts of provisioning or upgrading a Zulip development environment or server, e.g. `makemessages`, `compilemessages`, `populate_db`, `fill_memcached_caches`, etc. -* The actual scripts run by supervisord to run the persistent +- The actual scripts run by supervisord to run the persistent processes in a Zulip server, e.g. `runtornado` and `process_queue`. -* For a sysadmin to verify a Zulip server's configuration during +- For a sysadmin to verify a Zulip server's configuration during installation, e.g. `checkconfig`, `send_test_email`. -* As the interface for doing those rare operations that don't have a +- As the interface for doing those rare operations that don't have a UI yet, e.g. `deactivate_realm`, `reactivate_realm`, `change_user_email` (for the case where the user doesn't control the old email address). -* For a sysadmin to easily interact with and script common possible +- For a sysadmin to easily interact with and script common possible changes they might want to make to the database on a Zulip server. E.g. `send_password_reset_email`, `export`, `purge_queue`. @@ -38,14 +38,14 @@ command to write a new one. Some good examples are is good, but we have a few pieces advice specific to the Zulip project. -* If you need to access a realm or user, use the `ZulipBaseCommand` +- If you need to access a realm or user, use the `ZulipBaseCommand` class in `zerver/lib/management.py` so you don't need to write the tedious code of looking those objects up. This is especially important for users, since the library handles the issues around looking up users by email well (if there's a unique user with that email, just modify it without requiring the user to specify the realm as well, but if there's a collision, throw a nice error). -* Avoid writing a lot of code in management commands; management +- Avoid writing a lot of code in management commands; management commands are annoying to unit test, and thus easier to maintain if all the interesting logic is in a nice function that is unit tested (and ideally, also used in Zulip's existing code). Look for code in diff --git a/docs/subsystems/markdown.md b/docs/subsystems/markdown.md index 15a5e7d6a0..76476183c6 100644 --- a/docs/subsystems/markdown.md +++ b/docs/subsystems/markdown.md @@ -49,18 +49,18 @@ by both test suites; as a result, it is the preferred place to add new tests for Zulip's Markdown system. Some important notes on reading this file: -* `expected_output` is the expected output for the backend Markdown +- `expected_output` is the expected output for the backend Markdown processor. -* When the frontend processor doesn't support a feature and it should +- When the frontend processor doesn't support a feature and it should just be rendered on the backend, we set `backend_only_rendering` to `true` in the fixtures; this will automatically verify that `markdown.contains_backend_only_syntax` rejects the syntax, ensuring it will be rendered only by the backend processor. -* When the two processors disagree, we set `marked_expected_output` in +- When the two processors disagree, we set `marked_expected_output` in the fixtures; this will ensure that the syntax stays that way. If the differences are important (i.e. not just whitespace), we should also open an issue on GitHub to track the problem. -* For mobile push notifications, we need a text version of the +- For mobile push notifications, we need a text version of the rendered content, since the APNS and GCM push notification systems don't support richer markup. Mostly, this involves stripping HTML, but there's some syntax we take special care with. Tests for what @@ -91,44 +91,44 @@ tests with `tools/test-js-with-node markdown` and backend tests with First, you will likely find these third-party resources helpful: -* **[Python-Markdown](https://pypi.python.org/pypi/Markdown)** is the Markdown +- **[Python-Markdown](https://pypi.python.org/pypi/Markdown)** is the Markdown library used by Zulip as a base to build our custom Markdown syntax upon. -* **[Python's XML ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)** +- **[Python's XML ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)** is the part of the Python standard library used by Python Markdown and any custom extensions to generate and modify the output HTML. When changing Zulip's Markdown syntax, you need to update several places: -* The backend Markdown processor (`zerver/lib/markdown/__init__.py`). -* The frontend Markdown processor (`static/js/markdown.js` and sometimes +- The backend Markdown processor (`zerver/lib/markdown/__init__.py`). +- The frontend Markdown processor (`static/js/markdown.js` and sometimes `static/third/marked/lib/marked.js`), or `markdown.contains_backend_only_syntax` if your changes won't be supported in the frontend processor. -* If desired, the typeahead logic in `static/js/composebox_typeahead.js`. -* The test suite, probably via adding entries to `zerver/tests/fixtures/markdown_test_cases.json`. -* The in-app Markdown documentation (`markdown_help_rows` in `static/js/info_overlay.js`). -* The list of changes to Markdown at the end of this document. +- If desired, the typeahead logic in `static/js/composebox_typeahead.js`. +- The test suite, probably via adding entries to `zerver/tests/fixtures/markdown_test_cases.json`. +- The in-app Markdown documentation (`markdown_help_rows` in `static/js/info_overlay.js`). +- The list of changes to Markdown at the end of this document. Important considerations for any changes are: -* Security: A bug in the Markdown processor can lead to XSS issues. +- Security: A bug in the Markdown processor can lead to XSS issues. For example, we should not insert unsanitized HTML from a third-party web application into a Zulip message. -* Uniqueness: We want to avoid users having a bad experience due to +- Uniqueness: We want to avoid users having a bad experience due to accidentally triggering Markdown syntax or typeahead that isn't related to what they are trying to express. -* Performance: Zulip can render a lot of messages very quickly, and +- Performance: Zulip can render a lot of messages very quickly, and we'd like to keep it that way. New regular expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about expensive computations or third-party API requests. -* Database: The backend Markdown processor runs inside a Python thread +- Database: The backend Markdown processor runs inside a Python thread (as part of how we implement timeouts for third-party API queries), and for that reason we currently should avoid making database queries inside the Markdown processor. This is a technical implementation detail that could be changed with a few days of work, but is an important detail to know about until we do that work. -* Testing: Every new feature should have both positive and negative +- Testing: Every new feature should have both positive and negative tests; they're easy to write and give us the flexibility to refactor frequently. @@ -177,14 +177,14 @@ chat product; even though you can edit messages to fix formatting mistakes, you don't want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip: -* What fraction of the time, if you pasted a short technical email +- What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it to render in a reasonable way? This is the "accidental Markdown syntax" problem, common with Markdown syntax like the italics syntax interacting with talking about `char *`s. -* What fraction of the time do users attempting to use a particular +- What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figure substantially. @@ -207,71 +207,71 @@ accurate. ### Basic syntax -* Enable `nl2br` extension: this means one newline creates a line +- Enable `nl2br` extension: this means one newline creates a line break (not paragraph break). -* Allow only `*` syntax for italics, not `_`. This resolves an issue where +- Allow only `*` syntax for italics, not `_`. This resolves an issue where people were using `_` and hitting it by mistake too often. Asterisks surrounded by spaces won't trigger italics, either (e.g. with stock Markdown `You should use char * instead of void * there` would produce undesired results). -* Allow only `**` syntax for bold, not `__` (easy to hit by mistake if +- Allow only `**` syntax for bold, not `__` (easy to hit by mistake if discussing Python `__init__` or something). -* Add `~~` syntax for strikethrough. +- Add `~~` syntax for strikethrough. -* Disable special use of `\` to escape other syntax. Rendering `\\` as +- Disable special use of `\` to escape other syntax. Rendering `\\` as `\` was hugely controversial, but having no escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks. ### Lists -* Allow tacking a bulleted list or block quote onto the end of a +- Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it. -* Allow only `*` for bulleted lists, not `+` or `-` (previously +- Allow only `*` for bulleted lists, not `+` or `-` (previously created confusion with diff-style text sloppily not included in a code block). -* Disable ordered list syntax: stock Markdown automatically renumbers, which +- Disable ordered list syntax: stock Markdown automatically renumbers, which can be really confusing when sending a numbered list across multiple messages. ### Links -* Enable auto-linkification, both for `http://...` and guessing at +- Enable auto-linkification, both for `http://...` and guessing at things like `t.co/foo`. -* Force links to be absolute. `[foo](google.com)` will go to +- Force links to be absolute. `[foo](google.com)` will go to `http://google.com`, and not `https://zulip.com/google.com` which is the default behavior. -* Set `title=`(the URL) on every link tag. +- Set `title=`(the URL) on every link tag. -* Disable link-by-reference syntax, +- Disable link-by-reference syntax, `[foo][bar]` ... `[bar]: https://google.com`. -* Enable linking to other streams using `#**streamName**`. +- Enable linking to other streams using `#**streamName**`. ### Code -* Enable fenced code block extension, with syntax highlighting. +- Enable fenced code block extension, with syntax highlighting. -* Disable line-numbering within fenced code blocks -- the `` +- Disable line-numbering within fenced code blocks -- the `
` output confused our web client code. ### Other -* Disable headings, both `# foo` and `== foo ==` syntax: they don't +- Disable headings, both `# foo` and `== foo ==` syntax: they don't make much sense for chat messages. -* Disabled images with `![]()` (images from links are shown as an inline +- Disabled images with `![]()` (images from links are shown as an inline preview). -* Allow embedding any avatar as a tiny (list bullet size) image. This +- Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integrations. -* We added the `~~~ quote` block quote syntax. +- We added the `~~~ quote` block quote syntax. diff --git a/docs/subsystems/notifications.md b/docs/subsystems/notifications.md index 480bce01c7..53cc9dcaf8 100644 --- a/docs/subsystems/notifications.md +++ b/docs/subsystems/notifications.md @@ -11,10 +11,10 @@ the details of the email/mobile push notifications code path. Here we name a few corner cases worth understanding in designing this sort of notifications system: -* The **idle desktop problem**: We don't want the presence of a +- The **idle desktop problem**: We don't want the presence of a desktop computer at the office to eat all notifications because the user has an "online" client that they may not have used in 3 days. -* The **hard disconnect problem**: A client can lose its connection to +- The **hard disconnect problem**: A client can lose its connection to the Internet (or be suspended, or whatever) at any time, and this happens routinely. We want to ensure that races where a user closes their laptop shortly after a notifiable message is sent does not @@ -25,16 +25,16 @@ sort of notifications system: As a reminder, the relevant part of the flow for sending messages is as follows: -* `do_send_messages` is the synchronous message-sending code path, +- `do_send_messages` is the synchronous message-sending code path, and passing the following data in its `send_event` call: - * Data about the message's content (E.g. mentions, wildcard + - Data about the message's content (E.g. mentions, wildcard mentions, and alert words) and encodes it into the `UserMessage` table's `flags` structure, which is in turn passed into `send_event` for each user receiving the message. - * Data about user configuration relevant to the message, such as + - Data about user configuration relevant to the message, such as `push_notify_user_ids` and `stream_notify_user_ids`, are included alongside `flags` in the per-user data structure. - * The `presence_idle_user_ids` set, containing the subset of + - The `presence_idle_user_ids` set, containing the subset of recipient users who are mentioned, are PM recipients, have alert words, or otherwise would normally get a notification, but have not interacted with a Zulip client in the last few minutes. (Users who @@ -43,7 +43,7 @@ as follows: structure ignores users for whom the message is not notifiable, which is important to avoid this being thousands of `user_ids` for messages to large streams with few currently active users. -* The Tornado [event queue system](../subsystems/events-system.md) +- The Tornado [event queue system](../subsystems/events-system.md) processes that data, as well as data about each user's active event queues, to (1) push an event to each queue needing that message and (2) for notifiable messages, pushing an event onto the @@ -51,65 +51,65 @@ as follows: queues. This important message-processing logic has notable extra logic not present when processing normal events, both for details like splicing `flags` to customize event payloads per-user, as well. - * The Tornado system determines whether the user is "offline/idle". + - The Tornado system determines whether the user is "offline/idle". Zulip's email notifications are designed to not fire when the user is actively using Zulip to avoid spam, and this is where those checks are implemented. - * Users in `presence_idle_user_ids` are always considered idle: + - Users in `presence_idle_user_ids` are always considered idle: the variable name means "users who are idle because of presence". This is how we solve the idle desktop problem; users with an idle desktop are treated the same as users who aren't logged in for this check. - * However, that check does not handle the hard disconnect problem: + - However, that check does not handle the hard disconnect problem: if a user was present 1 minute before a message was sent, and then closed their laptop, the user will not be in `presence_idle_user_ids`, and so without an additional mechanism, messages sent shortly after a user leaves would never trigger a notification (!). - * We solve that problem by also notifying if + - We solve that problem by also notifying if `receiver_is_off_zulip` returns `True`, which checks whether the user has any current events system clients registered to receive `message` events. This check is done immediately (handling soft disconnects, where E.g. the user closes their last Zulip tab and we get the `DELETE /events/{queue_id}` request). - * The `receiver_is_off_zulip` check is effectively repeated when + - The `receiver_is_off_zulip` check is effectively repeated when event queues are garbage-collected (in `missedmessage_hook`) by looking for whether the queue being garbage-collectee was the only one; this second check solves the hard disconnect problem, resulting in notifications for these hard-disconnect cases usually coming 10 minutes late. - * The message-edit code path has parallel logic in + - The message-edit code path has parallel logic in `maybe_enqueue_notifications_for_message_update` for triggering notifications in cases like a mention added during message editing. - * The business logic for all these notification decisions made + - The business logic for all these notification decisions made inside Tornado has extensive automated test suites; e.g. `test_message_edit_notifications.py` covers all the cases around editing a message to add/remove a mention. - * We may in the future want to add some sort of system for letting + - We may in the future want to add some sort of system for letting users see past notifications, to help with explaining and debugging this system, since it has so much complexity. -* Desktop notifications are the simplest; they are implemented +- Desktop notifications are the simplest; they are implemented client-side by the web/desktop app's logic (`static/js/notifications.js`) inspecting the `flags` fields that were spliced into `message` events by the Tornado system, as well as the user's notification settings. -* The queue processors for those queues make the final determination +- The queue processors for those queues make the final determination for whether to send a notification, and do the work to generate an email (`zerver/lib/email_notifications.py`) or mobile (`zerver/lib/push_notifications.py`) notification. We'll detail this process in more detail for each system below, but it's important to know that it's normal for a message to sit in these queues for minutes (and in the future, possibly hours). -* Both queue processor code paths do additional filtering before +- Both queue processor code paths do additional filtering before sending a notification: - * Messages that have already been marked as read by the user before + - Messages that have already been marked as read by the user before the queue processor runs never trigger a notification. - * Messages that were already deleted never trigger a notification. - * The user-level settings for whether email/mobile notifications are + - Messages that were already deleted never trigger a notification. + - The user-level settings for whether email/mobile notifications are disabled are rechecked, as the user may have disabled one of these settings during the queuing period. - * The **Email notifications queue processor**, `MissedMessageWorker`, + - The **Email notifications queue processor**, `MissedMessageWorker`, takes care to wait for 2 minutes (hopefully in the future this will be a configuration setting) and starts a thread to batch together multiple messages into a single email. These features are unnecessary @@ -119,7 +119,7 @@ as follows: to GitHub's email notifications, with a clean, simple design that makes replying from an email client possible (using the [incoming email integration](../production/email-gateway.md)). - * The **Push notifications queue processor**, + - The **Push notifications queue processor**, `PushNotificationsWorker`, is a simple wrapper around the `push_notifications.py` code that actually sends the notification. This logic is somewhat complicated by having to track @@ -131,27 +131,27 @@ as follows: The following important constraints are worth understanding about the structure of the system, when thinking about changes to it: -* **Bulk database queries** are much more efficient for checking +- **Bulk database queries** are much more efficient for checking details from the database like "which users receiving this message are online". -* **Thousands of users**. Zulip supports thousands of users, and we +- **Thousands of users**. Zulip supports thousands of users, and we want to avoid `send_event()` pushing large amounts of per-user data to Tornado via RabbitMQ for scalability reasons. -* **Tornado doesn't do database queries**. Because the Tornado system +- **Tornado doesn't do database queries**. Because the Tornado system is an asynchronous event-driven framework, and our Django database library is synchronous, database queries are very expensive. So these queries need to be done in either `do_send_messages` or the queue processor logic. (For example, this means `presence` data should be checked in either `do_send_messages` or the queue processors, not in Tornado). -* **Future configuration**. Notification settings are an area that we +- **Future configuration**. Notification settings are an area that we expect to only expand with time, with upcoming features like following a topic (to get notifications for messages only within that topic in a stream). There are a lot of different workflows possible with Zulip's threading, and it's important to make it easy for users to set up Zulip's notification to fit as many of those workflows as possible. -* **Message editing**. Zulip supports editing messages, and that +- **Message editing**. Zulip supports editing messages, and that interacts with notifications in ways that require careful handling: Notifications should have the latest edited content (users often fix typos 30 seconds after diff --git a/docs/subsystems/performance.md b/docs/subsystems/performance.md index 574a7c1da9..b36e115c11 100644 --- a/docs/subsystems/performance.md +++ b/docs/subsystems/performance.md @@ -7,13 +7,13 @@ workload of usage without performance materially degrading. First, a few notes on philosophy. -* We consider it an important technical goal for Zulip to be fast, +- We consider it an important technical goal for Zulip to be fast, because that's an important part of user experience for a real-time collaboration tool like Zulip. Many UI features in the Zulip webapp are designed to load instantly, because all the data required for them is present in the initial HTTP response, and both the Zulip API and webapp are architected around that strategy. -* The Zulip database model and server implementation are carefully +- The Zulip database model and server implementation are carefully designed to ensure that every common operation is efficient, with automated tests designed to prevent the accidental introductions of inefficient or excessive database queries. We much prefer doing @@ -29,7 +29,7 @@ important to understand the load profiles for production uses. Zulip servers typically involve a mixture of two very different types of load profiles: -* Open communities like open source projects, online classes, +- Open communities like open source projects, online classes, etc. have large numbers of users, many of whom are idle. (Many of the others likely stopped by to ask a question, got it answered, and then didn't need the community again for the next year). Our own @@ -40,7 +40,7 @@ of load profiles: deactivation](../subsystems/sending-messages.html#soft-deactivation) to ensure idle users have minimal impact on both server-side scalability and request latency. -* Fulltime teams, like your typical corporate Zulip installation, +- Fulltime teams, like your typical corporate Zulip installation, have users who are mostly active for multiple hours a day and sending a high volume of messages each. This load profile is most important for self-hosted servers, since many of those are used exclusively by @@ -191,13 +191,13 @@ who have a lot of latency to the server. There are only a few exceptions where we fetch data in a separate AJAX request after page load: -* Message history is managed separately; this is why the Zulip webapp will +- Message history is managed separately; this is why the Zulip webapp will first render the entire site except for the middle panel, and then a moment later render the middle panel (showing the message history). -* A few very rarely accessed data sets like [message edit +- A few very rarely accessed data sets like [message edit history](https://zulip.com/help/view-a-messages-edit-history) are only fetched on demand. -* A few data sets that are only required for administrative settings +- A few data sets that are only required for administrative settings pages are fetched only when loading those parts of the UI. Requests to `GET /` and `/api/v1/register` that fetch `page_params` @@ -237,16 +237,16 @@ Bulk requests for message content and metadata ~3% of total HTTP requests. The zulip webapp has a few major reasons it does a large number of these requests: -* Most of these requests are from users clicking into different views +- Most of these requests are from users clicking into different views -- to avoid certain subtle bugs, Zulip's webapp currently fetches content from the server even when it has the history for the relevant stream/topic cached locally. -* When a browser opens the Zulip webapp, it will eventually fetch and +- When a browser opens the Zulip webapp, it will eventually fetch and cache in the browser all messages newer than the oldest unread message in a non-muted context. This can be in total extremely expensive for users with 10,000s of unread messages, resulting in a single browser doing 100 of these requests. -* When a new version of the Zulip server is deployed, every browser +- When a new version of the Zulip server is deployed, every browser will reload within 30 minutes to ensure they are running the latest code. For installations that deploy often like chat.zulip.org and zulip.com, this can result in a thundering herd effect for both `/` @@ -269,11 +269,11 @@ per-request basis. However, we have technical designs for optimizing the overall frequency with which clients need to make these requests in two major ways: -* Improving [client-side +- Improving [client-side caching](https://github.com/zulip/zulip/issues/15131) to allow caching of narrows that the user has viewed in the current session, avoiding repeat fetches of message content during a given session. -* Adjusting the behavior for clients with 10,000s of unread messages +- Adjusting the behavior for clients with 10,000s of unread messages to not fetch as much old message history into the cache. See [this issue](https://github.com/zulip/zulip/issues/16697) for relevant design work. diff --git a/docs/subsystems/pointer.md b/docs/subsystems/pointer.md index dadd01390a..9094c844a5 100644 --- a/docs/subsystems/pointer.md +++ b/docs/subsystems/pointer.md @@ -17,10 +17,10 @@ make to the model. First a bit of terminology: -* "Narrowing" is the process of filtering to a particular subset of +- "Narrowing" is the process of filtering to a particular subset of the messages the user has access to. -* The blue cursor box (the "pointer") is around is called the +- The blue cursor box (the "pointer") is around is called the "selected" message. Zulip ensures that the currently selected message is always in-view. @@ -78,11 +78,11 @@ How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might use the product. The algorithm is as follows: -* Any message which is selected or above a message which is selected +- Any message which is selected or above a message which is selected is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. -* If the whitespace at the very bottom of the feed is in view, all +- If the whitespace at the very bottom of the feed is in view, all messages in view are marked as read. These two simple rules, combined with the pointer logic above, end up diff --git a/docs/subsystems/presence.md b/docs/subsystems/presence.md index ec67ec972b..dcefd6ce8b 100644 --- a/docs/subsystems/presence.md +++ b/docs/subsystems/presence.md @@ -22,9 +22,9 @@ A client should report to the server every minute a `POST` request to requests contains a few parameters. The most important is "status", which had 2 valid values: -* "active" -- this means the user has interacted with the client +- "active" -- this means the user has interacted with the client recently. We use this for the "green" state in the webapp. -* "idle" -- the user has not interacted with the client recently. +- "idle" -- the user has not interacted with the client recently. This is important for the case where a user left a Zulip tab open on their desktop at work and went home for the weekend. We use this for the "orange" state in the webapp. @@ -34,7 +34,7 @@ for each user, contains their status and timestamp that we last heard from that client. There are a few important details to understand about that data structure: -* It's really important that the timestamp is the last time we heard +- It's really important that the timestamp is the last time we heard from the client. A client can only interpret the status to display about another user by doing a simple computation using the (status, timestamp) pair. E.g. a user who last used Zulip 1 week ago will @@ -45,13 +45,13 @@ about that data structure: accurately compute whether that user is offline (even if the last data from the server was 45 seconds ago, and the user was last online 4:30 before the client received that server data). -* Users can disable their own presence updates in user settings +- Users can disable their own presence updates in user settings (`UserProfile.presence_enabled` is the flag storing [this user preference](https://zulip.com/help/status-and-availability#disable-updating-availability)). -* The `status_from_timestamp` function in `static/js/presence.js` is +- The `status_from_timestamp` function in `static/js/presence.js` is useful sample code; the `OFFLINE_THRESHOLD_SECS` check is critical to correct output. -* We provide the data for e.g. whether the user was online on their +- We provide the data for e.g. whether the user was online on their desktop or the mobile app, but for a basic client, you will likely only want to parse the "aggregated" key, which shows the summary answer for "is this user online". diff --git a/docs/subsystems/queuing.md b/docs/subsystems/queuing.md index f09e12183a..726fadf809 100644 --- a/docs/subsystems/queuing.md +++ b/docs/subsystems/queuing.md @@ -3,24 +3,24 @@ Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: -* Asynchronously doing expensive operations like sending email +- Asynchronously doing expensive operations like sending email notifications which can take seconds per email and thus would otherwise time out when 100s are triggered at once (E.g. inviting a lot of new users to a realm). -* Asynchronously doing non-time-critical somewhat expensive operations +- Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g. UserActivityInternal) which don't have any immediate runtime effect. -* Communicating events to push to clients (browsers, etc.) from the +- Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. -* Processing mobile push notifications and email mirroring system +- Processing mobile push notifications and email mirroring system messages. -* Processing various errors, frontend tracebacks, and slow database +- Processing various errors, frontend tracebacks, and slow database queries in a batched fashion. Needless to say, the RabbitMQ-based queuing system is an important @@ -35,7 +35,7 @@ custom integration defined in `zerver/lib/queue.py`. To add a new queue processor: -* Define the processor in `zerver/worker/queue_processors.py` using +- Define the processor in `zerver/worker/queue_processors.py` using the `@assign_queue` decorator; it's pretty easy to get the template for an existing similar queue processor. This suffices to test your queue worker in the Zulip development environment @@ -44,7 +44,7 @@ To add a new queue processor: a single queue processor manually using e.g. `./manage.py process_queue --queue=user_activity`. -* So that supervisord will know to run the queue processor in +- So that supervisord will know to run the queue processor in production, you will need to add to the `queues` variable in `puppet/zulip/manifests/app_frontend_base.pp`; the list there is used to generate `/etc/supervisor/conf.d/zulip.conf`. diff --git a/docs/subsystems/realms.md b/docs/subsystems/realms.md index 9dd688a561..63985b41f3 100644 --- a/docs/subsystems/realms.md +++ b/docs/subsystems/realms.md @@ -19,8 +19,8 @@ are also relevant reading. There are two main methods for creating realms. -* Using unique link generator -* Enabling open realm creation +- Using unique link generator +- Enabling open realm creation #### Using unique link generator diff --git a/docs/subsystems/release-checklist.md b/docs/subsystems/release-checklist.md index 6cfdf2a549..9ecd642050 100644 --- a/docs/subsystems/release-checklist.md +++ b/docs/subsystems/release-checklist.md @@ -5,91 +5,91 @@ preparing a new release. ### A week before the release -* For a major release (e.g. 4.0): - * Upgrade all Python dependencies in +- For a major release (e.g. 4.0): + - Upgrade all Python dependencies in `requirements` to latest upstream versions so they can burn in (use `pip list --outdated`). - * [Upload strings to + - [Upload strings to Transifex](../translating/internationalization.html#translation-process) using `push-translations`. Post a Transifex [Announcement](https://www.transifex.com/zulip/zulip/announcements/) notifying translators that we're approaching a release. - * Merge draft updates to the [changelog](../overview/changelog.md) + - Merge draft updates to the [changelog](../overview/changelog.md) with changes since the last release. While doing so, take notes on things that might need follow-up work or documentation before we can happily advertise them in a release blog post. - * Inspect all `TODO/compatibility` comments for whether we can + - Inspect all `TODO/compatibility` comments for whether we can remove any backwards-compatibility code in this release. -* Create a burn-down list of issues that need to be fixed before we can +- Create a burn-down list of issues that need to be fixed before we can release, and make sure all of them are being worked on. -* Draft the release blog post (a.k.a. the release notes) in Paper. In +- Draft the release blog post (a.k.a. the release notes) in Paper. In it, list the important changes in the release, from most to least notable. ### Final release preparation -* Update the Paper blog post draft with any new commits. -* _Except minor releases:_ Download updated translation strings from +- Update the Paper blog post draft with any new commits. +- _Except minor releases:_ Download updated translation strings from Transifex and commit them. -* Use `build-release-tarball` to generate a release tarball. -* Test the new tarball extensively, both new install and upgrade from last +- Use `build-release-tarball` to generate a release tarball. +- Test the new tarball extensively, both new install and upgrade from last release, on both Bionic and Focal. -* Repeat until release is ready. -* Send around the Paper blog post draft for review. -* Move the blog post draft to Ghost. (For a draft in Dropbox Paper, +- Repeat until release is ready. +- Send around the Paper blog post draft for review. +- Move the blog post draft to Ghost. (For a draft in Dropbox Paper, use "··· > Export > Markdown" to get a pretty good markup conversion.) Proofread the post, especially for formatting. Tag the post with "Release announcements" in Ghost. ### Executing the release -* Create the release commit, on `main` (for major releases) or on the +- Create the release commit, on `main` (for major releases) or on the release branch (for minor releases): - * Copy the Markdown release notes for the release into + - Copy the Markdown release notes for the release into `docs/overview/changelog.md`. - * _Except minor releases:_ Adjust the `changelog.md` heading to have + - _Except minor releases:_ Adjust the `changelog.md` heading to have the stable release series boilerplate. - * Update `ZULIP_VERSION` and `LATEST_RELEASE_VERSION` in `version.py`. - * _Except minor releases:_ Update `API_FEATURE_LEVEL` to a feature + - Update `ZULIP_VERSION` and `LATEST_RELEASE_VERSION` in `version.py`. + - _Except minor releases:_ Update `API_FEATURE_LEVEL` to a feature level for the final release, and document a reserved range. -* Tag that commit with an unsigned Git tag named the release number. -* Use `build-release-tarball` to generate a final release tarball. -* Push the tag and release commit. -* Copy the tarball to `zulip.org`, and run +- Tag that commit with an unsigned Git tag named the release number. +- Use `build-release-tarball` to generate a final release tarball. +- Push the tag and release commit. +- Copy the tarball to `zulip.org`, and run `/etc/zulip/ship-release.sh` on it; this will put it in place, update the latest symlink, and update the SHA256 sums. -* Post the release by [editing the latest tag on +- Post the release by [editing the latest tag on GitHub](https://github.com/zulip/zulip/tags); use the text from `changelog.md` for the release notes. **Note:** This will trigger the [GitHub action](https://github.com/zulip/zulip/blob/main/tools/oneclickapps/README.md) for updating DigitalOcean one-click app image. The action uses the latest release tarball published on `zulip.org` for creating the image. -* Update the [Docker image](https://github.com/zulip/docker-zulip) and +- Update the [Docker image](https://github.com/zulip/docker-zulip) and do a release of that. -* Update the image of DigitalOcean one click app using +- Update the image of DigitalOcean one click app using [Fabric](https://github.com/zulip/marketplace-partners) and publish it to DO marketplace. -* Publish the blog post; check the box to "send by email." -* Email [zulip-announce](https://groups.google.com/g/zulip-announce), +- Publish the blog post; check the box to "send by email." +- Email [zulip-announce](https://groups.google.com/g/zulip-announce), post to [#announce](https://chat.zulip.org/#narrow/stream/1-announce), and [send a tweet](https://twitter.com/zulip). ### Post-release -* Following a major release (e.g. 4.0): - * Create a release branch (e.g. `4.x`). - * On the release branch, update `ZULIP_VERSION` in `version.py` to +- Following a major release (e.g. 4.0): + - Create a release branch (e.g. `4.x`). + - On the release branch, update `ZULIP_VERSION` in `version.py` to the present release with a `+git` suffix, e.g. `4.0+git`. - * On `main`, update `ZULIP_VERSION` to the future major release with + - On `main`, update `ZULIP_VERSION` to the future major release with a `-dev+git` suffix, e.g. `5.0-dev+git`. Make a Git tag for this update commit with a `-dev` suffix, e.g. `5.0-dev`. Push the tag to both zulip.git and zulip-internal.git to get a correct version number for future Cloud deployments. - * Consider removing a few old releases from ReadTheDocs; we keep about + - Consider removing a few old releases from ReadTheDocs; we keep about two years of back-versions. -* Following a minor release (e.g. 3.2), on the release branch: - * Update `ZULIP_VERSION` to the present release with a `+git` +- Following a minor release (e.g. 3.2), on the release branch: + - Update `ZULIP_VERSION` to the present release with a `+git` suffix, e.g. `3.2+git`. - * Update `LATEST_RELEASE_VERSION` with the released version. - * Cherry-pick the changelog changes back to `main`. + - Update `LATEST_RELEASE_VERSION` with the released version. + - Cherry-pick the changelog changes back to `main`. diff --git a/docs/subsystems/schema-migrations.md b/docs/subsystems/schema-migrations.md index 01a7bae2a7..c4a8a3b6f4 100644 --- a/docs/subsystems/schema-migrations.md +++ b/docs/subsystems/schema-migrations.md @@ -8,53 +8,53 @@ tutorial](../tutorials/new-feature-tutorial.md). This page documents some important issues related to writing schema migrations. -* If your database migration is just to reflect new fields in +- If your database migration is just to reflect new fields in `models.py`, you'll typically want to just: - * Rebase your branch before you start (this may save work later). - * Update the model class definitions in `zerver/models.py`. - * Run `./manage.py makemigrations` to generate a migration file - * Rename the migration file to have a descriptive name if Django + - Rebase your branch before you start (this may save work later). + - Update the model class definitions in `zerver/models.py`. + - Run `./manage.py makemigrations` to generate a migration file + - Rename the migration file to have a descriptive name if Django generated used a date-based name like `0089_auto_20170710_1353.py` (which happens when the changes are to multiple models and Django). - * `git add` the new migration file - * Run `tools/provision` to update your local database to apply the + - `git add` the new migration file + - Run `tools/provision` to update your local database to apply the migrations. - * Commit your changes. -* For more complicated migrations where you need to run custom Python + - Commit your changes. +- For more complicated migrations where you need to run custom Python code as part of the migration, it's best to read past migrations to understand how to write them well. `git grep RunPython zerver/migrations/02*` will find many good examples. Before writing migrations of this form, you should read Django's docs and the sections below. -* **Numbering conflicts across branches**: If you've done your schema +- **Numbering conflicts across branches**: If you've done your schema change in a branch, and meanwhile another schema change has taken place, Django will now have two migrations with the same number. There are two easy way to fix this: - * If your migrations were automatically generated using + - If your migrations were automatically generated using `manage.py makemigrations`, a good option is to just remove your migration and rerun the command after rebasing. Remember to `git rebase` to do this in the the commit that changed `models.py` if you have a multi-commit branch. - * If you wrote code as part of preparing your migrations, or prefer + - If you wrote code as part of preparing your migrations, or prefer this workflow, you can use run `./tools/renumber-migrations`, which renumbers your migration(s) and fixes up the "dependencies" entries in your migration(s). The tool could use a bit of work to prompt unnecessarily less, but it will update the working tree for you automatically (you still need to do all the `git add` commands, etc.). -* **Large tables**: For our very largest tables (e.g. Message and +- **Large tables**: For our very largest tables (e.g. Message and UserMessage), we often need to take precautions when adding columns to the table, performing data backfills, or building indexes. We have a `zerver/lib/migrate.py` library to help with adding columns and backfilling data. -* **Adding indexes** Regular `CREATE INDEX` SQL (corresponding to Django's +- **Adding indexes** Regular `CREATE INDEX` SQL (corresponding to Django's `AddIndex` operation) locks writes to the affected table. This can be problematic when dealing with larger tables in particular and we've generally preferred to use `CREATE INDEX CONCURRENTLY` to allow the index to be built while the server is active. While in historical migrations we've used `RunSQL` directly, newer versions of Django add the corresponding operation `AddIndexConcurrently` and thus that's what should normally be used. -* **Atomicity**. By default, each Django migration is run atomically +- **Atomicity**. By default, each Django migration is run atomically inside a transaction. This can be problematic if one wants to do something in a migration that touches a lot of data and would best be done in batches of e.g. 1000 objects (e.g. a `Message` or @@ -65,7 +65,7 @@ migrations. to use the batch update tools in `zerver/lib/migrate.py` (originally written to work with South) for doing larger database migrations. -* **Accessing code and models in RunPython migrations**. When writing +- **Accessing code and models in RunPython migrations**. When writing a migration that includes custom python code (aka `RunPython`), you almost never want to import code from `zerver` or anywhere else in the codebase. If you imagine the process of upgrading a Zulip @@ -101,25 +101,25 @@ migrations. non-atomic, splitting it into two migration files (recommended), or replacing the `RunPython` logic with pure SQL (though this can generally be difficult). -* **Making large migrations work**. Major migrations should have a +- **Making large migrations work**. Major migrations should have a few properties: - * **Unit tests**. You'll want to carefully test these, so you might + - **Unit tests**. You'll want to carefully test these, so you might as well write some unit tests to verify the migration works correctly, rather than doing everything by hand. This often saves a lot of time in re-testing the migration process as we make adjustments to the plan. - * **Run in batches**. Updating more than 1K-10K rows (depending on + - **Run in batches**. Updating more than 1K-10K rows (depending on type) in a single transaction can lock up a database. It's best to do lots of small batches, potentially with a brief sleep in between, so that we don't block other operations from finishing. - * **Rerunnability/idempotency**. Good migrations are ones where if + - **Rerunnability/idempotency**. Good migrations are ones where if operational concerns (e.g. it taking down the Zulip server for users) interfere with it finishing, it's easy to restart the migration without doing a bunch of hand investigation. Ideally, the migration can even continue where it left off, without needing to redo work. - * **Multi-step migrations**. For really big migrations, one wants + - **Multi-step migrations**. For really big migrations, one wants to split the transition into into several commits that are each individually correct, and can each be deployed independently: diff --git a/docs/subsystems/sending-messages.md b/docs/subsystems/sending-messages.md index ed831c2a5a..cfa34a78f6 100644 --- a/docs/subsystems/sending-messages.md +++ b/docs/subsystems/sending-messages.md @@ -17,13 +17,13 @@ and we generally don't repeat the content discussed there. This is just a bit of terminology: A "message list" is what Zulip calls the frontend concept of a (potentially narrowed) message feed. There are 3 related structures: -* A `message_list_data` just has the sequencing data of which message +- A `message_list_data` just has the sequencing data of which message IDs go in what order. -* A `message_list` is built on top of `message_list_data` and +- A `message_list` is built on top of `message_list_data` and additionally contains the data for a visible-to-the-user message list (E.g. where trailing bookends should appear, a selected message, etc.). -* A `message_list_view` is built on top of `message_list` and +- A `message_list_view` is built on top of `message_list` and additionally contains rendering details like a window of up to 400 messages that is present in the DOM at the time, scroll position controls, etc. @@ -48,10 +48,10 @@ process described in our [new application feature tutorial](../tutorials/new-feature-tutorial.md). This section details the ways in which it is different: -* There is significant custom code inside the `process_message_event` +- There is significant custom code inside the `process_message_event` function in `zerver/tornado/event_queue.py`. This custom code has a number of purposes: - * Triggering [email and mobile push + - Triggering [email and mobile push notifications](../subsystems/notifications.md) for any users who do not have active clients and have settings of the form "push notifications when offline". In order to avoid doing any real @@ -61,41 +61,41 @@ number of purposes: [queue](../subsystems/queuing.md) to actually send the message. See `maybe_enqueue_notifications` and related code for this part of the logic. - * Splicing user-dependent data (E.g. `flags` such as when the user + - Splicing user-dependent data (E.g. `flags` such as when the user was `mentioned`) into the events. - * Handling the [local echo details](#local-echo). - * Handling certain client configuration options that affect + - Handling the [local echo details](#local-echo). + - Handling certain client configuration options that affect messages. E.g. determining whether to send the plaintext/Markdown raw content or the rendered HTML (e.g. the `apply_markdown` and `client_gravatar` features in our [events API docs](https://zulip.com/api/register-queue)). -* Following our standard naming convention, input validation is done +- Following our standard naming convention, input validation is done inside the `check_message` function in `zerver/lib/actions.py`, which is responsible for validating the user can send to the recipient, [rendering the Markdown](../subsystems/markdown.md), etc. -- basically everything that can fail due to bad user input. -* The core `do_send_messages` function (which handles actually sending +- The core `do_send_messages` function (which handles actually sending the message) in `zerver/lib/actions.py` is one of the most optimized and thus complex parts of the system. But in short, its job is to atomically do a few key things: - * Store a `Message` row in the database. - * Store one `UserMessage` row in the database for each user who is + - Store a `Message` row in the database. + - Store one `UserMessage` row in the database for each user who is a recipient of the message (including the sender), with appropriate `flags` for whether the user was mentioned, an alert word appears, etc. See [the section on soft deactivation](#soft-deactivation) for a clever optimization we use here that is important for large open organizations. - * Do all the database queries to fetch relevant data for and then + - Do all the database queries to fetch relevant data for and then send a `message` event to the [events system](../subsystems/events-system.md) containing the data it will need for the calculations described above. This step adds a lot of complexity, because the events system cannot make queries to the database directly. - * Trigger any other deferred work caused by the current message, + - Trigger any other deferred work caused by the current message, e.g. [outgoing webhooks](https://zulip.com/api/outgoing-webhooks) or embedded bots. - * Every query is designed to be a bulk query; we carefully + - Every query is designed to be a bulk query; we carefully unit-test this system for how many database and memcached queries it makes when sending messages with large numbers of recipients, to ensure its performance. @@ -123,12 +123,12 @@ details on how that works and is tested. The rest of this section details how Zulip manages locally echoed messages. -* The core function in the frontend codebase +- The core function in the frontend codebase `echo.try_deliver_locally`. This checks whether correct local echo is possible (via `markdown.contains_backend_only_syntax`) and useful (whether the message would appear in the current view), and if so, causes Zulip to insert the message into the relevant feed(s). -* Since the message hasn't been confirmed by the server yet, it +- Since the message hasn't been confirmed by the server yet, it doesn't have a message ID. The frontend makes one up, via `local_message.next_local_id`, by taking the highest message ID it has seen and adding the decimal `0.01`. The use of a floating point @@ -141,14 +141,14 @@ messages. resort it to its proper place once it is confirmed by the server. We do it this way to minimize messages jumping around/reordering visually). -* The `POST /messages` API request to the server to send the message +- The `POST /messages` API request to the server to send the message is passed two special parameters that clients not implementing local echo don't use: `queue_id` and `local_id`. The `queue_id` is the ID of the client's event queue; here, it is used just as a unique identifier for the specific client (e.g. a browser tab) that sent the message. And the `local_id` is, by the construction above, a unique value within that namespace identifying the message. -* The `do_send_messages` backend code path includes the `queue_id` and +- The `do_send_messages` backend code path includes the `queue_id` and `local_id` in the data it passes to the [events system](../subsystems/events-system.md). The events system will extend the `message` event dictionary it delivers to @@ -156,7 +156,7 @@ messages. containing the `local_id` that the relevant client used when sending the message. This allows the client to know that the `message` event it is receiving is the same message it itself had sent. -* Using that information, rather than adding the "new message" to the +- Using that information, rather than adding the "new message" to the relevant message feed, it updates the (locally echoed) message's properties (at the very least, message ID and timestamp) and rerenders it in any message lists where it appears. This is @@ -176,30 +176,30 @@ implementation was under 150 lines of code. This section just has a brief review of the sequence of steps all in one place: -* User hits send in the compose box. -* Compose box validation runs; if it passes, the browser locally +- User hits send in the compose box. +- Compose box validation runs; if it passes, the browser locally echoes the message and then sends a request to the `POST /messages` API endpoint. -* The Django URL routes and middleware run, and eventually call the +- The Django URL routes and middleware run, and eventually call the `send_message_backend` view function in `zerver/views/messages.py`. (Alternatively, for an API request to send a message via Zulip's REST API, things start here). -* `send_message_backend` does some validation before triggering the +- `send_message_backend` does some validation before triggering the `check_message` + `do_send_messages` backend flow. -* That backend flow saves the data to the database and triggers a +- That backend flow saves the data to the database and triggers a `message` event in the `notify_tornado` queue (part of the events system). -* The events system processes, and dispatches that event to all +- The events system processes, and dispatches that event to all clients subscribed to receive notifications for users who should receive the message (including the sender). As a side effect, it adds queue items to the email and push notification queues (which, in turn, may trigger those notifications). - * Other clients receive the event and display the new message. - * For the client that sent the message, it instead replaces its + - Other clients receive the event and display the new message. + - For the client that sent the message, it instead replaces its locally echoed message with the final message it received back from the server (it indicates this to the sender by adding a display timestamp to the message). -* The `send_message_backend` view function returns +- The `send_message_backend` view function returns a 200 `HTTP` response; the client receives that response and mostly does nothing with it other than update some logging details. (This may happen before or after the client receives the event notifying @@ -210,17 +210,17 @@ one place: Message editing uses a very similar principle to how sending messages works. A few details are worth mentioning: -* `maybe_enqueue_notifications_for_message_update` is an analogue of +- `maybe_enqueue_notifications_for_message_update` is an analogue of `maybe_enqueue_notifications`, and exists to handle cases like a user was newly mentioned after the message is edited (since that should trigger email/push notifications, even if the original message didn't have one). -* We use a similar technique to what's described in the local echo +- We use a similar technique to what's described in the local echo section for doing client-side rerendering to update the message feed. -* In the default configuration, Zulip stores the message edit history +- In the default configuration, Zulip stores the message edit history (which is useful for forensics but also exposed in the UI), in the `message.edit_history` attribute. -* We support topic editing, including bulk-updates moving several +- We support topic editing, including bulk-updates moving several messages between topics. ### Inline URL previews @@ -232,16 +232,16 @@ from the target URL, and for slow websites, this could result in a significant delay in rendering the message and delivering it to other users. -* For this case, Zulip's backend Markdown processor will render the +- For this case, Zulip's backend Markdown processor will render the message without including the URL embeds/previews, but it will add a deferred work item into the `embed_links` queue. -* The [queue processor](../subsystems/queuing.md) for the +- The [queue processor](../subsystems/queuing.md) for the `embed_links` queue will fetch the URLs, and then if they return results, rerun the Markdown processor and notify clients of the updated message `rendered_content`. -* We reuse the `update_message` framework (used for +- We reuse the `update_message` framework (used for Zulip's message editing feature) in order to avoid needing custom code to implement the notification-and-rerender part of this implementation. @@ -327,10 +327,10 @@ organization for a few weeks, they are tagged as soft-deactivated. The way this works internally is: -* We (usually) skip creating UserMessage rows for soft-deactivated +- We (usually) skip creating UserMessage rows for soft-deactivated users when a message is sent to a stream where they are subscribed. -* If/when the user ever returns to Zulip, we can at that time +- If/when the user ever returns to Zulip, we can at that time reconstruct the UserMessage rows that they missed, and create the rows at that time (or, to avoid a latency spike if/when the user returns to Zulip, this work can be done in a nightly cron job). We can construct @@ -342,7 +342,7 @@ that the messages have not been marked as read by the user). This is done in the `add_missing_messages` function, which is the core of the soft-deactivation implementation. -* The “usually” above is because there are a few flags that result +- The “usually” above is because there are a few flags that result from content in the message (e.g., a message that mentions a user results in a “mentioned” flag in the UserMessage row), that we need to keep track of. Since parsing a message can be expensive (>10ms of @@ -356,13 +356,13 @@ time they were sent without any material performance impact. And then The end result is the best of both worlds: -* Nobody's view of the world is different because the user was +- Nobody's view of the world is different because the user was soft-deactivated (resulting in no visible user-experience impact), at least if one is running the cron job. If one does not run the cron job, then users returning after being away for a very long time will potentially have a (very) slow loading experience as potentially 100,000s of UserMessage rows might need to be reconstructed at once. -* On the latency-sensitive message sending and fanout code path, the +- On the latency-sensitive message sending and fanout code path, the server only needs to do work for users who are currently interacting with Zulip. @@ -374,13 +374,13 @@ it’ll arrive in the couple hundred milliseconds one would expect if the extra 4500 inactive subscribers didn’t exist. There are a few details that require special care with this system: -* [Email and mobile push +- [Email and mobile push notifications](../subsystems/notifications.md). We need to make sure these are still correctly delivered to soft-deactivated users; making this work required careful work for those code paths that assumed a `UserMessage` row would always exist for a message that triggers a notification to a given user. -* Digest emails, which use the `UserMessage` table extensively to +- Digest emails, which use the `UserMessage` table extensively to determine what has happened in streams the user can see. We can use the user's subscriptions to construct what messages they should have access to for this feature. diff --git a/docs/subsystems/settings.md b/docs/subsystems/settings.md index 74836572ee..21ce6efdca 100644 --- a/docs/subsystems/settings.md +++ b/docs/subsystems/settings.md @@ -5,9 +5,9 @@ help you decide how to correctly implement new settings you're adding to Zulip. We have two types of administrative settings in Zulip: -* **Server settings** are set via configuration files, and apply to +- **Server settings** are set via configuration files, and apply to the whole Zulip installation. -* **Realm settings** (or **organization settings**) are usually +- **Realm settings** (or **organization settings**) are usually set via the /#organization page in the Zulip web application, and apply to a single Zulip realm/organization. (Which, for most Zulip servers, is the only realm on the server). @@ -15,11 +15,11 @@ We have two types of administrative settings in Zulip: Philosophically, the goals of the settings system are to make it convenient for: -* Zulip server administrators to configure +- Zulip server administrators to configure Zulip's featureset for their server without needing to patch Zulip -* Realm administrators to configure settings for their organization +- Realm administrators to configure settings for their organization independently without needing to talk with the server administrator. -* Secrets (passwords, API keys, etc.) to be stored in a separate place +- Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration. ## Server settings @@ -49,7 +49,7 @@ of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settings for server administrators. In a production environment, we have: -* `/etc/zulip/settings.py` (the template is in the Zulip repo at +- `/etc/zulip/settings.py` (the template is in the Zulip repo at `zproject/prod_settings_template.py`) is the main system administrator-facing settings file for Zulip. It contains all the server-specific settings, such as how to send outgoing email, the @@ -64,7 +64,7 @@ In a production environment, we have: release](../production/upgrade-or-modify.html#updating-settings-py-inline-documentation) to pick up new inline documentation. -* `/etc/zulip/zulip-secrets.conf` (generated by +- `/etc/zulip/zulip-secrets.conf` (generated by `scripts/setup/generate_secrets.py` as part of installation) contains secrets used by the Zulip installation. These are read using the standard Python `ConfigParser`, and accessed in @@ -73,18 +73,18 @@ In a production environment, we have: stored here, and read using the `get_secret` function in `zproject/config.py`. -* `zproject/settings.py` is the main Django settings file for Zulip. +- `zproject/settings.py` is the main Django settings file for Zulip. It imports everything from `zproject/configured_settings.py` and `zproject/computed_settings.py`. -* `zproject/configured_settings.py` imports everything from +- `zproject/configured_settings.py` imports everything from `zproject/default_settings.py`, then in a prod environment imports `/etc/zulip/settings.py` via a symlink. -* `zproject/default_settings.py` has the default values for the settings the +- `zproject/default_settings.py` has the default values for the settings the user would set in `/etc/zulip/settings.py`. -* `zproject/computed_settings.py` contains all the settings that are +- `zproject/computed_settings.py` contains all the settings that are constant for all Zulip installations or computed as a function of `zproject/configured_settings.py` (e.g. configuration for logging, static assets, middleware, etc.). @@ -92,10 +92,10 @@ In a production environment, we have: In a development environment, we have `zproject/settings.py`, and additionally: -* `zproject/dev_settings.py` has the custom settings for the Zulip development +- `zproject/dev_settings.py` has the custom settings for the Zulip development environment; these are set after importing `prod_settings_template.py`. -* `zproject/dev-secrets.conf` replaces +- `zproject/dev-secrets.conf` replaces `/etc/zulip/zulip-secrets.conf`, and is not tracked by Git. This allows you to configure your development environment to support features like [authentication @@ -105,28 +105,28 @@ additionally: You can see a full list with `git grep development_only=True`, or add additional settings of this form if needed. -* `zproject/test_settings.py` imports everything from +- `zproject/test_settings.py` imports everything from `zproject/settings.py` and `zproject/test_extra_settings.py`. -* `zproject/test_extra_settings.py` has the (default) settings used +- `zproject/test_extra_settings.py` has the (default) settings used for the Zulip tests (both backend and Puppeteer), which are applied on top of the development environment settings. When adding a new server setting to Zulip, you will typically add it in two or three places: -* `zproject/default_settings.py`, with a default value +- `zproject/default_settings.py`, with a default value for production environments. -* If the settings has a secret key, +- If the settings has a secret key, you'll add a `get_secret` call in `zproject/computed_settings.py` (and the user will add the value when they configure the feature). -* In an appropriate section of `zproject/prod_settings_template.py`, +- In an appropriate section of `zproject/prod_settings_template.py`, with documentation in the comments explaining the setting's purpose and effect. -* Possibly also `zproject/dev_settings.py` and/or +- Possibly also `zproject/dev_settings.py` and/or `zproject/test_settings.py`, if the desired value of the setting for Zulip development and/or test environments is different from the default for production. @@ -159,10 +159,10 @@ process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be replaced with realm settings: -* `NAME_CHANGES_DISABLED` -* `INLINE_IMAGE_PREVIEW` -* `ENABLE_GRAVATAR` -* Which authentication methods are allowed should probably appear in +- `NAME_CHANGES_DISABLED` +- `INLINE_IMAGE_PREVIEW` +- `ENABLE_GRAVATAR` +- Which authentication methods are allowed should probably appear in both places; in server settings indicating the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants to allow users to log in with. diff --git a/docs/subsystems/typing-indicators.md b/docs/subsystems/typing-indicators.md index 800fbf88bd..f12124ed04 100644 --- a/docs/subsystems/typing-indicators.md +++ b/docs/subsystems/typing-indicators.md @@ -17,8 +17,8 @@ views. There are two major roles for users in this system: -* The "writing user" is composing a message. -* The "receiving user" is waiting to receive a message (or possibly +- The "writing user" is composing a message. +- The "receiving user" is waiting to receive a message (or possibly ready to shift their attention elsewhere). Any Zulip user can play either one of these roles, and sometimes @@ -28,9 +28,9 @@ being composed by the "writing user." On a high level the typing indicators system works like this: -* The client for the "writing user" sends requests to the server. -* The server broadcasts events to other users. -* The clients for "receiving users" receive events and conditionally +- The client for the "writing user" sends requests to the server. +- The server broadcasts events to other users. +- The clients for "receiving users" receive events and conditionally show typing indicators, depending on where the clients are narrowed. ## Writing user @@ -97,8 +97,8 @@ incoming "typing" events from the server, and the client will display typing notification only when both of these conditions are true: -* The "writing user" is still likely typing. -* The "receiving user" is in a view where they'd see the eventual +- The "writing user" is still likely typing. +- The "receiving user" is in a view where they'd see the eventual message. The client code starts by processing events, and it maintains data @@ -118,10 +118,10 @@ The web app client maintains a list of incoming "typists" using code in `static/js/typing_data.js`. The API here has functions like the following: -* `add_typist` -* `remove_typist` -* `get_group_typists` -* `get_all_typists` +- `add_typist` +- `remove_typist` +- `get_group_typists` +- `get_all_typists` One subtle thing that the client has to do here is to maintain timers for typing notifications. The constant diff --git a/docs/testing/continuous-integration.md b/docs/testing/continuous-integration.md index 06da228889..cb5497c3b7 100644 --- a/docs/testing/continuous-integration.md +++ b/docs/testing/continuous-integration.md @@ -13,10 +13,10 @@ many possible future bugs as possible, while minimizing both latency and false positives, both of which can waste a lot of developer time. There are a few implications of this overall goal: -* If a test is failing nondeterministically in CI, we consider that to +- If a test is failing nondeterministically in CI, we consider that to be an urgent problem. -* If the tests become a lot slower, that is also an urgent problem. -* Everything we do in CI should also have a way to run it quickly +- If the tests become a lot slower, that is also an urgent problem. +- Everything we do in CI should also have a way to run it quickly (under 1 minute, preferably under 3 seconds), in order to iterate fast in development. Except when working on the CI configuration itself, a developer should never have to repeatedly wait 10 minutes for a full CI @@ -26,16 +26,16 @@ run to iteratively debug something. ### Useful debugging tips and tools -* GitHub Actions stores timestamps for every line in the logs. They +- GitHub Actions stores timestamps for every line in the logs. They are hidden by default; you can see them by toggling the `Show timestamps` option in the menu on any job's log page. (You can get this sort of timestamp in a development environment by piping output to `ts`). -* GitHub Actions runs on every branch you push on your Zulip fork. +- GitHub Actions runs on every branch you push on your Zulip fork. This is helpful when debugging something complicated. -* You can also ssh into a container to debug failures. SSHing into +- You can also ssh into a container to debug failures. SSHing into the containers can be helpful, especially in rare cases where the tests are passing in your computer but failing in the CI. There are various @@ -111,8 +111,8 @@ between jobs the various caches that live under `/srv/` in a Zulip development or production environment. In particular, we cache the following: -* Python virtualenvs -* node_modules directories +- Python virtualenvs +- node_modules directories This has a huge impact on the performance of running tests in GitHub Actions CI; without these caches, the average test time would be several times diff --git a/docs/testing/linters.md b/docs/testing/linters.md index 7f373b94fe..33073ae4b6 100644 --- a/docs/testing/linters.md +++ b/docs/testing/linters.md @@ -46,12 +46,12 @@ You can also run them individually or pass specific files: `./tools/lint` has many useful options; you can read about them in its internal documentation using `./tools/lint --help`. Of particular note are: -* `--fix`: Several of our linters support automatically fixing basic +- `--fix`: Several of our linters support automatically fixing basic issues; this option will ask `tools/lint` to run those. -* `--verbose`: Provides detailed information on how to fix many common +- `--verbose`: Provides detailed information on how to fix many common linter errors not covered by `--fix`. -* `--skip` and `--only`: Only run certain linters. -* `-m`: Only check modified files. +- `--skip` and `--only`: Only run certain linters. +- `-m`: Only check modified files. Finally, you can rely on our continuous integration setup to run linters for you, but it is good practice to run lint checks locally. diff --git a/docs/testing/mypy.md b/docs/testing/mypy.md index fcafb4d4ac..835368e2c1 100644 --- a/docs/testing/mypy.md +++ b/docs/testing/mypy.md @@ -16,7 +16,7 @@ def get_user(email: str, realm: Realm) -> UserProfile: You can learn more about it at: -* The +- The [mypy cheat sheet for Python 3](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html) is the best resource for quickly understanding how to write the PEP 484 type annotations used by mypy correctly. The @@ -24,12 +24,12 @@ You can learn more about it at: is useful for understanding the type comment syntax needed for our few modules that need to support both Python 2 and 3. -* The +- The [Python type annotation spec in PEP 484](https://www.python.org/dev/peps/pep-0484/). -* Our [blog post on being an early adopter of mypy][mypy-blog-post] from 2016. +- Our [blog post on being an early adopter of mypy][mypy-blog-post] from 2016. -* Our [best practices](#best-practices) section below. +- Our [best practices](#best-practices) section below. The mypy type checker is run automatically as part of Zulip's Travis CI testing process in the `backend` build. @@ -76,12 +76,12 @@ errors, it's important to get to the bottom of the issue, not just do something quick to silence the warnings, before we merge the changes. Possible explanations include: -* A bug in any new type annotations you added. -* A bug in the existing type annotations. -* A bug in Zulip! -* Some Zulip code is correct but confusingly reuses variables with +- A bug in any new type annotations you added. +- A bug in the existing type annotations. +- A bug in Zulip! +- Some Zulip code is correct but confusingly reuses variables with different types. -* A bug in mypy (though this is increasingly rare as mypy is now +- A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project). Each explanation has its own solution, but in every case the result @@ -260,27 +260,27 @@ print(y.strip()) # runtime error If you think you need to use `Any`, consider the following safer alternatives first: -* To annotate a dictionary where different keys correspond to values +- To annotate a dictionary where different keys correspond to values of different types, instead of writing `Dict[str, Any]`, try declaring a [**`dataclass`**](https://mypy.readthedocs.io/en/stable/additional_features.html#dataclasses) or a [**`TypedDict`**](https://mypy.readthedocs.io/en/stable/more_types.html#typeddict). -* If you're annotating a class or function that might be used with +- If you're annotating a class or function that might be used with different data types at different call sites, similar to the builtin `List` type or the `sorted` function, [**generic types**](https://mypy.readthedocs.io/en/stable/generics.html) with `TypeVar` might be what you need. -* If you need to accept data of several specific possible types at a +- If you need to accept data of several specific possible types at a single site, you may want a [**`Union` type**](https://mypy.readthedocs.io/en/stable/kinds_of_types.html#union-types). `Union` is checked: before using `value: Union[str, int]` as a `str`, mypy requires that you validate it with an `instance(value, str)` test. -* If you really have no information about the type of a value, use the +- If you really have no information about the type of a value, use the **`object` type**. Since every type is a subtype of `object`, you can correctly annotate any value as `object`. The [difference between `Any` and @@ -289,7 +289,7 @@ alternatives first: `isinstance` before using it in a way that expects a more specific type. -* A common way for `Any` annotations to sneak into your code is the +- A common way for `Any` annotations to sneak into your code is the interaction with untyped third-party libraries. Mypy treats any value imported from an untyped library as annotated with `Any`, and treats any type imported from an untyped library as equivalent to @@ -310,7 +310,7 @@ print(x.strip()) # runtime error Instead of using `cast`: -* You can use a [variable +- You can use a [variable annotation](https://mypy.readthedocs.io/en/stable/type_inference_and_annotations.html#explicit-types-for-variables) to be explicit or to disambiguate types that mypy can check but cannot infer. @@ -319,7 +319,7 @@ Instead of using `cast`: l: List[int] = [] ``` -* You can use an [`isinstance` +- You can use an [`isinstance` test](https://mypy.readthedocs.io/en/stable/common_issues.html#complex-type-tests) to safely verify that a value really has the type you expect. @@ -341,12 +341,12 @@ issue. ### Avoid other unchecked constructs -* As mentioned +- As mentioned [above](#using-overload-to-accurately-describe-variations), we **discourage writing overloaded functions** because their bodies are not checked against the `@overload` signatures. -* **Avoid `Callable[..., T]`** (with literal ellipsis `...`), since +- **Avoid `Callable[..., T]`** (with literal ellipsis `...`), since mypy cannot check the types of arguments passed to it. Provide the specific argument types (`Callable[[int, str], T]`) in simple cases, or use [callback @@ -373,9 +373,9 @@ A collection such as `List` should only be `Optional` if `None` would have a different meaning than the natural meaning of an empty collection. For example: -* An include list where the default is to include everything should be +- An include list where the default is to include everything should be `Optional` with default `None`. -* An exclude list where the default is to exclude nothing should be +- An exclude list where the default is to exclude nothing should be non-`Optional` with default `[]`. Don't test an `Optional` value using truthiness (`if value:`, @@ -401,11 +401,11 @@ that was passed to it as an argument, especially by accident. To avoid this, prefer annotating function parameters with read-only types: -* [`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence) +- [`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence) instead of `List`, -* [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping) +- [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping) instead of `Dict`, -* [`AbstractSet`](https://docs.python.org/3/library/typing.html#typing.AbstractSet) +- [`AbstractSet`](https://docs.python.org/3/library/typing.html#typing.AbstractSet) instead of `Set`. This is especially important for parameters with default arguments, diff --git a/docs/testing/philosophy.md b/docs/testing/philosophy.md index e6f89a5d4d..c6a1b37a02 100644 --- a/docs/testing/philosophy.md +++ b/docs/testing/philosophy.md @@ -61,19 +61,19 @@ suite (`test-js-with-node`) to run in under 10 seconds. It'd be a long blog post to summarize everything we do to help achieve these goals, but a few techniques are worth highlighting: -* Our test suites are designed to not access the Internet, since the +- Our test suites are designed to not access the Internet, since the Internet might be down or unreliable in the test environment. Where outgoing HTTP requests are required to test something, we mock the responses with libraries like `responses`. -* We carefully avoid the potential for contamination of data inside +- We carefully avoid the potential for contamination of data inside services like PostgreSQL, Redis, and memcached from different tests. - * Every test case prepends a unique random prefix to all keys it + - Every test case prepends a unique random prefix to all keys it uses when accessing Redis and memcached. - * Every test case runs inside a database transaction, which is + - Every test case runs inside a database transaction, which is aborted after the test completes. Each test process interacts only with a fresh copy of a special template database used for server tests that is destroyed after the process completes. -* We rigorously investigate non-deterministically failing tests as though +- We rigorously investigate non-deterministically failing tests as though they were priority bugs in the product. ## Integration testing or unit testing? @@ -131,36 +131,36 @@ in any documentation for that interface. Some examples of this philosophy: -* If you have a web service that's mainly an API, you want to write +- If you have a web service that's mainly an API, you want to write your tests for that API. -* If you have a CLI program, you want to write your tests against the +- If you have a CLI program, you want to write your tests against the CLI. -* If you have a compiler, an interpreter, etc., you want essentially +- If you have a compiler, an interpreter, etc., you want essentially all your tests to be example programs, with a bit of metadata for things like "should give an error at this line" or "should build and run, and produce this output". In the Zulip context: -* Zulip uses the same API for our webapp as for our mobile clients and +- Zulip uses the same API for our webapp as for our mobile clients and third-party API clients, and most of our server tests are written against the Zulip API. -* The tests for Zulip's incoming webhooks work by sending actual +- The tests for Zulip's incoming webhooks work by sending actual payloads captured from the real third-party service to the webhook endpoints, and verifies that the webhook produces the expected Zulip message as output, to test the actual interface. So, to summarize our approach to integration vs. unit testing: -* While we aim to achieve test coverage of every significant code path +- While we aim to achieve test coverage of every significant code path in the Zulip server, which is commonly associated with unit testing, most of our tests are integration tests in the sense of sending a complete HTTP API query to the Zulip server and checking that the HTTP response and the internal state of the server following the request are both correct. -* Following the end-to-end principle in system design, where possible +- Following the end-to-end principle in system design, where possible we write tests that execute a complete flow (e.g. registering a new Zulip account) rather than testing the implementations of individual functions. -* We invest in the performance of Zulip in part to give users a great +- We invest in the performance of Zulip in part to give users a great experience, but just as much to make our test suite fast enough that we can write our tests this way. diff --git a/docs/testing/testing-with-django.md b/docs/testing/testing-with-django.md index b2398641ba..265dfdf84d 100644 --- a/docs/testing/testing-with-django.md +++ b/docs/testing/testing-with-django.md @@ -120,19 +120,19 @@ Here are some example action methods that tests may use for data setup: Some tests need to access the filesystem (e.g. `test_upload.py` tests for `LocalUploadBackend` and the data import tests). Doing this correctly requires care to avoid problems like: -* Leaking files after every test (which are clutter and can eventually +- Leaking files after every test (which are clutter and can eventually run the development environment out of disk) or -* Interacting with other parallel processes of this `test-backend` run +- Interacting with other parallel processes of this `test-backend` run (or another `test-backend` run), or with later tests run by this process. To avoid these problems, you can do the following: -* Use a subdirectory of `settings.TEST_WORKER_DIR`; this is a +- Use a subdirectory of `settings.TEST_WORKER_DIR`; this is a subdirectory of `/var//test-backend` that is unique to the test worker thread and will be automatically deleted when the relevant `test-backend` process finishes. -* Delete any files created by the test in the test class's `tearDown` +- Delete any files created by the test in the test class's `tearDown` method (which runs even if the test fails); this is valuable to avoid conflicts with other tests run later by the same test process. @@ -188,9 +188,9 @@ def greet(name_key: str) -> str: return "Hello" + name ``` -* You want to test `greet()`. +- You want to test `greet()`. -* In your test, you want to call `greet("Mario")` and verify that it returns the correct greeting: +- In your test, you want to call `greet("Mario")` and verify that it returns the correct greeting: ```python from greetings import greet @@ -204,9 +204,9 @@ def greet(name_key: str) -> str: a database. *You haven't created that database for your tests, so your test would fail, even though the code is correct.* -* Luckily, you know that `fetch_database("Mario")` should return "Mr. Mario Mario". +- Luckily, you know that `fetch_database("Mario")` should return "Mr. Mario Mario". - * *Hint*: Sometimes, you might not know the exact return value, but one that is equally valid and works + - *Hint*: Sometimes, you might not know the exact return value, but one that is equally valid and works with the rest of the code. In that case, just use this one. -> **Solution**: You mock `fetch_database()`. This is also referred to as "mocking out" `fetch_database()`. @@ -277,13 +277,13 @@ On the other hand, if we had used `import os.urandom`, we would need to call `mo #### Boilerplate code -* Including the Python mocking library: +- Including the Python mocking library: ```python from unittest import mock ``` -* Mocking a class with a context manager: +- Mocking a class with a context manager: ```python with mock.patch('module.ClassName', foo=42, return_value='I am a mock') as my_mock: @@ -292,7 +292,7 @@ On the other hand, if we had used `import os.urandom`, we would need to call `mo # var = module.ClassName() will assign 'I am a mock' to var. ``` -* Mocking a class with a decorator: +- Mocking a class with a decorator: ```python @mock.patch('module.ClassName', foo=42, return_value='I am a mock') @@ -301,7 +301,7 @@ On the other hand, if we had used `import os.urandom`, we would need to call `mo # In here, 'module.ClassName' will behave as in the previous example. ``` -* Mocking a class attribute: +- Mocking a class attribute: ```python with mock.patch.object(module.ClassName, 'class_method', return_value=42) diff --git a/docs/testing/testing-with-node.md b/docs/testing/testing-with-node.md index 7ff2a65dec..4f9558ce69 100644 --- a/docs/testing/testing-with-node.md +++ b/docs/testing/testing-with-node.md @@ -191,15 +191,15 @@ These instructions assume you're using the Vagrant development environment. 2. In WebStorm, navigate to `Preferences -> Tools -> Vagrant` and configure the following: - * `Instance folder` should be the root of the `zulip` repository on + - `Instance folder` should be the root of the `zulip` repository on your host (where the Vagrantfile is located). - * `Provider` should be `virtualbox` on macOS and Docker on Linux - * In `Boxes`, choose the one used for Zulip (unless you use + - `Provider` should be `virtualbox` on macOS and Docker on Linux + - In `Boxes`, choose the one used for Zulip (unless you use Virtualbox for other things, there should only be one option). You shouldn't need to set these additional settings: - * `Vagrant executable` should already be correctly `vagrant`. - * `Environment Variables` is not needed. + - `Vagrant executable` should already be correctly `vagrant`. + - `Environment Variables` is not needed. 3. You'll now need to set up a WebStorm "Debug Configuration". Open the `Run/Debug Configuration` menu and create a new `Node.js` config: diff --git a/docs/testing/testing-with-puppeteer.md b/docs/testing/testing-with-puppeteer.md index 598a8a8b22..2f6628b758 100644 --- a/docs/testing/testing-with-puppeteer.md +++ b/docs/testing/testing-with-puppeteer.md @@ -65,24 +65,24 @@ The following questions are useful when debugging Puppeteer test failures you might see in [continuous integration](../testing/continuous-integration.md): -* Does the flow being tested work properly in the Zulip browser UI? +- Does the flow being tested work properly in the Zulip browser UI? Test failures can reflect real bugs, and often it's easier and more interactive to debug an issue in the normal Zulip development environment than in the Puppeteer test suite. -* Does the change being tested adjust the HTML structure in a way that +- Does the change being tested adjust the HTML structure in a way that affects any of the selectors used in the tests? If so, the test may just need to be updated for your changes. -* Does the test fail deterministically when you run it locally using +- Does the test fail deterministically when you run it locally using E.g. `./tools/test-js-with-puppeteer compose.ts`? If so, you can iteratively debug to see the failure. -* Does the test fail nondeterministically? If so, the problem is +- Does the test fail nondeterministically? If so, the problem is likely that a `waitForSelector` statement is either missing or not waiting for the right thing. Tests fail nondeterministically much more often on very slow systems like those used for Continuous Integration (CI) services because small races are amplified in those environments; this often explains failures in CI that cannot be easily reproduced locally. -* Does the test fail when you are typing (filling the form) on a modal +- Does the test fail when you are typing (filling the form) on a modal or other just-opened UI widget? Puppeteer starts typing after focusing on the text field, sending keystrokes one after another. So, if application code explicitly focuses the modal after it @@ -96,22 +96,22 @@ integration](../testing/continuous-integration.md): These tools/features are often useful when debugging: -* You can use `console.log` statements both in Puppeteer tests and the +- You can use `console.log` statements both in Puppeteer tests and the code being tested to print-debug. -* Zulip's Puppeteer tests are configured to generate screenshots of +- Zulip's Puppeteer tests are configured to generate screenshots of the state of the test browser when an assert statement fails; these are stored under `var/puppeteer/*.png` and are extremely helpful for debugging test failures. -* TODO: Mention how to access Puppeteer screenshots in CI. -* TODO: Add an option for using the `headless: false` debugging mode +- TODO: Mention how to access Puppeteer screenshots in CI. +- TODO: Add an option for using the `headless: false` debugging mode of Puppeteer so you can watch what's happening, and document how to make that work with Vagrant. -* TODO: Document `--interactive`. -* TODO: Document how to run 100x in CI to check for nondeterminstic +- TODO: Document `--interactive`. +- TODO: Document how to run 100x in CI to check for nondeterminstic failures. -* TODO: Document any other techniques/ideas that were helpful when porting +- TODO: Document any other techniques/ideas that were helpful when porting the Casper suite. -* The Zulip server powering these tests is just `run-dev.py` with some +- The Zulip server powering these tests is just `run-dev.py` with some extra [Django settings](../subsystems/settings.md) from `zproject/test_extra_settings.py` to configure an isolated database so that the tests will not interfere/interact with a normal diff --git a/docs/testing/testing.md b/docs/testing/testing.md index 77d2504f98..d7fb55117a 100644 --- a/docs/testing/testing.md +++ b/docs/testing/testing.md @@ -107,9 +107,9 @@ As a policy matter, the Zulip test suites should never make outgoing HTTP or other network requests. This is important for 2 major reasons: -* Tests that make outgoing Internet requests will fail when the user +- Tests that make outgoing Internet requests will fail when the user isn't on the Internet. -* Tests that make outgoing Internet requests often have a hidden +- Tests that make outgoing Internet requests often have a hidden dependency on the uptime of a third-party service, and will fail nondeterministically if that service has a temporary outage. Nondeterministically failing tests can be a big waste of diff --git a/docs/testing/typescript.md b/docs/testing/typescript.md index a3b31d58d8..dda223e067 100644 --- a/docs/testing/typescript.md +++ b/docs/testing/typescript.md @@ -27,7 +27,7 @@ setdefault(key: K, value: V): V { The following resources are valuable for learning TypeScript: -* The main documentation on [TypeScript syntax][typescript-handbook]. +- The main documentation on [TypeScript syntax][typescript-handbook]. ## Type checking @@ -49,11 +49,11 @@ we plan to start with a style that is closer to the existing JavaScript code, so that we can easily migrate individual modules without too much code churn. A few examples: -* TypeScript generally prefers explicit `return undefined;`, whereas +- TypeScript generally prefers explicit `return undefined;`, whereas our existing JavasScript style uses just `return;`. -* With TypeScript, we expect to make heavy use of `let` and `const` +- With TypeScript, we expect to make heavy use of `let` and `const` rather than `var`. -* With TypeScript/ES6, we may no longer need to use `_.each()` as our +- With TypeScript/ES6, we may no longer need to use `_.each()` as our recommended way to do loop iteration. For each of the details, we will either want to bulk-migrate the @@ -71,13 +71,13 @@ converts them from JS to TS. Our plan is to order which modules we migrate carefully, starting with those that: -* Appear frequently as reverse dependencies of other modules +- Appear frequently as reverse dependencies of other modules (e.g. `people.js`). These are most valuable to do first because then we have types on the data being interacted with by other modules when we migrate those. -* Don't have large open pull requests (to avoid merge conflicts); one +- Don't have large open pull requests (to avoid merge conflicts); one can scan for these using [TinglingGit](https://github.com/zulip/TinglingGit). -* Have good unit test coverage, which limits the risk of breaking +- Have good unit test coverage, which limits the risk of breaking correctness through refactoring. Use `tools/test-js-with-node --coverage` to get a coverage report. @@ -85,14 +85,14 @@ When migrating a module, we want to be especially thoughtful about putting together a commit structure that makes mistakes unlikely and the changes easy to verify. E.g.: -* First a commit that just converts the language to TypeScript adding +- First a commit that just converts the language to TypeScript adding types. The result may potentially have some violations of the long-term style we want (e.g. not using `const`). Depending on how we're handling linting, we set some override eslint rules at the top of the module at this stage so CI still passes. -* Then a commit just migrating use of `var` to `const/let` without +- Then a commit just migrating use of `var` to `const/let` without other changes (other than removing any relevant linter overrides). -* Etc. +- Etc. With this approach, we should be able to produce a bunch of really simple commits that can be merged the same day they're written without diff --git a/docs/translating/chinese.md b/docs/translating/chinese.md index 64833d5ad3..15ccdc8259 100644 --- a/docs/translating/chinese.md +++ b/docs/translating/chinese.md @@ -14,7 +14,7 @@ Zulip的文风比较口语化，考虑到大多数中国用户的习惯，翻译 ## Terms(术语) -* Message - **消息** +- Message - **消息** "Message" can be literally translated as "消息" and "信息", both OK. Here "消息" is chosen for translation. For example, "Stream @@ -28,7 +28,7 @@ Message可直译为“消息”、“信息”等，两者皆可，这里统一 内微博、微信的使用习惯保持一致。“Starred Message”类似于QQ邮箱中的“星标 邮件”功能，这里也借鉴翻译为“星标消息”。 -* Stream - **频道** +- Stream - **频道** There were several other optional translations, e.g. "群组(Group)", " 主题(Subject)", and "栏目(Column)". The "频道(Channel)" is in use now, @@ -47,9 +47,9 @@ group. However, "讨论组" has one more Chinese character than "频道 是“讨论组”，字多一个，稍微有点啰嗦。主要参考自以前QQ的“讨论组”，在QQ中 是一种临时的群组。 -* Topic - **话题** +- Topic - **话题** -* Invite-Only/Public Stream - **私有/公开频道** +- Invite-Only/Public Stream - **私有/公开频道** "Invite-Only Stream" requires users must be invited explicitly to subscribe, which assures a high privacy. Other users cannot perceive @@ -60,9 +60,9 @@ read, it is translated sense to sense as "私有频道(Private Stream)"。 Stream”具有非常好的私密性，用户在没有订阅时是不能感知这类频道的存在的。 直译读起来有点拗口，因此选择译为“私有频道”。 -* Bot - **机器人** +- Bot - **机器人** -* Integration - **应用整合** +- Integration - **应用整合** "Integration" is literally translated as "集成" or "整合". It means integrating Zulip production with other applications and services. For @@ -72,13 +72,13 @@ integrity in Chinese expression, it is translated as "应用整合 “Integration”原意为“集成”与“整合”，这里表示将其它的应用或者服务与Zulip 实现整合。为表达意思完整，补充翻译为“应用整合”。 -* Notification - **通知** +- Notification - **通知** -* Alert Word - **提示词** +- Alert Word - **提示词** ## Phrases(习惯用语) -* Subscribe/Unsubscribe - **订阅/退订** +- Subscribe/Unsubscribe - **订阅/退订** The perfect tense subscribed/unsubscribed is translated as "已订阅/已 退订". Sometimes "Join" is used to express the same meanings as @@ -87,7 +87,7 @@ The perfect tense subscribed/unsubscribed is translated as "已订阅/已 完成时态译为“已订阅（Subscribed）”和“已退订（Unsubscribed）”。有时， “Join”也会用来表达与“Subscribe”相同的意思，也一并翻译为“订阅”。 -* Narrow to ... - **筛选** +- Narrow to ... - **筛选** "Narrow to" is translated as "筛选(Filter by)" for now, based on two considerations: @@ -114,7 +114,7 @@ readability considerations. 另外，在搜索功能的语境中，“Narrow to ...”没有翻译为“筛选”，而翻译为“搜 索”，这是出于可读性的考虑。 -* Mute/Unmute - **开启/关闭免打扰** +- Mute/Unmute - **开启/关闭免打扰** "Mute" is mostly translated as "静音(Silent)", which is common in TV set. Such a translation is not appropriate for Zulip. "开启/关闭免打 @@ -124,7 +124,7 @@ is also borrowed from the WeChat. “Mute”常见的中文翻译为“静音”，在电视设备中常见，用在Zulip中并不太合适。 这里取意译，与大家常用的微信（WeChat）中“消息免打扰”用语习惯一致。 -* Deactivate/Reactivate - **禁用/启用(帐户)，关闭/激活(社区)** +- Deactivate/Reactivate - **禁用/启用(帐户)，关闭/激活(社区)** When applied to a user account, translated as "禁用/启用 (Disable/Enable)", for example, "Deactivated users" to "禁用的用户"; @@ -135,7 +135,7 @@ example "Your realm has been deactivated." to "您的社区已关闭". 为“禁用的用户”；当应用于“社区”（Realm）时，选择翻译为“关闭/激活”，如 “Your realm has been deactivated.”翻译为“您的社区已关闭”。 -* Invalid - **不正确** +- Invalid - **不正确** "Invalid" is mainly used in exception information, which is uncommon for general users. Other translations "错误(Error)", "非法(Illegal)", @@ -147,7 +147,7 @@ translated as "API码不正确". 译有“错误”、“非法”、“不合法”；为保持一致的习惯，这里统一翻译为“不正确”。 例如“Invalid API key”翻译为“API码不正确”。 -* I want - **开启** +- I want - **开启** Mainly used in the settings page, literally translated as "I want to ...", which is colloquial and inappropriate in Chinese expression. It @@ -157,7 +157,7 @@ is translated sense to sense as "开启(Turn on some options)". 译过于口语化，并不合乎中文的使用习惯。因此这里取意译，翻译为“开启（某 某功能选项）”。 -* User/People/Person - **用户** +- User/People/Person - **用户** All translated as "用户(User)". @@ -165,14 +165,14 @@ All translated as "用户(User)". ## Others(其它) -* You/Your - **您/您的** +- You/Your - **您/您的** It is translated as 您/您的(You/Your) rather than "你/你的(You/Your)", so as to express respect to the user. 出于尊重用户的目的，翻译为敬语“您/您的”，而不翻译为“你/你的”。 -* We - **我们（或不翻）** +- We - **我们（或不翻）** "We" is generally translated as the first person "我们(We)", while in formal Chinese, extensive use of "We" is relatively rare. So in many @@ -186,7 +186,7 @@ resend.)". 翻译：“Still no email? We can resend it” 译为 “仍然没有收到邮件？点击 重新发送”。 -* The Exclamation/Dot - (一般省略) +- The Exclamation/Dot - (一般省略) The exclamation appears in many places in Zulip. The tone that the exclamation expresses should be stronger in Chinese than in diff --git a/docs/translating/french.md b/docs/translating/french.md index cac9826bee..cad5df0b74 100644 --- a/docs/translating/french.md +++ b/docs/translating/french.md @@ -2,27 +2,27 @@ ## Rules -* Use of *vous* instead of *tu*, -* A space before and after a colon and a semi-colon and a space after a dot and a comma, -* Follow english capitalization, -* Prefer the infinitive form of a verb: *Save* into *Sauver* (instead of *Sauvez*). +- Use of *vous* instead of *tu*, +- A space before and after a colon and a semi-colon and a space after a dot and a comma, +- Follow english capitalization, +- Prefer the infinitive form of a verb: *Save* into *Sauver* (instead of *Sauvez*). Some translations can be tricky, so please don't hesitate to ask the community or contribute to this guide. ## Terms -* stream - **canal** -* topic - **sujet** -* edit - **modifier** -* email - **courriel** -* upload - **envoyer** -* bot - **robot** -* alert word - **mot alerte** -* emoji - **emoji** -* subscription - **abonnement** -* mute - **rendre muet**/**muet** -* home - **accueil** -* narrow - **restreindre** -* pin - **épingler** -* star - **favori**/**mettre en favori** -* organization - **organisation** +- stream - **canal** +- topic - **sujet** +- edit - **modifier** +- email - **courriel** +- upload - **envoyer** +- bot - **robot** +- alert word - **mot alerte** +- emoji - **emoji** +- subscription - **abonnement** +- mute - **rendre muet**/**muet** +- home - **accueil** +- narrow - **restreindre** +- pin - **épingler** +- star - **favori**/**mettre en favori** +- organization - **organisation** diff --git a/docs/translating/german.md b/docs/translating/german.md index cdab6fc23b..b746e48f5b 100644 --- a/docs/translating/german.md +++ b/docs/translating/german.md @@ -17,10 +17,10 @@ in a more colloquial style, German translations should be rather informal as wel **Don't use slang or regional phrases in the German translation:** -* Instead of *"So'n Dreck kann jedem mal passieren."*, you could +- Instead of *"So'n Dreck kann jedem mal passieren."*, you could say *"Dieser Fehler tritt häufiger auf."* -* "Das ist die Seite, wo der Quelltext steht." - the "*wo*" is regional, +- "Das ist die Seite, wo der Quelltext steht." - the "*wo*" is regional, say *"Das ist die Seite, auf der der Quelltext steht."* instead. ### Form of address @@ -60,10 +60,10 @@ concatenations whenever possible, since they can break the layout of the Zulip frontend. Try to stick to a maximum length of 20 characters and follow your intuition. -* A term like *Tastaturkürzel* for *Keyboard shortcuts* is fine - it is +- A term like *Tastaturkürzel* for *Keyboard shortcuts* is fine - it is shorter than 20 characters and commonly used in web applications. -* A term like *Benachrichtigungsstichwörter* for *Alert words* should +- A term like *Benachrichtigungsstichwörter* for *Alert words* should not be used, it sounds odd and is longer than 20 characters. You could use "*Stichwörter, die mich benachrichtigen*" instead. @@ -76,10 +76,10 @@ This becomes even more evident in internet applications, so you should not be afraid of using them if they provide an advantage over the German equivalent. Take the following two examples as a reference: -* Translating *Stream*: Use the German word *Kanal*, since it is just as short +- Translating *Stream*: Use the German word *Kanal*, since it is just as short and used in other web apps. -* Translating *Bot*: Use *Bot*, as a completely accurate German +- Translating *Bot*: Use *Bot*, as a completely accurate German equivalent **doesn't** exist (e.g. *Roboter*) and the term *Bot* is not unknown to German speakers. @@ -105,19 +105,19 @@ Make sure to not walk into such a trap. ### Other -* Try to keep words and phrases short and understandable. The front-end +- Try to keep words and phrases short and understandable. The front-end developers will thank you ;) -* Be consistent. Use the same terms for the same things, even if that +- Be consistent. Use the same terms for the same things, even if that means repeating. Have a look at other German translations on Zulip to get a feeling for the vocabulary. -* Balance common verbs and nouns with specific IT-related translations +- Balance common verbs and nouns with specific IT-related translations of English terms - this can be tricky, try to check how other resources were translated (e.g. Gmail, Microsoft websites, Facebook) to decide what wouldn't sound awkward / rude in German. -* For additional translation information, feel free to check out +- For additional translation information, feel free to check out [this](https://en.wikipedia.org/wiki/Wikipedia:Translating_German_WP) Wikipedia guide on translating German Wikipedia articles into English. @@ -127,11 +127,11 @@ speakers in the community. It's all about making Zulip friendly and usable. ## Terms (Begriffe) -* Message - **Nachricht** +- Message - **Nachricht** *"Nachricht" (Facebook, WhatsApp, Transifex)* -* Private Message (PM) - **Private Nachricht (PN)** +- Private Message (PM) - **Private Nachricht (PN)** Since we try to avoid concatenating words whenever possible, don't use "Privatnachricht" . PN is the officially used abbreviation for @@ -139,7 +139,7 @@ Since we try to avoid concatenating words whenever possible, don't use *"Private Nachricht" (YouTube, Transifex)* -* Starred Message - **Markierte Nachricht** +- Starred Message - **Markierte Nachricht** We go with "markiert" instead of "gesternt" (which is not even a proper German word) here, since it comes closer to the original meaning of "starred". @@ -149,7 +149,7 @@ German word) here, since it comes closer to the original meaning of "starred". *"Bereich" (Transifex), "Community" (Google+)* -* Stream - **Stream** +- Stream - **Stream** Even though the term **Stream** is not commonly used in German web applications, it is both understood well enough by many Germans with only little English @@ -161,11 +161,11 @@ no differentiation between streams and topics. *"Stream" (Transifex), "Kanal" (KDE IRC documentation, various small German forums)* -* Topic - **Thema** +- Topic - **Thema** *(Gmail - for email subjects, Transifex)* -* Invite-Only Stream - **Geschlossener Stream** +- Invite-Only Stream - **Geschlossener Stream** For users to be able to join to an "invite-only" stream, they must have been invited by some user in this stream. This type of stream is equivalent to @@ -177,7 +177,7 @@ search returns only paraphrases of this term. *"Geschlossener Stream" (Transifex), "Geschlossene Gruppe" (Facebook), paraphrases (Linguee)* -* Public Stream - **Öffentlicher Stream** +- Public Stream - **Öffentlicher Stream** While some might find this direct translation a tad long, the alternative "Offener Stream" can be ambiguous - especially users who are inexperienced @@ -185,14 +185,14 @@ with Zulip could think of this as streams that are online. *"Öffentlicher Stream" (Transifex)* -* Bot - **Bot** +- Bot - **Bot** Not only is "bot" a short and easily memorable term, it is also widely used in German technology magazines, forums, etc. *"Bot" (Transifex, Heise, Die Zeit)* -* Integration - **Integration** +- Integration - **Integration** While the German translation of "Integration" is spelled just like the English version, the translation is referring to the German term. For this reason, @@ -202,14 +202,14 @@ translation, but "Integration" has the same meaning in German and English. *"Integration/-en" (Transifex)* -* Notification - **Benachrichtigung** +- Notification - **Benachrichtigung** Nice and easy. Other translations for "notification" like "Erwähnung", "Bescheid" or "Notiz" don't fit here. *"Benachrichtigung" (Facebook, Gmail, Transifex, Wikipedia)* -* Alert Word - **Signalwort** +- Alert Word - **Signalwort** This one is tricky, since one might initially think of "Alarmwort" as a proper translation. "Alarm", however, has a negative connotation, people link it to @@ -220,7 +220,7 @@ shows that some websites misuse "Alarm" for the translation. *"Signalwort" (Transifex), "Wort-Alarm" (Linguee)* -* View - **View** (Developer documentation) +- View - **View** (Developer documentation) Since this is a Zulip-specific term for > every path that the Zulip server supports (doesn’t show a 404 page for), @@ -229,7 +229,7 @@ and there is no German equivalent, talking of "Views" is preferable in the developer documentation and makes it easier to rely on parts of the German *and* parts of the English documentation. -* View - **Ansicht** (User documentation) +- View - **Ansicht** (User documentation) For the user documentation, we want to use "Ansicht" instead of "view", as "Ansicht" provides a translated description for what you think of when hearing @@ -239,13 +239,13 @@ which Germans often link to IT-related definitions). *"Ansicht" (Transifex)* -* Home - **Startseite** +- Home - **Startseite** Nice and easy. "Zuhause" obviously doesn't fit here ;). *"Startseite" (Facebook, Transifex)* -* Emoji - **Emoji** +- Emoji - **Emoji** "Emoji" is the standard term for Emojis. Any other Germanized translation like "Bildschriftzeichen" (which exists!) would sound stiff and outdated. "Emoticon" @@ -256,13 +256,13 @@ works as well, but is not that common in German. ## Phrases (Ausdrücke) -* Subscribe/Unsubscribe - **Abonnieren/Deabonnieren** +- Subscribe/Unsubscribe - **Abonnieren/Deabonnieren** This translation is unambiguous. *"Deabonnieren" (YouTube, Transifex)* -* Narrow to - **Begrenzen auf** +- Narrow to - **Begrenzen auf** Transifex has two different translations for "Narrow to" - "Schränke auf ... ein." and "Begrenze auf ... ." Both sound a bit strange to a @@ -274,28 +274,28 @@ would be too long for many labels, the infinitive "begrenzen auf" is preferable. *"Schränke auf ... ein." (Transifex) "Begrenze auf ... ." (Transifex)* -* Filter - **Filtern** +- Filter - **Filtern** A direct translation is fine here. Watch out to to use the infinitive instead of the imperative, e.g. "Nachrichten filtern" instead of "Filtere Nachrichten". *"Filtern" (Thunderbird, LinkedIn)* -* Mute/Unmute - **Stummschalten/Lautschalten** +- Mute/Unmute - **Stummschalten/Lautschalten** "Lautschalten" is rarely used in German, but so is "Stummschaltung deaktivieren". Since anyone can understand the idea behind "Lautschalten", it is preferable due to its brevity. -* Deactivate/Reactivate - **Deaktivieren/Reaktivieren** +- Deactivate/Reactivate - **Deaktivieren/Reaktivieren** *"Deaktivieren/Reaktivieren" (Transifex)* -* Search - **Suchen** +- Search - **Suchen** *"Suchen" (YouTube, Google, Facebook, Transifex)* -* Pin/Unpin - **Anpinnen/Loslösen** +- Pin/Unpin - **Anpinnen/Loslösen** While "pinnen" is shorter than "anpinnen", "anpinnen" sweeps any amiguity out of the way. This term is not used too often on Zulip, so the length shouldn't be a @@ -303,35 +303,35 @@ problem. *"Anpinnen/Ablösen" (Transifex), "Pinnen" (Pinterest)* -* Mention/@mention - **Erwähnen/"@-Erwähnen** +- Mention/@mention - **Erwähnen/"@-Erwähnen** Make sure to say "@-erwähnen", but "die @-Erwähnung" (capitalized). *"Erwähnen/@-Erwähnen" (Transifex)* -* Invalid - **Ungültig** +- Invalid - **Ungültig** *"Ungültig" (Transifex)* -* Customization - **Anpassen** +- Customization - **Anpassen** The literal translation "Anpassung" would sound weird in most cases, so we use the infinitive form "anpassen". -* I want - **Ich möchte** +- I want - **Ich möchte** "Ich möchte" is the polite form of "Ich will". *"Ich möchte" - (Transifex, general sense of politeness)* -* User - **Nutzer** +- User - **Nutzer** "Benutzer" would work as well, but "Nutzer" is shorter and more commonly used in web applications. *"Nutzer" (Facebook, Gmail), "Benutzer" (Transifex)* -* Person/People - Nutzer/Personen +- Person/People - Nutzer/Personen We use "Personen" instead of plural "Nutzer" for "people", as "Nutzer" stays the same in plural. @@ -340,7 +340,7 @@ the same in plural. ## Other (Verschiedenes) -* You - **Du** +- You - **Du** Why not "Sie"? In brief, Zulip and other web applications tend to use a rather informal language. If you would like to read more about the reasoning behind @@ -349,7 +349,7 @@ translating German. *"Du" (Google, Facebook), "Sie" (Transifex)* -* We - **Wir** (rarely used) +- We - **Wir** (rarely used) German guides don't use "wir" very often - they tend to reformulate the phrases instead. diff --git a/docs/translating/hindi.md b/docs/translating/hindi.md index f1466cccec..c2fe36e9e5 100644 --- a/docs/translating/hindi.md +++ b/docs/translating/hindi.md @@ -2,19 +2,19 @@ Use informal Hindi for translation: -* Informal "you" (*तु*) instead of formal form *आप*. Many top software +- Informal "you" (*तु*) instead of formal form *आप*. Many top software companies (e.g. Google) use the informal one, because it's much more common in the daily language and avoids making translations look like they were written by machines. -* Imperative, active, and continuous verbs, e.g. *manage streams* - +- Imperative, active, and continuous verbs, e.g. *manage streams* - *चैनल प्रबंधित करें*, not *चैनल प्रबंधन*. -* Warm and friendly phrasing whenever appropriate. +- Warm and friendly phrasing whenever appropriate. -* No slang or regional phrases that could be unclear or too informal. +- No slang or regional phrases that could be unclear or too informal. -* Balance common verbs and nouns with specific IT-related translations +- Balance common verbs and nouns with specific IT-related translations of English terms - this can be tricky, try to check how other resources were translated (e.g. Gmail, Microsoft websites, Facebook) to decide what wouldn't sound awkward / rude in Hindi. @@ -24,44 +24,44 @@ with other Hindi speakers in the community. It's all about making Zulip friendly and usable. ## Terms(शर्तें) -* Message - **संदेश** -* Private message (PM) - **निजी संदेश** -* Stream - **धारा**: the use of the literal Hindi word for stream +- Message - **संदेश** +- Private message (PM) - **निजी संदेश** +- Stream - **धारा**: the use of the literal Hindi word for stream "प्रवाह" is very confusing and not the correct metaphor for Hindi speaking people. The correct term would be "धारा". -* Topic - **विषय** -* Private/invite-only stream - **निजी / केवल-आमंत्रण धारा** -* Public stream - **सार्वजनिक धारा** -* Bot - **बॉट** -* Integration - **एकीकरण** -* Notification - **अधिसूचना** -* Alert word - **सतर्क शब्द**: this is only *alert*. Nonetheless, adding *word* may +- Topic - **विषय** +- Private/invite-only stream - **निजी / केवल-आमंत्रण धारा** +- Public stream - **सार्वजनिक धारा** +- Bot - **बॉट** +- Integration - **एकीकरण** +- Notification - **अधिसूचना** +- Alert word - **सतर्क शब्द**: this is only *alert*. Nonetheless, adding *word* may make the term confusing (something like *danger!* could be a "चेतावनी के शब्द" as well). Google Alerts uses "सतर्क शब्द" in its Hindi translation. -* View - **राय** -* Filter - **छानना**: as used with narrowing (see below). -* Home - **मुख पृष्ठ**: we never use the term "घर" (literally home) in Hindi. -* Emoji - **इमोजी** +- View - **राय** +- Filter - **छानना**: as used with narrowing (see below). +- Home - **मुख पृष्ठ**: we never use the term "घर" (literally home) in Hindi. +- Emoji - **इमोजी** ## Phrases (वाक्यांशों) -* Subscribe/Unsubscribe to a stream - **एक धारा में सदस्यता लें/सदस्यता समाप्त करें** -* Narrow to - **अकेले फ़िल्टर करें**: this is *filter only*, because there's no other +- Subscribe/Unsubscribe to a stream - **एक धारा में सदस्यता लें/सदस्यता समाप्त करें** +- Narrow to - **अकेले फ़िल्टर करें**: this is *filter only*, because there's no other word that's common enough in Hindi for *to narrow*. -* Mute/Unmute - **शांत/अशांत** -* Deactivate/Reactivate - **निष्क्रिय करें / पुन: सक्रिय करें** -* Search - **खोज करें/ढूंढे** -* Pin - **ठीक करना** -* Mention/@mention - **ज़िक्र करना / @ ज़िक्र करना** -* Invalid - **अमान्य** -* Customization - **अनुकूलन** -* I want - **मुझे चाहिए** -* User - **उपयोगकर्ता** -* Person/People - **व्यक्ति/लोग**: "लोग" is the correct plural for +- Mute/Unmute - **शांत/अशांत** +- Deactivate/Reactivate - **निष्क्रिय करें / पुन: सक्रिय करें** +- Search - **खोज करें/ढूंढे** +- Pin - **ठीक करना** +- Mention/@mention - **ज़िक्र करना / @ ज़िक्र करना** +- Invalid - **अमान्य** +- Customization - **अनुकूलन** +- I want - **मुझे चाहिए** +- User - **उपयोगकर्ता** +- Person/People - **व्यक्ति/लोग**: "लोग" is the correct plural for "व्यक्ति", but when talking of *लोग* referring to it as a crowd, we use "भीड़" instead. ## Others(अन्य) -* You - **तुम**: also "आप" if it's in plural. -* We - **हम** -* Message table - **संदेश बोर्ड** -* Enter/Intro - **दर्ज / परिचय** +- You - **तुम**: also "आप" if it's in plural. +- We - **हम** +- Message table - **संदेश बोर्ड** +- Enter/Intro - **दर्ज / परिचय** diff --git a/docs/translating/internationalization.md b/docs/translating/internationalization.md index e0c0cf0dbf..607a6fd45d 100644 --- a/docs/translating/internationalization.md +++ b/docs/translating/internationalization.md @@ -9,22 +9,22 @@ internationalization and Zulip's tools for it so that they can make correct decisions about how to tag strings for translation. A few principles are important in how we think about internationalization: -* Our goal is for **all end-user facing strings** in Zulip to be +- Our goal is for **all end-user facing strings** in Zulip to be tagged for translation in both [HTML templates](#html-templates) and code, and our linters attempt to enforce this. There are some exceptions: we don't tag strings in Zulip's landing pages (e.g. /features) and other documentation (e.g. /help) for translation at this time (though we do aim for those pages to be usable with tools like Google Translate). -* Translating all the strings in Zulip for a language and maintaining +- Translating all the strings in Zulip for a language and maintaining that translation is a lot of work, and that work scales with the number of strings tagged for translation in Zulip. For this reason, we put significant effort into only tagging for translation content that will actually be displayed to users, and minimizing unnecessary user-facing strings in the product. -* In order for a translated user experience to be good, every UI +- In order for a translated user experience to be good, every UI element needs to be built in a way that supports i18n. -* This is more about string consistency in general, but we have a +- This is more about string consistency in general, but we have a "Sentence case" [capitalization policy](../translating/translating.html#capitalization) that we enforce using linters that check all strings tagged for translation in Zulip. @@ -41,22 +41,22 @@ their style guidelines. There are a few critical details about human language that are important to understand when implementing an internationalized application: -* **Punctuation** varies between languages (e.g. Japanese doesn't use +- **Punctuation** varies between languages (e.g. Japanese doesn't use `.`s at the end of sentences). This means that you should always include end-of-sentence symbols like `.` and `?` inside the to-be-translated strings, so that translators can correctly translate the content. -* **Word order** varies between languages (e.g. some languages put +- **Word order** varies between languages (e.g. some languages put subjects before verbs, others the other way around). This means that **concatenating translateable strings** produces broken results (more details with examples are below). -* The **width of the string needed to express something** varies +- The **width of the string needed to express something** varies dramatically between languages; this means you can't just hardcode a button or widget to look great for English and expect it to work in all languages. German is a good test case, as it has a lot of long words, as is Japanese (as character-based languages use a lot less width). -* This is more about how i18n tooling works, but in code, the +- This is more about how i18n tooling works, but in code, the translation function must be passed the string to translate, not a variable containing the target string. Otherwise, the parsers that extract the strings in a project to send to translators will not diff --git a/docs/translating/polish.md b/docs/translating/polish.md index 89f786a5ab..5ffb37cb54 100644 --- a/docs/translating/polish.md +++ b/docs/translating/polish.md @@ -2,33 +2,33 @@ Use semi-formal Polish for translation, some specifics: -* Informal "you" (*ty*) instead of more formal approaches (e.g. plural +- Informal "you" (*ty*) instead of more formal approaches (e.g. plural "you" (*wy*), using any formal titles like *Państwo*, *Pan/Pani*). -* Gender-neutral forms of verbs, e.g. *unsubscribed* - *odsubskrybowano*, +- Gender-neutral forms of verbs, e.g. *unsubscribed* - *odsubskrybowano*, not *odsubskrybowałeś". -* Imperative, active and continuous verbs, e.g. *manage streams* - +- Imperative, active and continuous verbs, e.g. *manage streams* - *zarządzaj kanałami*, not *zarządź kanałami*. -* Not using reflexive *się*, e.g. *log out* would be simply *wyloguj*, +- Not using reflexive *się*, e.g. *log out* would be simply *wyloguj*, not *wyloguj się*. -* Warm and friendly phrasing whenever appropriate. +- Warm and friendly phrasing whenever appropriate. -* No slang or regional phrases that could be unclear or too informal, +- No slang or regional phrases that could be unclear or too informal, e.g. *zajawka*. -* Consistent usage of Zulip-specific terms and common verbs for +- Consistent usage of Zulip-specific terms and common verbs for actions, even if it means repeating - this is one of the key aspects of "semi-formal", as synonyms would be often more appropriate in written Polish. -* Mindful usage of long words and phrases - it's sometimes hard to +- Mindful usage of long words and phrases - it's sometimes hard to translate English to Polish concisely, be mindful of how it looks on the frontend after translating. -* Balance common verbs and nouns with specific IT-related translations +- Balance common verbs and nouns with specific IT-related translations of English terms - this can be tricky, try to check how other resources were translated (e.g. Gmail, Microsoft websites, Facebook) to decide what wouldn't sound awkward in Polish. diff --git a/docs/translating/russian.md b/docs/translating/russian.md index 4c8de9018c..287d4ffb7f 100644 --- a/docs/translating/russian.md +++ b/docs/translating/russian.md @@ -4,56 +4,56 @@ качества перевода. Они продиктованы опытом - собственным и коллег, здравым смыслом и техническими особенностями продукта. -* Переводите you, как "вы", не как "Вы" -* Не используйте "ё" -* Не переводите Zulip -* Избегайте IT-жаргонизмы и кальки. Например, не приватный, а личный; +- Переводите you, как "вы", не как "Вы" +- Не используйте "ё" +- Не переводите Zulip +- Избегайте IT-жаргонизмы и кальки. Например, не приватный, а личный; не email, а электронная почта / адрес электронной почты. -* Фразы должны правильно читаться и передавать исходный смысл. Для +- Фразы должны правильно читаться и передавать исходный смысл. Для этого анализируйте контекст фразы. -* Вместо изобретения собственных терминов, используйте устоявшиеся +- Вместо изобретения собственных терминов, используйте устоявшиеся фразы из других чатов: Slack, VKontakte, Skype - то, к чему уже привыкли люди. ## Перевод некоторых терминов -* login - **войти** -* sign up - **зарегистрироваться** -* stream - **канал** -* organization - **организация** -* message - **сообщение** -* messages - **переписка** -* conversation - **беседа** -* chat - **чат** -* send - **отправить** -* email - **адрес электронной почты** -* feedback - **обратная связь** -* user - **пользователь** -* topic - **тема** -* narrow - **показать только** -* narrowing - **уточнение** -* private - **личный** -* alert word - **сигнальное слово** -* upload - **загрузить** -* uploads - **файлы** -* settings - **настройки** -* invalid - **неверный** -* incorrect - **неправильный** -* unknown - **неизвестный** -* account - **учетная запись** -* subdomain - **поддомен** -* API key - **API-ключ** -* recipient - **получатель** -* subscriber - **участник** -* invite-only - **закрытый** -* public - **открытый** -* name - **название** for things, **имя** for people -* id - **код** -* notifications - **оповещения** -* @-mentions - **@-упоминания** -* mute - **заглушить** -* emoji - **эмодзи** -* user - **пользователь** -* compose - **написать** -* custom - **дополнительный** -* home view - **общий список сообщений** +- login - **войти** +- sign up - **зарегистрироваться** +- stream - **канал** +- organization - **организация** +- message - **сообщение** +- messages - **переписка** +- conversation - **беседа** +- chat - **чат** +- send - **отправить** +- email - **адрес электронной почты** +- feedback - **обратная связь** +- user - **пользователь** +- topic - **тема** +- narrow - **показать только** +- narrowing - **уточнение** +- private - **личный** +- alert word - **сигнальное слово** +- upload - **загрузить** +- uploads - **файлы** +- settings - **настройки** +- invalid - **неверный** +- incorrect - **неправильный** +- unknown - **неизвестный** +- account - **учетная запись** +- subdomain - **поддомен** +- API key - **API-ключ** +- recipient - **получатель** +- subscriber - **участник** +- invite-only - **закрытый** +- public - **открытый** +- name - **название** for things, **имя** for people +- id - **код** +- notifications - **оповещения** +- @-mentions - **@-упоминания** +- mute - **заглушить** +- emoji - **эмодзи** +- user - **пользователь** +- compose - **написать** +- custom - **дополнительный** +- home view - **общий список сообщений** diff --git a/docs/translating/spanish.md b/docs/translating/spanish.md index 189d87dc9c..f5192a845e 100644 --- a/docs/translating/spanish.md +++ b/docs/translating/spanish.md @@ -2,28 +2,28 @@ Use informal Spanish for translation: -* Informal "you" (*tú*) instead of formal form *usted*. Many top software +- Informal "you" (*tú*) instead of formal form *usted*. Many top software companies (e.g. Google) use the informal one, because it's much more common in the daily language and avoids making translations look like they were written by machines. -* Imperative, active, and continuous verbs, e.g. *manage streams* - +- Imperative, active, and continuous verbs, e.g. *manage streams* - *gestionar canales*, not *gestión de canales*. -* Not using reflexive *se* e.g. *log out* should be *salir*, not *salirse*, +- Not using reflexive *se* e.g. *log out* should be *salir*, not *salirse*, whenever the infinitive form is possible without making the translation awkward. -* Warm and friendly phrasing whenever appropriate. +- Warm and friendly phrasing whenever appropriate. -* No slang or regional phrases that could be unclear or too informal. +- No slang or regional phrases that could be unclear or too informal. -* Balance common verbs and nouns with specific IT-related translations +- Balance common verbs and nouns with specific IT-related translations of English terms - this can be tricky, try to check how other resources were translated (e.g. Gmail, Microsoft websites, Facebook) to decide what wouldn't sound awkward / rude in Spanish. -* Latest RAE rule ("solo" should +- Latest RAE rule ("solo" should [**never**](https://www.rae.es/consultas/el-adverbio-solo-y-los-pronombres-demostrativos-sin-tilde) have accent, even when it can be replaced with "solamente"). @@ -32,54 +32,54 @@ with other Spanish speakers in the community. It's all about making Zulip friendly and usable. ## Términos -* Message - **Mensaje** -* Private message (PM) - **Mensaje privado (MP)** -* Group PM - **mensaje privado grupal**: even though "MP grupal" is the most +- Message - **Mensaje** +- Private message (PM) - **Mensaje privado (MP)** +- Group PM - **mensaje privado grupal**: even though "MP grupal" is the most precise translation, preferably don't use that one. Many users may not associate "MP" with *private message* in a group context. Better use it without abbreviations. -* Stream - **Canal**: the use of the literal Spanish word for stream +- Stream - **Canal**: the use of the literal Spanish word for stream "Flujo" is very confusing and not the correct metaphor for Spanish speaking people. The correct term would be "canal" (*channel*). -* Topic - **Tema** -* Private/invite-only stream - **Canal privado/limitado por invitación**: (lit. +- Topic - **Tema** +- Private/invite-only stream - **Canal privado/limitado por invitación**: (lit. *channel limited by invitation*) -* Public stream - **Canal público** -* Bot - **Bot** -* Embedded bot - **Bot integrado** -* Interactive bot - **Bot interactivo** -* Integration - **Integración** -* Notification - **Notificación** -* Alert word - **Alerta**: this is only *alert*. Nonetheless, adding *word* may +- Public stream - **Canal público** +- Bot - **Bot** +- Embedded bot - **Bot integrado** +- Interactive bot - **Bot interactivo** +- Integration - **Integración** +- Notification - **Notificación** +- Alert word - **Alerta**: this is only *alert*. Nonetheless, adding *word* may make the term confusing (something like *danger!* could be a "palabra de alerta" as well). Google Alerts uses "alerta" in its Spanish translation. -* View - **Vista** -* Filter - **Filtro**: as used with narrowing (see below). -* Home - **Inicio**: we never use the term "Hogar" (literally home) in Spanish. -* Emoji - **Emoticono** (plural: **emoticonos**) -* Slash command - **/comando** -* Webhook - **Webhook** -* Endpoint - **Endpoint** +- View - **Vista** +- Filter - **Filtro**: as used with narrowing (see below). +- Home - **Inicio**: we never use the term "Hogar" (literally home) in Spanish. +- Emoji - **Emoticono** (plural: **emoticonos**) +- Slash command - **/comando** +- Webhook - **Webhook** +- Endpoint - **Endpoint** ## Frases -* Subscribe/Unsubscribe to a stream - **Suscribir a/Desuscribir de un canal** -* Narrow to - **Filtrar solo**: this is *filter only*, because there's no other +- Subscribe/Unsubscribe to a stream - **Suscribir a/Desuscribir de un canal** +- Narrow to - **Filtrar solo**: this is *filter only*, because there's no other word that's common enough in Spanish for *to narrow* except for "filtrar". -* Mute/Unmute - **Silenciar/No silenciar** -* Deactivate/Reactivate - **Desactivar/Reactivar** -* Search - **Buscar** -* Pin - **Fijar** (lit. *to fixate*) -* Mention/@mention - **Mencionar/@mención** -* Invalid - **Inválido** -* Customization - **Personalización** -* I want - **Yo quiero** -* User - **Usuario** -* Person/People - **Persona/Personas**: "personas" is the correct plural for +- Mute/Unmute - **Silenciar/No silenciar** +- Deactivate/Reactivate - **Desactivar/Reactivar** +- Search - **Buscar** +- Pin - **Fijar** (lit. *to fixate*) +- Mention/@mention - **Mencionar/@mención** +- Invalid - **Inválido** +- Customization - **Personalización** +- I want - **Yo quiero** +- User - **Usuario** +- Person/People - **Persona/Personas**: "personas" is the correct plural for "person", but when talking of *people* referring to it as a crowd, we use "gente" instead. ## Otros -* You - **Tú**: also "vosotros" if it's in plural. -* We - **Nosotros** -* Message table - **Tablón de mensajes** -* Enter/Intro - **Enter/Intro** +- You - **Tú**: also "vosotros" if it's in plural. +- We - **Nosotros** +- Message table - **Tablón de mensajes** +- Enter/Intro - **Enter/Intro** diff --git a/docs/translating/translating.md b/docs/translating/translating.md index 4ab77af022..5854dc2eb0 100644 --- a/docs/translating/translating.md +++ b/docs/translating/translating.md @@ -44,13 +44,13 @@ to any languages that you'd like to contribute to (or add new ones). 1. Translate the strings for your language in Transifex. Zulip has several resource files: - * `mobile.json` is for the iOS/Android mobile apps. - * `desktop.json` is for the parts of the Zulip desktop apps that + - `mobile.json` is for the iOS/Android mobile apps. + - `desktop.json` is for the parts of the Zulip desktop apps that are not shared with the Zulip webapp. - * `django.po` and `translations.json` have strings for the next + - `django.po` and `translations.json` have strings for the next major release of the Zulip server and webapp (which is what we run on chat.zulip.org and Zulip Cloud). - * The variants of `django.po` and `translations.json` with names + - The variants of `django.po` and `translations.json` with names starting with a version, like, `4-x--`, are strings for Zulip's current [stable release series](../overview/release-lifecycle.md). @@ -108,7 +108,7 @@ This section assumes you have a if setting one up is a problem for you, ask in chat.zulip.org and we can usually just deploy the latest translations there. -* First, download the updated resource files from Transifex using the +- First, download the updated resource files from Transifex using the `tools/i18n/sync-translations` command (it will require some [initial setup](../translating/internationalization.html#transifex-cli-setup)). This command will download the resource files from Transifex and replace @@ -117,19 +117,19 @@ now test your translation work in the Zulip UI. There are a few ways to see your translations in the Zulip UI: -* You can insert the language code as a URL prefix. For example, you +- You can insert the language code as a URL prefix. For example, you can view the login page in German using `http://localhost:9991/de/login/`. This works for any part of the Zulip UI, including portico (logged-out) pages. -* For Zulip's logged-in UI (i.e. the actual webapp), you can [pick the +- For Zulip's logged-in UI (i.e. the actual webapp), you can [pick the language](https://zulip.com/help/change-your-language) in the Zulip UI. -* If your system has languages configured in your OS/browser, Zulip's +- If your system has languages configured in your OS/browser, Zulip's portico (logged-out) pages will automatically use your configured language. Note that we only tag for translation strings in pages that individual users need to use (e.g. `/login/`, `/register/`, etc.), not marketing pages like `/features/`. -* In case you need to understand how the above interact, Zulip figures +- In case you need to understand how the above interact, Zulip figures out the language the user requests in a browser using the following prioritization (mostly copied from the Django docs): @@ -139,7 +139,7 @@ There are a few ways to see your translations in the Zulip UI: 1. It looks for the `Accept-Language` HTTP header in the HTTP request (this is how browsers tell Zulip about the OS/browser language). -* Using an HTTP client library like `requests`, `cURL` or `urllib`, +- Using an HTTP client library like `requests`, `cURL` or `urllib`, you can pass the `Accept-Language` header; here is some sample code to test `Accept-Language` header using Python and `requests`: @@ -159,13 +159,13 @@ Zulip should be translated into specific languages (e.g. what word to translate words like "stream" to), with reasoning, so that future translators can understand and preserve those decisions: -* [Chinese](chinese.md) -* [French](french.md) -* [German](german.md) -* [Hindi](hindi.md) -* [Polish](polish.md) -* [Russian](russian.md) -* [Spanish](spanish.md) +- [Chinese](chinese.md) +- [French](french.md) +- [German](german.md) +- [Hindi](hindi.md) +- [Polish](polish.md) +- [Russian](russian.md) +- [Spanish](spanish.md) Some translated languages don't have these, but we highly encourage translators for new languages (or those updating a language) write a @@ -181,13 +181,13 @@ We expect that all the English translatable strings in Zulip are properly capitalized in a way consistent with how Zulip does capitalization in general. This means that: -* The first letter of a sentence or phrase should be capitalized. +- The first letter of a sentence or phrase should be capitalized. - Correct: "Manage streams" - Incorrect: "Manage Streams" -* All proper nouns should be capitalized. +- All proper nouns should be capitalized. - Correct: "This is Zulip" - Incorrect: "This is zulip" -* All common words like URL, HTTP, etc. should be written in their +- All common words like URL, HTTP, etc. should be written in their standard forms. - Correct: "URL" - Incorrect: "Url" diff --git a/docs/tutorials/life-of-a-request.md b/docs/tutorials/life-of-a-request.md index ca3d172404..604c9c9177 100644 --- a/docs/tutorials/life-of-a-request.md +++ b/docs/tutorials/life-of-a-request.md @@ -89,15 +89,15 @@ With the exception of incoming webhooks (which we do not usually control the format of), legacy endpoints, and logged-out endpoints, Zulip uses REST for its API. This means that we use: -* POST for creating something new where we don't have a unique +- POST for creating something new where we don't have a unique ID. Also used as a catch-all if no other verb is appropriate. -* PUT for creating something for which we have a unique ID. -* DELETE for deleting something -* PATCH for updating or editing attributes of something. -* GET to get something (read-only) -* HEAD to check the existence of something to GET, without getting it; +- PUT for creating something for which we have a unique ID. +- DELETE for deleting something +- PATCH for updating or editing attributes of something. +- GET to get something (read-only) +- HEAD to check the existence of something to GET, without getting it; useful to check a link without downloading a potentially large link -* OPTIONS (handled automatically, see more below) +- OPTIONS (handled automatically, see more below) Of these, PUT, DELETE, HEAD, OPTIONS, and GET are *idempotent*, which means that we can send the request multiple times and get the same diff --git a/docs/tutorials/new-feature-tutorial.md b/docs/tutorials/new-feature-tutorial.md index 19c2378634..04cf4df77c 100644 --- a/docs/tutorials/new-feature-tutorial.md +++ b/docs/tutorials/new-feature-tutorial.md @@ -202,9 +202,9 @@ dictionary. `property_types`.** However, there are some properties that need custom logic and thus cannot use this framework. For example: -* The realm `authentication_methods` attribute is a bitfield and needs +- The realm `authentication_methods` attribute is a bitfield and needs additional code for validation and updating. -* The `allow_message_editing` and `message_content_edit_limit_seconds` +- The `allow_message_editing` and `message_content_edit_limit_seconds` fields depend on one another, so they are also handled separately and not included in `property_types`. diff --git a/docs/tutorials/writing-views.md b/docs/tutorials/writing-views.md index 4de15a7ef6..e60a4a35ff 100644 --- a/docs/tutorials/writing-views.md +++ b/docs/tutorials/writing-views.md @@ -129,11 +129,11 @@ view, you need to write code to parse and validate that the arguments exist and have the correct form. For many applications, this leads to one of several bad outcomes: -* The code isn't written, so arguments aren't validated, leading to +- The code isn't written, so arguments aren't validated, leading to bugs and confusing error messages for users of the API. -* Every function starts with a long list of semi-redundant validation +- Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error messages. -* Every view function comes with another function that does the +- Every view function comes with another function that does the validation that has the problems from the last bullet point. In Zulip, we solve this problem with a the special decorator called @@ -178,22 +178,22 @@ in REQ also helps us with request variable validation. For example: -* `msg_ids = REQ(json_validator=check_list(check_int))` will check +- `msg_ids = REQ(json_validator=check_list(check_int))` will check that the `msg_ids` HTTP parameter is a list of integers, marshalled as JSON, and pass it into the function as the `msg_ids` Python keyword argument. -* `streams_raw = REQ("subscriptions", json_validator=check_list(check_string))` +- `streams_raw = REQ("subscriptions", json_validator=check_list(check_string))` will check that the "subscriptions" HTTP parameter is a list of strings, marshalled as JSON, and pass it into the function with the Python keyword argument `streams_raw`. -* `message_id=REQ(converter=to_non_negative_int)` will check that the +- `message_id=REQ(converter=to_non_negative_int)` will check that the `message_id` HTTP parameter is a string containing a non-negative integer (`converter` differs from `json_validator` in that it does not automatically marshall the input from JSON). -* Since there is no need to JSON-encode strings, usually simply +- Since there is no need to JSON-encode strings, usually simply `my_string=REQ()` is correct. One can pass e.g. `str_validator=check_string_in(...)` where one wants to run a validator on the value of a string. diff --git a/scripts/README.md b/scripts/README.md index 5da531cd60..17892b32ed 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,9 +1,9 @@ This directory contains scripts that: -* Generally do not require access to Django or the database (those are +- Generally do not require access to Django or the database (those are "management commands"), and thus are suitable to run operationally. -* Are useful for managing a production deployment of Zulip (many are +- Are useful for managing a production deployment of Zulip (many are also used in a Zulip development environment, though development-only scripts live in `tools/`). diff --git a/static/shared/README.md b/static/shared/README.md index 5625337593..5d350fd127 100644 --- a/static/shared/README.md +++ b/static/shared/README.md @@ -3,12 +3,12 @@ and are also incorporated by the Zulip mobile app. Note that the deployment cycles are different: - * In the webapp, this code is deployed in the same way as the rest of + - In the webapp, this code is deployed in the same way as the rest of the web frontend: it's part of the server tree, and the browser gets it from the server, so the client is always running the same version the server just gave it. - * In the mobile app, this code is deployed in the same way as the + - In the mobile app, this code is deployed in the same way as the rest of the mobile app: it's bundled up into the app binary which is uploaded to app stores and users install on their devices. The client will be running the version built into their version of the diff --git a/tools/droplets/README.md b/tools/droplets/README.md index 1f41e32de5..78319078d9 100644 --- a/tools/droplets/README.md +++ b/tools/droplets/README.md @@ -52,8 +52,8 @@ Now you're ready to use the script. `create.py` takes two arguments -* GitHub username -* Tags (Optional argument) +- GitHub username +- Tags (Optional argument) ``` $ python3 create.py diff --git a/tools/linter_lib/custom_check.py b/tools/linter_lib/custom_check.py index 740ee29d69..6ab8725b08 100644 --- a/tools/linter_lib/custom_check.py +++ b/tools/linter_lib/custom_check.py @@ -521,7 +521,7 @@ prose_style_rules: List["Rule"] = [ { "pattern": "[oO]rganisation", # exclude usage in hrefs/divs "description": "Organization is spelled with a z", - "exclude_line": {("docs/translating/french.md", "* organization - **organisation**")}, + "exclude_line": {("docs/translating/french.md", "- organization - **organisation**")}, }, {"pattern": "!!! warning", "description": "!!! warning is invalid; it's spelled '!!! warn'"}, {"pattern": "Terms of service", "description": "The S in Terms of Service is capitalized"}, diff --git a/tools/oneclickapps/README.md b/tools/oneclickapps/README.md index d6cd207eb1..608d5d1f77 100644 --- a/tools/oneclickapps/README.md +++ b/tools/oneclickapps/README.md @@ -18,42 +18,42 @@ server release. You also need to set the following secrets in your GitHub repository to make the action work correctly. These secrets are passed as environment variables to the GitHub action. -* `ONE_CLICK_ACTION_DIGITALOCEAN_API_KEY` - DigitalOcean API key used for creating droplets, snapshots etc. -* `ONE_CLICK_ACTION_ZULIP_BOT_API_KEY` - The API key of the Zulip bot used for sending messages. -* `ONE_CLICK_ACTION_ZULIP_BOT_EMAIL` - The email of the Zulip bot. +- `ONE_CLICK_ACTION_DIGITALOCEAN_API_KEY` - DigitalOcean API key used for creating droplets, snapshots etc. +- `ONE_CLICK_ACTION_ZULIP_BOT_API_KEY` - The API key of the Zulip bot used for sending messages. +- `ONE_CLICK_ACTION_ZULIP_BOT_EMAIL` - The email of the Zulip bot. Also pass the following as environment variables in `.github/workflows/update-oneclick-apps.yml`. -* `PYTHON_DIGITALOCEAN_REQUEST_TIMEOUT_SEC` - This configures the maximum number of seconds +- `PYTHON_DIGITALOCEAN_REQUEST_TIMEOUT_SEC` - This configures the maximum number of seconds to wait before the requests made by `python-digitalocean` time out. If not configured, it's common for the requests to take 20+ minutes before getting timed out. ### Verifying the one click app image -* The action will send the image name and test droplet details to the stream configured in the +- The action will send the image name and test droplet details to the stream configured in the above steps. -* SSH into the test droplet by following the instructions in the message. -* After logging into the test droplet, exit the installer and run the following script. +- SSH into the test droplet by following the instructions in the message. +- After logging into the test droplet, exit the installer and run the following script. https://raw.githubusercontent.com/digitalocean/marketplace-partners/master/scripts/img_check.sh This script checks whether the image created is valid. It is also run by the DigitalOcean team before they approve the image submission in the one click app marketplace. -* If there are no errors (you can ignore most of the warnings), exit the SSH connection and +- If there are no errors (you can ignore most of the warnings), exit the SSH connection and reconnect. -* Populate the details asked by the installer and verify that the installer completes successfully. +- Populate the details asked by the installer and verify that the installer completes successfully. If there are errors see the section below. -* Use the link generated by the installer to create a new Zulip organization. Do some basic +- Use the link generated by the installer to create a new Zulip organization. Do some basic testing like sending a bunch of messages and reloading the webpage. -* If there are no issues, submit the image in the +- If there are no issues, submit the image in the [DigitalOcean vendor portal](https://marketplace.digitalocean.com/vendorportal). You need to be added to the vendor portal team for doing this. Ask Tim to add you if required. During the submission, make sure to update the blog post URL if it's a major release. -* Keep checking the vendor portal for change in status of the submission. DigitalOcean does nominally send +- Keep checking the vendor portal for change in status of the submission. DigitalOcean does nominally send emails when there are updates on the submission, but we have found the emails to not always arrive. -* If there are any issues with submission, rebuild the image by manually invoking the script and +- If there are any issues with submission, rebuild the image by manually invoking the script and resubmit. The issues we have seen mostly in the past are caused by the dependencies getting outdated by the time the DigitalOcean team run the checks. In that case you have to just rebuild the image by invoking the script. -* Delete the test droplet `oneclickapp-{release_version}-test` after you have completed testing +- Delete the test droplet `oneclickapp-{release_version}-test` after you have completed testing by going to the [DigitalOcean Zulip team account](https://cloud.digitalocean.com/droplets?i=0242e0). If there are other existing test droplets with the same name format but with with older release versions feel free to delete them as well. These droplets are also tagged with `github-action` and`temporary` @@ -62,9 +62,9 @@ Also pass the following as environment variables in `.github/workflows/update-on **Errors** If there are any errors while setting up the one click app installer, you have three options -* Include the fix in the Fabric script that setups the installer. +- Include the fix in the Fabric script that setups the installer. [01-initial-setup](https://raw.githubusercontent.com/zulip/marketplace-partners/master/marketplace_docs/templates/Fabric/scripts/01-initial-setup) file should be a good place to include the fix. See [zulip/marketplace-partners#4](https://github.com/zulip/marketplace-partners/pull/4/files) for an example fix. -* Wait for the next release to fix the error. +- Wait for the next release to fix the error.