mirror of
https://github.com/zulip/zulip.git
synced 2025-11-02 04:53:36 +00:00
docs: Apply sentence single-spacing from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit is contained in:
Anders Kaseorg
committed by
Anders Kaseorg
Anders Kaseorg
parent
915884bff7
commit
35c1c8d41b
@@ -1,22 +1,22 @@
|
||||
# Provisioning and third-party dependencies
|
||||
|
||||
Zulip is a large project, with well over 100 third-party dependencies,
|
||||
and managing them well is essential to the quality of the project. In
|
||||
and managing them well is essential to the quality of the project. In
|
||||
this document, we discuss the various classes of dependencies that
|
||||
Zulip has, and how we manage them. Zulip's dependency management has
|
||||
Zulip has, and how we manage them. Zulip's dependency management has
|
||||
some really nice properties:
|
||||
|
||||
- **Fast provisioning**. When switching to a different commit in the
|
||||
- **Fast provisioning**. When switching to a different commit in the
|
||||
Zulip project with the same dependencies, it takes under 5 seconds
|
||||
to re-provision a working Zulip development environment after
|
||||
switching. If there are new dependencies, one only needs to wait to
|
||||
switching. If there are new dependencies, one only needs to wait to
|
||||
download the new ones, not all the pre-existing dependencies.
|
||||
- **Consistent provisioning**. Every time a Zulip development or
|
||||
- **Consistent provisioning**. Every time a Zulip development or
|
||||
production environment is provisioned/installed, it should end up
|
||||
using the exactly correct versions of all major dependencies.
|
||||
- **Low maintenance burden**. To the extent possible, we want to
|
||||
- **Low maintenance burden**. To the extent possible, we want to
|
||||
avoid manual work and keeping track of things that could be
|
||||
automated. This makes it easy to keep running the latest versions
|
||||
automated. This makes it easy to keep running the latest versions
|
||||
of our various dependencies.
|
||||
|
||||
The purpose of this document is to detail all of Zulip's third-party
|
||||
@@ -25,11 +25,11 @@ dependencies and how we manage their versions.
|
||||
## Provisioning
|
||||
|
||||
We refer to "provisioning" as the process of installing and
|
||||
configuring the dependencies of a Zulip development environment. It's
|
||||
configuring the dependencies of a Zulip development environment. It's
|
||||
done using `tools/provision`, and the output is conveniently logged by
|
||||
`var/log/provision.log` to help with debugging. Provisioning makes
|
||||
use of a lot of caching. Some of those caches are not immune to being
|
||||
corrupted if you mess around with files in your repository a lot. We
|
||||
`var/log/provision.log` to help with debugging. Provisioning makes
|
||||
use of a lot of caching. Some of those caches are not immune to being
|
||||
corrupted if you mess around with files in your repository a lot. We
|
||||
have `tools/provision --force` to (still fairly quickly) rerun most
|
||||
steps that would otherwise have been skipped due to caching.
|
||||
|
||||
@@ -42,13 +42,13 @@ also run an initial provision the first time only.
|
||||
In `version.py`, we have a special parameter, `PROVISION_VERSION`,
|
||||
which is used to help ensure developers don't spend time debugging
|
||||
test/linter/etc. failures that actually were caused by the developer
|
||||
rebasing and forgetting to provision". `PROVISION_VERSION` has a
|
||||
rebasing and forgetting to provision". `PROVISION_VERSION` has a
|
||||
format of `x.y`; when `x` doesn't match the value from the last time
|
||||
the user provisioned, or `y` is higher than than the value from last
|
||||
time, most Zulip tools will crash early and ask the user to provision.
|
||||
This has empirically made a huge impact on how often developers spend
|
||||
time debugging a "weird failure" after rebasing that had an easy
|
||||
solution. (Of course, the other key part of achieving this is all the
|
||||
solution. (Of course, the other key part of achieving this is all the
|
||||
work that goes into making sure that `provision` reliably leaves the
|
||||
development environment in a good state.)
|
||||
|
||||
@@ -58,29 +58,29 @@ require re-running provision, so don't forget about it!
|
||||
## Philosophy on adding third-party dependencies
|
||||
|
||||
In the Zulip project, we take a pragmatic approach to third-party
|
||||
dependencies. Overall, if a third-party project does something well
|
||||
dependencies. Overall, if a third-party project does something well
|
||||
that Zulip needs to do (and has an appropriate license), we'd love to
|
||||
use it rather than reinventing the wheel. If the third-party project
|
||||
use it rather than reinventing the wheel. If the third-party project
|
||||
needs some small changes to work, we prefer to make those changes and
|
||||
contribute them upstream. When the upstream maintainer is slow to
|
||||
contribute them upstream. When the upstream maintainer is slow to
|
||||
respond, we may use a fork of the dependency until the code is merged
|
||||
upstream; as a result, we usually have a few packages in
|
||||
`requirements.txt` that are installed from a GitHub URL.
|
||||
|
||||
What we look for in choosing dependencies is whether the project is
|
||||
well-maintained. Usually one can tell fairly quickly from looking at
|
||||
well-maintained. Usually one can tell fairly quickly from looking at
|
||||
a project's issue tracker how well-managed it is: a quick look at how
|
||||
the issue tracker is managed (or not) and the test suite is usually
|
||||
enough to decide if a project is going to be a high-maintenance
|
||||
dependency or not. That said, we do still take on some smaller
|
||||
dependency or not. That said, we do still take on some smaller
|
||||
dependencies that don't have a well-managed project, if we feel that
|
||||
using the project will still be a better investment than writing our
|
||||
own implementation of that project's functionality. We've adopted a
|
||||
own implementation of that project's functionality. We've adopted a
|
||||
few projects in the past that had a good codebase but whose maintainer
|
||||
no longer had time for them.
|
||||
|
||||
One case where we apply added scrutiny to third-party dependencies is
|
||||
JS libraries. They are a particularly important concern because we
|
||||
JS libraries. They are a particularly important concern because we
|
||||
want to keep the Zulip web app's JS bundle small, so that Zulip
|
||||
continues to load quickly on systems with low network bandwidth.
|
||||
We'll look at large JS libraries with much greater scrutiny for
|
||||
@@ -94,10 +94,10 @@ For the third-party services like PostgreSQL, Redis, Nginx, and RabbitMQ
|
||||
that are documented in the
|
||||
[architecture overview](../overview/architecture-overview.md), we rely on the
|
||||
versions of those packages provided alongside the Linux distribution
|
||||
on which Zulip is deployed. Because Zulip
|
||||
on which Zulip is deployed. Because Zulip
|
||||
[only supports Ubuntu in production](../production/requirements.md), this
|
||||
usually means `apt`, though we do support
|
||||
[other platforms in development](../development/setup-advanced.md). Since
|
||||
[other platforms in development](../development/setup-advanced.md). Since
|
||||
we don't control the versions of these dependencies, we avoid relying
|
||||
on specific versions of these packages wherever possible.
|
||||
|
||||
@@ -107,7 +107,7 @@ few places:
|
||||
the `Package` and `SafePackage` directives.
|
||||
- For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`.
|
||||
- The packages needed to build a Zulip virtualenv, in
|
||||
`VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are
|
||||
`VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are
|
||||
separate from the rest because (1) we may need to install a
|
||||
virtualenv before running the more complex scripts that, in turn,
|
||||
install other dependencies, and (2) because that list is shared
|
||||
@@ -119,7 +119,7 @@ extension, used by our [full-text search](full-text-search.md).
|
||||
## Python packages
|
||||
|
||||
Zulip uses the version of Python itself provided by the host OS for
|
||||
the Zulip server. We currently support Python 3.6 and newer, with
|
||||
the Zulip server. We currently support Python 3.6 and newer, with
|
||||
Ubuntu Bionic being the platform requiring 3.6 support. The comments
|
||||
in `.github/workflows/zulip-ci.yml` document the Python versions used
|
||||
by each supported platform.
|
||||
@@ -127,79 +127,79 @@ by each supported platform.
|
||||
We manage Python packages via the Python-standard `requirements.txt`
|
||||
system and virtualenvs, but there’s a number of interesting details
|
||||
about how Zulip makes this system work well for us that are worth
|
||||
highlighting. The system is largely managed by the code in
|
||||
highlighting. The system is largely managed by the code in
|
||||
`scripts/lib/setup_venv.py`
|
||||
|
||||
- **Using `pip` to manage dependencies**. This is standard in the
|
||||
- **Using `pip` to manage dependencies**. This is standard in the
|
||||
Python ecosystem, and means we only need to record a list of
|
||||
versions in a `requirements.txt` file to declare what we're using.
|
||||
Since we have a few different installation targets, we maintain
|
||||
several `requirements.txt` format files in the `requirements/`
|
||||
directory (e.g. `dev.in` for development, `prod.in` for
|
||||
production, `docs.in` for ReadTheDocs, `common.in` for the vast
|
||||
majority of packages common to prod and development, etc.). We use
|
||||
majority of packages common to prod and development, etc.). We use
|
||||
`pip install --no-deps` to ensure we only install the packages we
|
||||
explicitly declare as dependencies.
|
||||
- **virtualenv with pinned versions**. For a large application like
|
||||
- **virtualenv with pinned versions**. For a large application like
|
||||
Zulip, it is important to ensure that we're always using consistent,
|
||||
predictable versions of all of our Python dependencies. To ensure
|
||||
predictable versions of all of our Python dependencies. To ensure
|
||||
this, we install our dependencies in a [virtualenv][] that contains
|
||||
only the packages and versions that Zulip needs, and we always pin
|
||||
exact versions of our dependencies in our `requirements.txt` files.
|
||||
We pin exact versions, not minimum versions, so that installing
|
||||
Zulip won't break if a dependency makes a buggy release. A side
|
||||
Zulip won't break if a dependency makes a buggy release. A side
|
||||
effect is that it's easy to debug problems caused by dependency
|
||||
upgrades, since we're always doing those upgrades with an explicit
|
||||
commit updating the `requirements/` directory.
|
||||
- **Pinning versions of indirect dependencies**. We "pin" or "lock"
|
||||
- **Pinning versions of indirect dependencies**. We "pin" or "lock"
|
||||
the versions of our indirect dependencies files with
|
||||
`tools/update-locked-requirements` (powered by `pip-compile`). What
|
||||
`tools/update-locked-requirements` (powered by `pip-compile`). What
|
||||
this means is that we have some "source" requirements files, like
|
||||
`requirements/common.in`, that declare the packages that Zulip
|
||||
depends on directly. Those packages have their own recursive
|
||||
dependencies. When adding or removing a dependency from Zulip, one
|
||||
depends on directly. Those packages have their own recursive
|
||||
dependencies. When adding or removing a dependency from Zulip, one
|
||||
simply edits the appropriate "source" requirements files, and then
|
||||
runs `tools/update-locked-requirements`. That tool will use
|
||||
runs `tools/update-locked-requirements`. That tool will use
|
||||
`pip-compile` to generate the locked requirements files like
|
||||
`prod.txt`, `dev.txt` etc files that explicitly declare versions of
|
||||
all of Zulip's recursive dependencies. For indirect dependencies
|
||||
all of Zulip's recursive dependencies. For indirect dependencies
|
||||
(i.e. dependencies not explicitly declared in the source
|
||||
requirements files), it provides helpful comments explaining which
|
||||
direct dependency (or dependencies) needed that indirect dependency.
|
||||
The process for using this system is documented in more detail in
|
||||
`requirements/README.md`.
|
||||
- **Caching of virtualenvs and packages**. To make updating the
|
||||
- **Caching of virtualenvs and packages**. To make updating the
|
||||
dependencies of a Zulip installation efficient, we maintain a cache
|
||||
of virtualenvs named by the hash of the relevant `requirements.txt`
|
||||
file (`scripts/lib/hash_reqs.py`). These caches live under
|
||||
`/srv/zulip-venv-cache/<hash>`. That way, when re-provisioning a
|
||||
file (`scripts/lib/hash_reqs.py`). These caches live under
|
||||
`/srv/zulip-venv-cache/<hash>`. That way, when re-provisioning a
|
||||
development environment or deploying a new production version with
|
||||
the same Python dependencies, no downloading or installation is
|
||||
required: we just use the same virtualenv. When the only changes
|
||||
required: we just use the same virtualenv. When the only changes
|
||||
are upgraded versions, we'll use [virtualenv-clone][] to clone the
|
||||
most similar existing virtualenv and then just upgrade the packages
|
||||
needed, making small version upgrades extremely efficient. And
|
||||
needed, making small version upgrades extremely efficient. And
|
||||
finally, we use `pip`'s built-in caching to ensure that a specific
|
||||
version of a specific package is only downloaded once.
|
||||
- **Garbage-collecting caches**. We have a tool,
|
||||
- **Garbage-collecting caches**. We have a tool,
|
||||
`scripts/lib/clean_venv_cache.py`, which will clean old cached
|
||||
virtualenvs that are no longer in use. In production, the algorithm
|
||||
virtualenvs that are no longer in use. In production, the algorithm
|
||||
preserves recent virtualenvs as well as those in use by any current
|
||||
production deployment directory under `/home/zulip/deployments/`.
|
||||
This helps ensure that a Zulip installation doesn't leak large
|
||||
amounts of disk over time.
|
||||
- **Scripts**. Often, we want a script running in production to use
|
||||
the Zulip virtualenv. To make that work without a lot of duplicated
|
||||
- **Scripts**. Often, we want a script running in production to use
|
||||
the Zulip virtualenv. To make that work without a lot of duplicated
|
||||
code, we have a helpful function,
|
||||
`scripts.lib.setup_path.setup_path`, which on import will put the
|
||||
currently running Python script into the Zulip virtualenv. This is
|
||||
currently running Python script into the Zulip virtualenv. This is
|
||||
called by `./manage.py` to ensure that our Django code always uses
|
||||
the correct virtualenv as well.
|
||||
- **Mypy type checker**. Because we're using mypy in a strict mode,
|
||||
- **Mypy type checker**. Because we're using mypy in a strict mode,
|
||||
when you add use of a new Python dependency, you usually need to
|
||||
either adds stubs to the `stubs/` directory for the library, or edit
|
||||
`pyproject.toml` in the root of the Zulip project to configure
|
||||
`ignore_missing_imports` for the new library. See
|
||||
`ignore_missing_imports` for the new library. See
|
||||
[our mypy docs][mypy-docs] for more details.
|
||||
|
||||
### Upgrading packages
|
||||
@@ -220,29 +220,29 @@ reasoning here.
|
||||
|
||||
- In a fashion very analogous to the Python codebase,
|
||||
`scripts/lib/node_cache.py` manages cached `node_modules`
|
||||
directories in `/srv/zulip-npm-cache`. Each is named by its hash,
|
||||
directories in `/srv/zulip-npm-cache`. Each is named by its hash,
|
||||
computed by the `generate_sha1sum_node_modules` function.
|
||||
`scripts/lib/clean_node_cache.py` handles garbage-collection.
|
||||
- We use [yarn][], a `pip`-like tool for JavaScript, to download most
|
||||
JavaScript dependencies. Yarn talks to standard the [npm][]
|
||||
repository. We use the standard `package.json` file to declare our
|
||||
JavaScript dependencies. Yarn talks to standard the [npm][]
|
||||
repository. We use the standard `package.json` file to declare our
|
||||
direct dependencies, with sections for development and
|
||||
production. Yarn takes care of pinning the versions of indirect
|
||||
production. Yarn takes care of pinning the versions of indirect
|
||||
dependencies in the `yarn.lock` file; `yarn install` updates the
|
||||
`yarn.lock` files.
|
||||
- `tools/update-prod-static`. This process is discussed in detail in
|
||||
- `tools/update-prod-static`. This process is discussed in detail in
|
||||
the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline)
|
||||
article, but we don't use the `node_modules` directories directly in
|
||||
production. Instead, static assets are compiled using our static
|
||||
production. Instead, static assets are compiled using our static
|
||||
asset pipeline and it is the compiled assets that are served
|
||||
directly to users. As a result, we don't ship the `node_modules`
|
||||
directly to users. As a result, we don't ship the `node_modules`
|
||||
directory in a Zulip production release tarball, which is a good
|
||||
thing, because doing so would more than double the size of a Zulip
|
||||
release tarball.
|
||||
- **Checked-in packages**. In contrast with Python, we have a few
|
||||
- **Checked-in packages**. In contrast with Python, we have a few
|
||||
JavaScript dependencies that we have copied into the main Zulip
|
||||
repository under `static/third`, often with patches. These date
|
||||
from an era before `npm` existed. It is a project goal to eliminate
|
||||
repository under `static/third`, often with patches. These date
|
||||
from an era before `npm` existed. It is a project goal to eliminate
|
||||
these checked-in versions of dependencies and instead use versions
|
||||
managed by the npm repositories.
|
||||
|
||||
@@ -256,7 +256,7 @@ its version) and `scripts/lib/install-yarn`.
|
||||
its own path, which we use, though we install a `/usr/local/bin/node`
|
||||
wrapper to access the desired version conveniently and efficiently
|
||||
(`nvm` has a lot of startup overhead).
|
||||
- We install `yarn` at `/srv/zulip-yarn`. We don't do anything
|
||||
- We install `yarn` at `/srv/zulip-yarn`. We don't do anything
|
||||
special to try to manage multiple versions of `yarn`.
|
||||
|
||||
## ShellCheck and shfmt
|
||||
@@ -264,7 +264,7 @@ special to try to manage multiple versions of `yarn`.
|
||||
In the development environment, the `tools/setup/install-shellcheck`
|
||||
and `tools/setup/install-shfmt` scripts download binaries for
|
||||
ShellCheck and shfmt from GitHub, check them against a known hash, and
|
||||
install them to `/usr/local/bin`. These tools are run as part of the
|
||||
install them to `/usr/local/bin`. These tools are run as part of the
|
||||
[linting system](../testing/linters.md).
|
||||
|
||||
## Puppet packages
|
||||
@@ -272,7 +272,7 @@ install them to `/usr/local/bin`. These tools are run as part of the
|
||||
Third-party puppet modules are downloaded from the Puppet Forge into
|
||||
subdirectories under `/srv/zulip-puppet-cache`, hashed based on their
|
||||
versions; the latest is always symlinked as
|
||||
`/srv/zulip-puppet-cache/current`. `zulip-puppet-apply` installs
|
||||
`/srv/zulip-puppet-cache/current`. `zulip-puppet-apply` installs
|
||||
these dependencies immediately before they are needed.
|
||||
|
||||
## Other third-party and generated files
|
||||
@@ -285,9 +285,9 @@ maintain them.
|
||||
### Emoji
|
||||
|
||||
Zulip uses the [iamcal emoji data package][iamcal] for its emoji data
|
||||
and sprite sheets. We download this dependency using `npm`, and then
|
||||
and sprite sheets. We download this dependency using `npm`, and then
|
||||
have a tool, `tools/setup/build_emoji`, which reformats the emoji data
|
||||
into the files under `static/generated/emoji`. Those files are in
|
||||
into the files under `static/generated/emoji`. Those files are in
|
||||
turn used by our [Markdown processor](../subsystems/markdown.md) and
|
||||
`tools/update-prod-static` to make Zulip's emoji work in the various
|
||||
environments where they need to be displayed.
|
||||
@@ -296,9 +296,9 @@ Since processing emoji is a relatively expensive operation, as part of
|
||||
optimizing provisioning, we use the same caching strategy for the
|
||||
compiled emoji data as we use for virtualenvs and `node_modules`
|
||||
directories, with `scripts/lib/clean_emoji_cache.py` responsible for
|
||||
garbage-collection. This caching and garbage-collection is required
|
||||
garbage-collection. This caching and garbage-collection is required
|
||||
because a correct emoji implementation involves over 1000 small image
|
||||
files and a few large ones. There is a more extended article on our
|
||||
files and a few large ones. There is a more extended article on our
|
||||
[emoji infrastructure](emoji.md).
|
||||
|
||||
### Translations data
|
||||
@@ -306,7 +306,7 @@ files and a few large ones. There is a more extended article on our
|
||||
Zulip's [translations infrastructure](../translating/translating.md) generates
|
||||
several files from the source data, which we manage similar to our
|
||||
emoji, but without the caching (and thus without the
|
||||
garbage-collection). New translations data is downloaded from
|
||||
garbage-collection). New translations data is downloaded from
|
||||
Transifex and then compiled to generate both the production locale
|
||||
files and also language data in `locale/language*.json` using
|
||||
`manage.py compilemessages`, which extends the default Django
|
||||
@@ -315,7 +315,7 @@ implementation of that tool.
|
||||
### Pygments data
|
||||
|
||||
The list of languages supported by our Markdown syntax highlighting
|
||||
comes from the [pygments][] package. `tools/setup/build_pygments_data` is
|
||||
comes from the [pygments][] package. `tools/setup/build_pygments_data` is
|
||||
responsible for generating `static/generated/pygments_data.json` so that
|
||||
our JavaScript Markdown processor has access to the supported list.
|
||||
|
||||
@@ -324,12 +324,12 @@ our JavaScript Markdown processor has access to the supported list.
|
||||
When making changes to Zulip's provisioning process or dependencies,
|
||||
usually one needs to think about making changes in 3 places:
|
||||
|
||||
- `tools/lib/provision.py`. This is the main provisioning script,
|
||||
- `tools/lib/provision.py`. This is the main provisioning script,
|
||||
used by most developers to maintain their development environment.
|
||||
- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation"
|
||||
documentation. Strategically, we'd like to move the support for more
|
||||
- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation"
|
||||
documentation. Strategically, we'd like to move the support for more
|
||||
versions of Linux from here into `tools/lib/provision.py`.
|
||||
- Production. Our tools for compiling/generating static assets need
|
||||
- Production. Our tools for compiling/generating static assets need
|
||||
to be called from `tools/update-prod-static`, which is called by
|
||||
`tools/build-release-tarball` (for doing Zulip releases) as well as
|
||||
`tools/upgrade-zulip-from-git` (for deploying a Zulip server off of
|
||||
|
||||
Reference in New Issue
Block a user