Commit Graph

634 Commits

Author SHA1 Message Date
rht
888388cf89 setup-yum-repo: Add -y flag to yum update. 2018-12-17 11:49:43 -08:00
Tim Abbott
2558f101af docs: Add documentation for if False mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott
bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
Tim Abbott
47c4a2d21a upgrade-zulip-from-git: Remove buggy typing import.
This broke running this on Python 3.4, where typing isn't in stdlib.
2018-12-17 10:33:52 -08:00
rht
30834b7dd8 build-tsearch-extras: Abstract out postgres version. 2018-12-16 13:26:29 -08:00
rht
3afb299aec setup-yum-repo: Abstract out postgres version. 2018-12-16 13:21:55 -08:00
rht
903bce9bdd provision: Install pycurl manually on CentOS.
This works around a pycurl bug on EL7 where the fix hasn't been
released yet.
2018-12-16 10:09:07 -08:00
rht
25a8dbef11 provision: Build tsearch_extras from source for CentOS support. 2018-12-16 10:04:27 -08:00
rht
448303b3f0 provision: Initialize setup-yum-repo. 2018-12-16 10:04:27 -08:00
shubham-padia
3231306736 upgrade-zulip: Use deploy_options specified in zulip.conf.
Fixes #10534.
upgrade-zulip can now also accept the same deploy options as
upgrade-zulip-from-git and pass it as arguments to
upgrade-zulip-stage-2.
2018-12-16 07:52:47 -08:00
shubham-padia
29dce7c9b9 upgrade-zulip-from-git: Refactor deploy_options logic to zulip_tools.py.
This a preparatory commit moving the deploy_options logic to
zulip_tools.py so it can be imported and used in upgrade-zulip.
2018-12-16 07:52:47 -08:00
rht
d54fb5f40d provision: Add venv dependencies for Centos 7. 2018-12-14 04:59:46 +00:00
Sumanth V Rao
76c6cf8c3a upgrade-zulip-stage-2: Added argument to skip purging old deployments.
This makes it possible to add --skip-purge-old-deployments in the
deploy_options section of /etc/zulip/zulip.conf, and control whether
old deployments are purged automatically on a system.

We still need to do https://github.com/zulip/zulip/issues/10534 and
probably also to add these arguments to be directly passed into
upgrade-zulip, but that can wait for future work.

Fixes #10946.
2018-12-13 10:10:43 -08:00
Anders Kaseorg
149132348f lint: Update ShellCheck to 0.6.0.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-12 08:24:01 -08:00
rht
e0ec288928 parse_lsb_release: Add CentOS support. 2018-12-11 13:08:26 -08:00
rht
c9d54f7854 puppet: Remove vendored puppetlabs apt and stdlibs dependencies.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.

Fixes #7423.
2018-12-11 13:03:26 -08:00
rht
711b0fda8b scripts: Initialize setup-apt-repo-debathena. 2018-12-11 13:01:26 -08:00
Tim Abbott
0d48f47eba scripts: Make setup-apt-repo a no-op if nothing has changed.
This performance optimization is important for being able to run this
from puppet in our production environment.
2018-12-10 17:36:28 -08:00
Tim Abbott
9423f8e31c node cache: Don't shell out to yarn to get the version.
This can easily be done by just reading the package.json file and
extracting the version value, just like we now do in install-node.
2018-12-07 14:44:52 -08:00
Tim Abbott
630968b632 provision: Use a more efficient approach for getting yarn version.
Since yarn has a package.json conveniently available, we can parse
that with jq, saving the expensive operation of starting up yarn.

This saves ~300ms in a no-op provision.
2018-12-07 14:36:54 -08:00
rht
a1ca8c262a setup-apt-repo: Add set -e and ensure the sources file exists. 2018-12-07 14:00:59 -08:00
Tim Abbott
b3a4b78c6d puppet: Pass zulip scripts path as a FACTER to puppet.
This makes it possible for the Puppet codebase to access the path to
the relevant /home/zulip/deployments type directory that puppet was
run from, which in turn makes it possible to safely call scripts from
here.

Based on work by Rein Zustand.
2018-12-06 11:38:55 -08:00
Tim Abbott
e13de3e629 scripts: Fix incorrect garbage-collection of emoji/node caches.
Apparently, we were incorrectly expressing the paths in the
caches_in_use data structures for these two cache-cleaning algorithms,
resulting in the default threshhold_days algorithm controlling which
caches could be garbage-collected.  While the emoji one was just a
performance optimization for upgrade-zulip-from-git, it was possible
for the main `node_modules` cache in use in production to be GCed,
resulting in LaTeX rendering being broken.
2018-12-03 11:59:08 -08:00
Tim Abbott
1303f8df20 docs: Fix missing quotes in su zulip -c documentation.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
2018-11-30 12:12:27 -08:00
Tim Abbott
31556e45d8 setup-apt-repo: Install gnupg as part of installation.
Apparently, on Debian stretch, the gnupg package isn't installed by
default, which means that our `apt-key add` commands were failing with
these errors on an ultra-minimal Debian installation:

+ apt-key add ./scripts/setup/packagecloud.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
+ apt-key add ./scripts/setup/pgroonga-debian.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

Fixes #10480.
2018-11-30 10:45:04 -08:00
Anders Kaseorg
ed0292629b zulip_tools.run: Remove shell=True support.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg
d581ad82ca scripts/upgrade-zulip-from-git: Avoid shelling out for mkdir, ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg
a61eebd9e3 scripts/lib/unpack-zulip: Avoid shelling out for rm, mv, mkdir, ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg
9bad4003c6 scripts/lib/upgrade-zulip-stage-2: Avoid shelling out for ln.
The original code was actually broken, in that it checked the wrong
path, but it didn't matter because it used `ln -nsf`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg
a694c3cafd scripts/restart-server: Avoid shelling out for ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:26:54 -08:00
Anders Kaseorg
d60b697cc5 scripts/lib/create-production-venv: Avoid shelling out for ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:25:01 -08:00
Anders Kaseorg
33a4d12101 scripts: Add zulip_tools.overwrite_symlink function to replace ln -nsf.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:24:59 -08:00
Anders Kaseorg
1597511de0 install: Check whether universe repository is enabled on Ubuntu.
Fixes #10417.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:18:43 -08:00
Tim Abbott
5a56925495 restart-server: Fix restarting server with multiple tornado processes.
Previously, we unconditionally tried to restart the Tornado process
name corresponding to the historically always-true case of a single
Tornado process.  This resulted in Tornado not being automatically
restarted on a production deployment on servers with more than one
Tornado process configured.
2018-11-27 17:20:05 -08:00
Tim Abbott
adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00
Tim Abbott
3e3eb2aa7f scripts: Clarify names of running-as-root assertions.
This should make it more obvious that these functions will exit the
script if the check fails.
2018-11-19 10:58:34 -08:00
Rohitt Vashishtha
eae9251cb7 scripts: Make upgrade-zulip-* use root checking from zulip_tools.
This is mostly just a nice code deduplication/cleanup.
2018-11-19 10:58:34 -08:00
Rohitt Vashishtha
0975bbb39e scripts: Make zulip-puppet-apply check if the user is root.
Fixes #10833.
2018-11-19 10:58:34 -08:00
Rohitt Vashishtha
767acfa2ac scripts: Add util functions for checking root to zulip_tools. 2018-11-19 10:58:16 -08:00
Rohitt Vashishtha
95ba947f13 setup-cerbot: Allow issuing certificates for multiple domains.
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:

https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/

Fixes #10674.
2018-11-13 12:47:31 -08:00
Tim Abbott
453c35d0f7 install: Provide a suggestive error message when missing Universe.
By far the dominant cause of errors when installing apt packages is
not having the Universe repository enabled in Ubuntu bionic (this
seems to have started happening a lot recently; I wonder if Ubuntu
changed the defaults for new server installs or something?).

In any case, providing that suggestion in the error output should help
reduce these a lot.
2018-11-12 10:56:39 -08:00
Tim Abbott
e7bb833a37 install: Improve some error output for common errors.
This uses `set +x` to hide the `echo` output, and then sets the font
color to red.
2018-11-12 10:55:06 -08:00
Tim Abbott
60ed41081b scripts: Write to upgrade.log in upgrade-zulip-from-git.
This should simplify debugging when doing Git-based upgrades; this log
has long been useful for the main upgrade-zulip tool.
2018-11-06 17:56:01 -08:00
Tim Abbott
3f03dcdf5e nagios: Support multiple tornado processes.
This allows our Tornado monitoring to correctly report whether
multiple configured Tornado processes are running.

This setup isn't ideal, in that it can't detect cases where the wrong
set of Tornado processes are running, but it's nice and simple and
should catch most actual problems.
2018-11-06 16:50:03 -08:00
Tim Abbott
a4f5d825b7 Revert "upgrade-zulip-from-git: Support specifying tag or commit ID for refname."
This reverts commit 5ea7feee7e.

We had to revert this because it didn't work with non-master branch names.
2018-11-06 12:40:45 -08:00
Tim Abbott
0cac7e1cd3 tornado: Extract functions for Tornado queue names.
This moves all control for what queue to use for which realm in our
Tornado system to just the sharding.py file; no actual sharding is
done yet.
2018-11-02 17:00:10 -07:00
Shubham Padia
5ea7feee7e upgrade-zulip-from-git: Support specifying tag or commit ID for refname.
Fixes #10706.
Issue: Before this commit, the `refname` positional argument to
`upgrade-zulip-from-git` script would run successfully for a branch
name on the given remote, but the script would fail if it was
provided with a tag or commit ID.
Solution: 'git clone -q -b refname LOCAL_GIT_CACHE_DIR deploy_path`
would be split into two commands:
1.) `git clone -q LOCAL_GIT_CACHE_DIR deploy_path`
2.) `git checkout -b deploy_timestamp refname` which makes a new
branch with the same name as the timestamp used in make_deploy_path.
2018-10-23 11:11:08 -07:00
Tim Abbott
adf918cf5a upgrade-zulip-from-git: Fix linter error from too-long line. 2018-10-22 14:11:40 -07:00
Shubham Padia
bdb5f43370 upgrade-zulip-from-git: Support specifying remote on the command line.
Adds an optional argument `--remote-url` to specify the remote URL.
Command line remote URL will be given preference above the one
in /etc/zulip/zulip.conf.

Fixes #6092.
2018-10-22 12:01:05 -07:00
Shubham Padia
094e50753e upgrade-zulip-from-git: Rename git_url variable to remote_url. 2018-10-22 12:00:40 -07:00