Commit Graph

743 Commits

Author SHA1 Message Date
Mayank Madan
a86eccb2b6 pgroonga: Upgrade from 2.1.6 to 2.1.8.
This is necessary to support postgres 11 in Debian Buster.
2019-05-05 17:52:32 -07:00
Anders Kaseorg
9937734e50 terminate-psql-sessions: Only terminate if we have permissions.
We have been semi-accidentally relying on the fact that terminate-psql-sessions 
fails silently when there are PIDs we don't have permission to terminate.

This actually happens somewhat often, generally when we're doing a series of
operations in quick succession by different users, because postgres processes
live a little longer than the `psql` shell that started them.

As part of adding ON_STOP_ERROR to all of our postgres commands, it makes
sense to enforce we don't fail here, but that means we need to actually filter
the target PIDs to only ones we can actually kill.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 14:41:42 -07:00
Tim Abbott
74143a0801 scripts: Disable terminate-psql-sessions ON_ERROR_STOP for now.
This appears to fail nondeterministically in CI; there's likely a
better fix but this should let us get CI happy again.
2019-04-22 15:49:48 -07:00
Anders Kaseorg
5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Tim Abbott
18146ed947 script: Add static asset pipeline option to support OS upgrades.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
2019-04-17 12:58:42 -07:00
Anders Kaseorg
ed100d694f restore-backup: Open backup tarball as root.
Fixes permission errors when running restore-backup on a tarball
inaccessible to the zulip user.

Fixes #12125.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg
206741117c restore-backup: Reset cwd when switching to postgres user.
Fixes permission errors when running restore-backup from a cwd
inaccessible to the postgres user.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg
9fc0aa7ffa scripts: Fix exec invocation for in-process virtualenv activation.
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.

As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:

    >>> exec('a = 1; print([a])', {}, {})
    [1]
    >>> exec('a = 1; print([a for b in [1]])', {}, {})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
      File "<string>", line 1, in <listcomp>
    NameError: name 'a' is not defined
    >>> exec('a = 1; print([a for b in [1]])', {})
    [1]

Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.

Fixes #12030.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-16 17:42:44 -07:00
Wyatt Hoodes
d67dc2eb94 install-node: Upgrade yarn to 1.15.2.
In addition to upgrading dependencies being generally useful, this may
fix situations where yarn fails but returns a success status code in the 
presence of an HTTP proxy.
2019-04-14 22:36:14 -07:00
Tim Abbott
ad81f700a1 scripts: Remove nagios overrides for missedmessage_emails.
Since 5cec566cb9, the
missedmessage_emails queue no longer is expected to grow a backlog
over time.
2019-04-13 20:43:07 -07:00
Puneeth Chaganti
9876f1b14e check_rabbitmq_queue: Fix the time period when we ignore long queues.
The commit 87d1809657 changed the time when
digests are sent by 3 hours to account for moving from the US East Coast to the
West Coast, but didn't change the time period exception in the
`check-rabbitmq-queue` script.

Closes #5415
2019-04-13 20:43:07 -07:00
Anders Kaseorg
460abe82df terminate-psql-sessions: Allow running as postgres user.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg
53879c4673 scripts: Rename DEFAULT_USER to POSTGRES_USER.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg
d03b8308a5 backup: Use tar --transform to arrange the tarball instead of symlinks.
This allows tar to print the real paths in error messages if something
goes wrong.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-04-12 16:48:34 -07:00
Rohitt Vashishtha
22a3b4230c Revert "install: Ensure that sudo is installed."
Now that we have the run_as_root helper function, we don't need to
install sudo to run Zulip in production

This reverts commit a7d7d181ea.

Fixes #10036.
2019-04-04 16:52:20 -07:00
Rohitt Vashishtha
327b70cb19 terminate-psql-sessions: Allow running without sudo. 2019-04-04 16:52:20 -07:00
Tim Abbott
042893a448 configure-rabbitmq: Allow running as root.
This makes it possible to run this in production without access to
sudo.
2019-04-04 16:52:20 -07:00
Anders Kaseorg
f78970c679 restore-backup: Set the recreated database owner to zulip.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-04 11:50:13 -07:00
Mayank Madan
264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
Tim Abbott
0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott
5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00
Tim Abbott
8469c7774f scripts: Remove create-zulip-admin.
This was used only by ancient versions of the Docker project.
2019-03-04 15:33:51 -08:00
Tim Abbott
0e7a5d82b4 setup_venv: Remove virtualenv-clone legacy upgrade code.
Few folks will be upgrading from versions of Zulip old enough to not
have virtualenv-clone, and those who are won't be able to use it due
to older dependencies having been removed.
2019-03-04 15:32:15 -08:00
Tim Abbott
4157d15ff1 setup_venv: Fix check for whether virtualenv-clone is unavailable.
We don't need a bare except here.
2019-03-04 15:32:15 -08:00
Tim Abbott
478e215acd setup_venv: Use run_as_root helper.
Also, don't use split() to break up strings to pass into run().
2019-03-04 15:32:15 -08:00
Anders Kaseorg
d9adc9d7bc get_dev_uuid_var_path: Fix theoretical shell quoting problem.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-03-04 15:20:07 -08:00
Tim Abbott
b3444354aa su_to_zulip: Fix detection of zulip user ID.
Apparently, while upgrade-zulip-from-git always ensures that zulip
deployment directories are owned by the Zulip user, unpack-zulip (aka
the tarball code path) has them owned by root.

The user ID detection logic in su_to_zulip's helper get_zulip_uid was
intended to support both development environments (where the user ID
might vary) and production environments.  For development
environments, the existing code is fine, but given this unpack-zulip
permissions issue, we need to have code to fallback to 'zulip' if the
detection logic detects the "zulip" user has having UID 0.
2019-03-04 14:27:39 -08:00
Rohitt Vashishtha
3d427d02cc scripts/zulip_tools: Use run_as_root instead of subprocess.check_call. 2019-03-01 11:21:16 -08:00
Rohitt Vashishtha
ac48925977 scripts: Use run_as_root instead of run([sudo, ...]). 2019-03-01 11:21:16 -08:00
Anders Kaseorg
1e2bd553fb setup-certbot: Remove --force-renewal. (#11652)
There’s no reason to do this unless you’re, like, trying to trip the
Let’s Encrypt rate limits (or perhaps trying to manually test this code).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 15:50:38 -08:00
Tim Abbott
57e1307a3a provision: Fix virtualenv-clone handling of success-stamp.
Apparently, virtualenv-clone ends up copying the success-stamp file
that we use to track whether a virtualenv was successfully
provisioned, which results in problems if we get a network error in
the pip install stage afterwards.

The comment explains our fix, but basically we just delete
success-stamp after the clone.

Fixes #11301.
2019-02-16 11:24:10 -08:00
Anders Kaseorg
e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Anders Kaseorg
ebad0b7cbf zulip_tools: Get the zulip uid from the owner of DEPLOY_ROOT.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-11 17:00:37 -08:00
Anders Kaseorg
e4832cc9da setup_path_on_import: Don’t re-add BASE_DIR to sys.path.
BASE_DIR must already be in sys.path if we got as far as importing
this.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:17:31 -08:00
Anders Kaseorg
70bfcd3402 zulip_tools: Extract get_deploy_root function.
Modified by tabbott from the original to preserve the implementation;
see https://github.com/zulip/zulip/pull/11295#discussion_r254925032
for why this is correct.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:09:29 -08:00
Anders Kaseorg
e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Anders Kaseorg
4fef10f4e6 install: Improve error handling.
On usage errors (except --help), write usage message to stderr and
exit with nonzero status.

Forbid setting the hostname and email to the example values.  Those
are specifically checked for and would fail later.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 11:18:42 -08:00
Anders Kaseorg
893e1475bb setup_virtualenv: Do not activate the virtualenv on creation.
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py).  This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-31 17:55:04 -08:00
Tim Abbott
df436a55b4 scripts: Recommend apt update after enabling universe.
One needs to manually do an apt update after add-apt-repository, or it
won't actually work.
2019-01-26 12:29:05 -08:00
Anders Kaseorg
5ba84d75ce create-zulip-admin: Fix shell quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
Anders Kaseorg
981f09d950 terminate-psql-sessions: Fix shell and SQL quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
Harshit Bansal
50ef91bb08 scripts: Add argparse option to restart-zerver for --fill-cache.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Anders Kaseorg
b9f7b08af2 setup_path_on_import: Check that the virtualenv is compatible.
The site_packages variable points to (e.g.)
zulip-py3-venv/lib/python3.4/site-packages.  If that doesn’t exist,
we’re probably running the wrong Python version.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:09 -08:00
Anders Kaseorg
7a58f574bf setup_path_on_import: Skip activate_this if already activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:07 -08:00
Anders Kaseorg
2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
rht
91348a6aee create-thumbor-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht
68c20c610e create-production-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht
15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
rht
d8a081b94c setup-yum-repo: Add prod flag for doing IUS install on CentOS. 2019-01-05 15:49:02 -08:00
rht
a55f5b59f6 generate-self-signed-cert: Generalize to CentOS, Fedora, RHEL. 2019-01-05 15:49:02 -08:00