Commit Graph

24 Commits

Author SHA1 Message Date
Tim Abbott
8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Anders Kaseorg
becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Vishnu Ks
123bcea518 management: Don't use sys.exit(1).
Using sys.exit in a management command makes it impossible
to unit test the code in question.  The correct approach to do the same
thing in Django management commands is to raise CommandError.

Followup of b570c0dafa
2019-05-03 14:20:39 -07:00
Tim Abbott
f04d6ed19e python: Sort imports in management commands. 2017-11-15 15:43:47 -08:00
rht
a311678190 zerver/management: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00
derAnfaenger
d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
rht
e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Vishnu Ks
1b0b135bfc management: Remove unused imports from process_queue. 2017-08-08 14:13:19 -07:00
Umair Khan
0e8231d0f1 process_queue: Recover gracefully after PostgreSQL restart.
- For threaded workers:
Django's autoreloader catches SIGQUIT(3) to reload the program. If
a process being watched by autoreloader exits with status code 3,
reloader will restart the process. To reload, we send SIGUSR1(10)
signal from consumers to a handler in process_queue which then
exits with status code 3.

- For single worker per process:
Catch the SIGUSR1 and quit; supervisorctl will restart the worker
automatically.

Fixes #5512
2017-07-07 16:33:15 -07:00
rht
940cf9db3b Run queue processors multithreaded in production if system memory <3.5GB.
While running queue processors multithreaded will limit the
performance available to very small systems, it's easy to fix that by
adding more RAM, and previously, Zulip didn't work on such systems at
all, so this is unambiguously an improvement there.

Fixes #32.
Fixes #34.

(Commit message expanded significantly by tabbott.)
2017-06-03 12:19:58 -07:00
Rafid Aslam
41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Steve Howell
ca43cbc654 logging: Reducing logging in run-dev.py for queue workers.
In dev, we no longer log that individual queue workers were launched.

Instead, in dev (and prod as well), we log a message with the total
count.
2016-11-17 11:12:02 -08:00
Tim Abbott
86e933a4a1 process_queue: Suppress USING_RABBITMQ warnings in test suite. 2016-10-27 12:36:06 -07:00
Tim Abbott
1e54897ca7 process_queue: Add missing type annotation. 2016-08-04 15:57:03 -07:00
Tim Abbott
92062b9526 process_queue: Add missing annotation. 2016-08-04 15:57:02 -07:00
rahuldeve
a3745178e5 Use django.utils.autoreload to restart queue workers at code change.
Fixes #621, #1045.
2016-06-26 20:12:11 -07:00
Tim Abbott
a1a27b1789 Annotate most Zulip management commands. 2016-06-04 10:12:06 -07:00
Tim Abbott
06b33da709 process_queue: Fix missing worker.setup() in single-threaded codepath. 2016-03-27 23:17:16 -07:00
Tim Abbott
cd2348e9ae Run queue processers multithreaded in development.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.

On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.".  Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.

Progress towards #32.
2016-03-20 18:04:24 -07:00
Tim Abbott
7595e4b05f process_queue: Fix worker variable being accessed before initialization. 2016-02-03 19:29:44 -08:00
Reid Barton
ae0ae3dde8 Django 1.8: declare positional arguments in management commands
(imported from commit d9efca1376de92c8187d25f546c79fece8d2d8c6)
2015-08-20 23:35:40 -07:00
Zev Benjamin
ffb6266319 Refuse to run a queue processor if USING_RABBITMQ is False
(imported from commit 39beff47cdbb18ba39756989e6f07facbd16864f)
2013-10-28 14:30:53 -04:00
Zev Benjamin
871afde142 Prepare process_queue to be used with supervisor's numprocs
(imported from commit 5a652b93f1e8b32b5ed89d622035161abaedfb11)
2013-09-24 20:44:15 -04:00
Zev Benjamin
6cf5ebc63d Add new system for defining and running queue-processing workers
(imported from commit 7e77da57b8ad0b70837785c85f546601ba5b1957)
2013-09-24 20:44:15 -04:00