Commit Graph

43 Commits

Author SHA1 Message Date
Tim Abbott
332add3bb6 import: Fix propagation of subdomain error messages.
The previous logic would provide a very confusing error message if the
subdomain was already in use.
2021-06-09 13:22:23 -07:00
Anders Kaseorg
6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg
11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg
5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Tushar912
410bb8ad89 imports: Add better checking for subdomains.
Add a `--allow-reserved-subdomain` flag which allows creation of
reserved keyword domains.  This also always enforces that the domain
is not in use, which was removed in 0258d7d.

Fixes #16924.
2021-01-12 17:54:01 -08:00
Anders Kaseorg
a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg
fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg
b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg
a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Mateusz Mandera
60c4896c3a import: Fix import and export parallelism defaults.
Fixes #14960.
The default of 6 thread may not be appropriate in certain
configurations. Taking half of the numer of CPUs available to the
process will be more flexible.
2020-06-21 09:39:31 -07:00
Anders Kaseorg
365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg
67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Tim Abbott
8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Anders Kaseorg
becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Vishnu Ks
123bcea518 management: Don't use sys.exit(1).
Using sys.exit in a management command makes it impossible
to unit test the code in question.  The correct approach to do the same
thing in Django management commands is to raise CommandError.

Followup of b570c0dafa
2019-05-03 14:20:39 -07:00
Anders Kaseorg
2a2e744b27 management: Remove unused imports in management commands.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:12:36 -08:00
Vishnu Ks
bec875a9af import realm: Use processes for resizing avatar images.
This should significantly improve the data import performance when
importing large open source realms from Slack.

Fixes #11009.
2019-01-25 12:37:12 -08:00
Tim Abbott
d66967f851 import: Use os.path.expanduser when accessing paths.
This fixes an issue where passing a path like `~/exports/foo` would
result in a `~` directory being created and the export/import not
working correctly.
2018-08-13 13:32:53 -07:00
Rhea Parekh
66e3fc4042 import: Clean up the management command.
Significantly tweaked by tabbott because:
* Argparse was already handling the early checks
* Splitting the bottom loop into two loops means we validate all the
input before trying to run actual import code on anything.
* The argparse documentation was confusing about whether the paths
  should be files or directories.
2018-05-28 11:54:04 -07:00
Tim Abbott
c4b886d8ae import: Split out import.py into its own module.
This should make it a bit easier to find the code.
2018-04-23 15:21:12 -07:00
Preston Hansen
0258d7db0d slack import: Be less strict in check_subdomain_available.
If the sysadmin is doing something explicit in a management command,
it's OK to take a reserved or short subdomain.

Fixes #9166.
2018-04-23 11:48:12 -07:00
Tim Abbott
7e91e66987 import: Fix ordering of subdomain availability check.
When you're importing with --destroy-rebuild-database, we need to
check subdomain availability after we've cleared out the database;
otherwise, trying to reuse the same subdomain doesn't work.
2018-04-20 10:39:30 -07:00
Rhea Parekh
ccefaf7b26 scripts: Remove the depreciated script 'postgres-reset-sequences'. 2018-04-10 13:07:14 -07:00
Rhea Parekh
2baa9bc16e Import: Add subdomain in the import script.
Also remove user input of subdomain in the slack data
conversion script.
2018-04-06 09:12:56 -07:00
Rhea Parekh
ed4054d110 Import script: Check and add system bots after every import.
This checks for the existing system bots and adds them if they
aren't included in the import.
2017-12-27 07:52:45 -05:00
Rhea Parekh
460e9b4634 Scripts: Add script for resetting postgres sequences for imports. 2017-12-01 06:38:08 -08:00
Tim Abbott
f04d6ed19e python: Sort imports in management commands. 2017-11-15 15:43:47 -08:00
rht
a93b7c40a7 zerver: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
rht
a311678190 zerver/management: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00
Tim Abbott
ca59b28fc1 import: Fix positional argument parsing. 2017-10-06 08:45:22 -07:00
Tim Abbott
bfedb82097 import: Fix argparse usage message breakage. 2017-10-06 08:41:01 -07:00
rht
32650c8fdc zerver/management: Remove print_function. 2017-09-27 18:05:45 -07:00
rht
e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Vishnu Ks
05af206392 management: Remove unused imports from import command. 2017-08-08 14:13:19 -07:00
Aditya Bansal
7271fc8a7b pep8: Add compliance with rule E261 to import.py. 2017-05-07 23:21:50 -07:00
Harshit Bansal
ac2172e233 models: Rename RealmAlias model to RealmDomain.
Includes a migration.
2017-04-04 15:48:03 -07:00
Tim Abbott
4d3e97f304 actions: Remove unnecessary do_create_stream function. 2017-01-29 22:07:14 -08:00
reallyly
94b28e0afb pep8: Fix E128 violations.
With some line-wrapping tweaks by tabbott.
2016-12-13 10:50:14 -08:00
bulat22101
adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
Anders Kaseorg
207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Umair Khan
682aa1f298 Django 1.10: Use add_argument for options in BaseCommand. 2016-11-04 10:20:23 -07:00
Steve Howell
85e30a6d35 Annotate commands/import.py. 2016-09-11 17:15:35 -07:00
Tim Abbott
6264ff7039 Add new Zulip realm import/export tool.
The previous export tool would only work properly for small realms,
and was missing a number of important features:
* Export of avatars and uploads from S3
* Export of presence data, activity data, etc.
* Faithful export/import of timestamps
* Parallel export of messages
* Not OOM killing for large realms

The new tool runs as a pair of documented management commands, and
solves all of those problems.

Also we add a new management command for exporting the data of an
individual user.
2016-08-08 14:58:18 -07:00