Commit Graph

21 Commits

Author SHA1 Message Date
Anders Kaseorg
643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Vishnu Ks
6e3720e0b7 gitter: Fix minor comment typo in build_userprofile. 2019-03-20 10:12:18 -07:00
Anders Kaseorg
56a675d5ec export: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:25:27 -08:00
Tim Abbott
e9900b2bdf gitter: Do something reasonable with invalid fullnames. 2018-12-12 10:07:52 -08:00
Steve Howell
d86dd165da gitter/slack/hipchat: Remove "subject" from conversions.
We (lexically) remove "subject" from the conversion code.  The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.

There was good code coverage on `build_message`.
2018-11-12 15:47:11 -08:00
Steve Howell
5cb60f7bea conversions: Use subscriber_map for Slack/Gitter.
We now use subscriber_map for building UserMessage
rows in Slack/Gitter conversions.

This is mostly designed to simplify the code, rather
than having to scan the entire subscribers for each
message.

I am guessing this will improve performance for most
conversions.  We sort small lists on every message,
in order to be deterministic, but the sorting cost
is probably more than offset by avoiding the O(N)
scans across all subscriptions.  Also, it's probably
negligible in the grand scheme of things, compared
to JSON parsing, file I/O, etc.

This commits also fixes some typos with mentioned_users_id ->
mentioned_user_ids and cleans up a test a bit as well.
2018-10-29 13:24:50 -07:00
Steve Howell
5194701787 conversions: Use NEXT_ID for usermessage_id.
This is mostly complicated due to the way that the
Slack import passes around tuples of ids to maintain
four different parallel sequences.
2018-10-29 13:24:50 -07:00
Tim Abbott
f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Steve Howell
23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Rhea Parekh
f70b9a3eba import: Move 'build_message' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
a5bc701181 import: Move 'build_stream' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
c4f8abbd30 import: Build Message with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh
4ea7302e14 import: Add missing fields in UserProfile object.
The missing fields are checked by `full_clean()` method.
The datetime field errors are ignored as they are fixed
in the `import_realm` script. The field that are
allowed to be null are not included while building
this object.
2018-08-19 22:27:13 -07:00
Rhea Parekh
c77763bd8e import: Move 'build_realm' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
b6ccc0bc52 import: Move 'build_defaultstream' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
bee3964f14 import: Move 'build_usermessages' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh
87cc1a6280 import: Move 'build_subscription' and 'build_recipient' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh
a516f80646 import: Move 'build_avatar' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh
1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh
ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00