Commit Graph

30 Commits

Author SHA1 Message Date
Steve Howell
78f6e3ac7d hipchat import: Fix data issues with PMs.
We now set the is_private flag on UserMessage
rows for PMs and set their subject to ''.
2018-10-25 09:11:36 -05:00
Steve Howell
6e8ae2e3fd hipchat import: Support private stream subscribers.
We now create private stream subscriptions that are
based off of `members` and `owner` from room data
in `rooms.json`.
2018-10-25 08:31:01 -05:00
Steve Howell
25f532ca2f refactor: Break up build_subscriptions.
Having two smaller functions should make it
easier to customize the behavior for each specific
use case.  The only reason they were ever coupled
was to keep ids in sequence, but the recent NEXT_ID
changes make that a non-issue now.
2018-10-25 08:31:01 -05:00
Steve Howell
2ed9fbd25b conversions: Use NEXT_ID for recipient and subscription ids.
The NEXT_ID scheme seems pretty robust, so I'm fixing a
few easy places.
2018-10-25 08:31:01 -05:00
Steve Howell
481488a35e Extract make_subscriber_map().
We extract this function and put it in the shared
library `import_util.py`.

Also, we make it one time higher up in the call
stack, rather than re-building it for every batch
of messages.  I doubt this was super expensive, but
there's no reason to repeatedly execute this.
2018-10-23 17:27:37 -05:00
Steve Howell
d1ff903534 refactor: Rename build_user -> build_user_profile.
This makes greps less confusing.
2018-10-23 17:27:37 -05:00
Steve Howell
ff61c56f47 hipchat import: Add NotificationMessage support. 2018-10-17 12:11:08 -07:00
Tim Abbott
f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Steve Howell
8accc60ca7 import_util: Support multiple message ids for attachments. 2018-10-13 16:47:44 -07:00
Steve Howell
23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Steve Howell
4b82326376 hipchat import: Support guest users.
We simplify the code for is_realm_admin
and set is_guest as well.

I verified that build_user() is not used
by Slack/Gitter, so the extra argument there
should be fine.

Fixes #10639
2018-10-11 15:28:58 -07:00
Steve Howell
4da664817b hipchat conversion: Add messages. 2018-10-02 16:55:16 -07:00
Steve Howell
f296d60dad hipchat conversion: Add emoji support. 2018-10-02 16:55:16 -07:00
Steve Howell
9518b1344a hipchat conversion: Process avatars.
This processes the avatar payloads that we
get in users.json.
2018-10-02 16:55:16 -07:00
Steve Howell
c0f15c3860 hipchat conversion: Include deactivated users/streams.
We now include deleted/deactivated data from the old system.
2018-10-02 16:55:16 -07:00
Steve Howell
faea26783b Create convert_hipchat_data.
This is a very early version of a tool to convert Hipchat
tar files into data files that can be used by the Zulip
import process.

We include the most fundamental entities--users and
streams.  Customers who don't care about past messages
or customizations could start an instance off of this
and start communicating.

Of course, there are a lot of things missing in the
initial version:

    * messages!
    * file assets -- avatars, emojis, attachments
    * probably lots of other minor things

We currently ignore any incoming dates from Hipchat data
and just use the current time.  This is consistent with
other imports.

We also don't have any docs yet, although the process
will be extremely similar to the "Slack" process:

    https://zulipchat.com/help/import-from-slack

Also, there's a comment at the top of convert_hipchat_data.py
that describes how to test this in dev mode.

I tested this by following the steps in the comment above.
The users just "show up" in /devlogin, so that's nice, and
you can send messages to other users.  To verify the stream
data you have to go into the gear menu and click on "All
Streams", then you can subscribe and send a message.

Production users will need to get new passwords and
re-subscribe to streams.  We will probably auto-subscribe
all users to public streams.
2018-10-02 16:55:16 -07:00
Rhea Parekh
7822ef38c2 import: Change absolute path of downloaded avatars in records.json to relative path. 2018-09-09 09:18:18 -04:00
Rhea Parekh
f70b9a3eba import: Move 'build_message' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
a5bc701181 import: Move 'build_stream' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
66d34b23ef import: Build Attachment with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh
9617b1fbc5 import: Build Recipient and Subscription with model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh
c77763bd8e import: Move 'build_realm' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh
18a4904437 import: Move 'build_attachment' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
b6ccc0bc52 import: Move 'build_defaultstream' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
bee3964f14 import: Move 'build_usermessages' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
eefe7cccd2 import: Move 'process_uploads' and 'process_emojis' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh
30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh
87cc1a6280 import: Move 'build_subscription' and 'build_recipient' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh
a516f80646 import: Move 'build_avatar' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh
1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00