Commit Graph

40113 Commits

Author SHA1 Message Date
Steve Howell
1d1e45e9ec digests: Use UserActivityInterval for user activity.
Note that we are much more efficient about finding
active users here:

    - we do one query per realm (instead of per-user)
    - we pass the cutoff date to the database
    - we get back just a list of distinct ids
2020-11-16 08:59:29 -08:00
Steve Howell
b52f56080e performance: Just get user_ids to queue digest emails. 2020-11-16 08:59:29 -08:00
Steve Howell
e13e5d104d refactor: Only require user_id for inactive_since().
This function is going away completely soon.  It is
querying everybody's entire UserActivity history instead
of passing the cutoff date to the database!
2020-11-16 08:59:29 -08:00
Steve Howell
d0260392f7 digests: Get user objects from the database.
The query counts increase here for somewhat
contrived reasons.  The tests before this
commit reflected a successful trip to the
UserProfile cache, but that's not actually
realistic in practice.
2020-11-16 08:59:29 -08:00
Steve Howell
7737413cec digest tests: Improve gather_new_streams test.
We don't need to mock the dates here.  We also
explicitly clear out all streams first, and then
we explicitly test with both the stream being
current and the stream being old.
2020-11-16 08:59:28 -08:00
Steve Howell
9538edde06 digest tests: Simplify bots test.
We can use the _enqueue_emails_for_realm helper
to avoid all the Tuesday-related logic here.

We also don't bother to create UserActivity
records, since the bot gets excluded by virtue
of its being a bot.  (Also, the date ranges
here were sketchy due to the time mocking.)
2020-11-16 08:59:28 -08:00
Steve Howell
0624833af6 digest tests: Improve Tuesday tests.
If we're mocking time, we should do it consistently.
2020-11-16 08:59:28 -08:00
Steve Howell
2f4d7a6171 tests: Fix test_inactive_users_queued_for_digest.
We can avoid all the date mocking now for all
but a couple tests that exercise the is-it-Tuesday
logic.

And this test now correctly tests that we exclude
recently active users.

And this allows us to remove the other test.
2020-11-16 08:59:28 -08:00
Steve Howell
e49a482baf email digests: Make transactions atomic. 2020-11-16 08:59:28 -08:00
Steve Howell
cf6bcfb84a digest emails: Exclude users who had recent digests.
This code protects us in case we ever need to re-run
email digests twice in the same day.
2020-11-16 08:59:28 -08:00
Steve Howell
fb3d4c1618 digest tests: Avoid warnings about naive time. 2020-11-16 08:59:28 -08:00
Steve Howell
4271442fba email digests: Write RealmAuditLog rows. 2020-11-16 08:59:28 -08:00
Alex Vandiver
90ca06d873 puppet: Allow unattended upgrades of -updates in addition to -security.
This ensures that software will be fully up-to-date, not just with
security patches.
2020-11-13 16:45:05 -08:00
Mateusz Mandera
4f47f35cb4 auth: Handle the case of invalid subdomain at /fetch_api_key endpoint. 2020-11-13 16:43:17 -08:00
Alex Vandiver
2e20ab1658 puppet: Log the "Host" header and total response time.
Logging `Host` is useful for determining access patterns to realms,
especially if ROOT_DOMAIN_LANDING_PAGE is set.  Total response time is
useful in debugging access and performance patterns.
2020-11-13 16:42:32 -08:00
Anders Kaseorg
1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg
29050918a6 mypy: Exclude broken typeshed stubs for markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg
254758733e mypy: Remove unused yamole stub.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg
8ba95063d5 test_markdown: Construct FencedBlockPreprocessor with a real Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg
e7e1fde6ec fenced_code: Use immutable type for codehilite_conf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg
fbf8ce0305 markdown: Add types for extra Markdown members.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
b48bdc65b9 markdown: Fix AlertWordNotificationProcessor.run type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
9573f6dc00 markdown: Fix build_block_parser type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
4398eecd2b markdown: Use immutable type for extension config.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
060036dfd5 markdown: Merge build_engine into Markdown constructor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
08c64f5cfa markdown: Fix imports for compatibility with typeshed stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
f294688157 test_stripe: Add CallableT cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg
2a8a59f548 test_queue_worker: Simplify worker_queue_names computation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg
dc84e9696c mypy: Fix types for redis.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg
bcb571c187 test-backend: Fix argparse error type.
ArgumentError takes two arguments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 14:17:22 -08:00
Aman Agrawal
9eb82f31c9 wsl_dev_setup: Warn user about cloning inside windows mounted disk.
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
2020-11-10 00:21:41 -08:00
Nikhil
9c3ff29d35 docs: Added number of selected participants in GSoC 2020. 2020-11-09 20:26:56 -08:00
Anders Kaseorg
3a8cf869db python: Convert os.open(…, O_EXCL) to open(…, "x").
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-09 14:31:01 -08:00
Puneeth Chaganti
358f1f9ba7 webhooks/sentry: Support integration configured as webhook.
Sentry allows adding simple webhooks without going through the process
of creating an Internal Integration in Sentry's Integration
Platform[1] (which our docs recommend).

The payload from sent from such a (simple) webhook integration is
slightly different from the payload sent by an Internal Integration
webhook. This commit tries to wrangle this payload into a form that is
usable by our webhook handler to send a notification message.

[1]: https://sentry.io/integration-platform/
2020-11-09 12:02:49 -08:00
Mateusz Mandera
47228f3a95 actions: Implement do_delete_user.
To have a reasonable way of creating the dummy user without duplicating
code, we need change create_user to have the optional force_id argument.
2020-11-09 11:58:02 -08:00
YashRE42
d9b14b5b74 widgetize: Don't warn for deleted tictactoe widget. 2020-11-09 06:01:39 -05:00
Tim Abbott
909b484d64 docs: Improve headings for changelog. 2020-11-06 16:50:28 -08:00
akshatdalton
806c1a0b8b markdown: Fix flickering of embedded link inside Italic.
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.

This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.

Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.

Fixes #14845.
2020-11-06 10:09:15 -08:00
Steve Howell
5da4332620 minor: Add order-by-id to digest message query.
The order-by-id is now explicit, and I add
comments to explain the select_related tables.
2020-11-06 10:05:46 -08:00
Steve Howell
936171d258 refactor: Extract DigestTopic class.
This gets us away from a lot of dictionary soup.
2020-11-06 10:05:46 -08:00
Steve Howell
e8b6c56322 refactor: Simplify get_hot_topics().
The code we deleted here was no longer
doing anything.

Maybe the code was always dead, or maybe it
was written during a time when topics_by_diversity
and topics_by_length actually had different keys.

But now it's clearly cruft.

If we have 4 or more topics, then the code above
it would already have populated the list with 4
elements, and the `if num_convos < 4` condition
would evaluate to False.

And if we had 3 or fewer topics, then we would
have already put all possible topics into our
result, and the `topics_by_diversity[num_convos:4]`
slice would be empty.

It's possible that we should just have a simple
heuristic for topic hotness like `10*num_senders
+ messages`, so we don't have to maintain this
fiddly function, and we can just do something like
`topics_by_score[:4]`.
2020-11-06 10:05:46 -08:00
Vishnu KS
bae9dc5f2f billing: Use name selector for overriding the spin button for license input. 2020-11-06 09:59:33 -08:00
Steve Howell
c5dc9d386f refactor: Use sets of stream_ids for email digests.
I now use sets for stream_ids in more of the digest
code.

As part of this I replaced exclude_subscription_modified_streams
with streams_recently_modified_for_user.

It's easier for the caller to just ask for ids
to delete from its callee than it is to pass
in a set/list to mutate.

The simpler boundary between the functions makes
the tests easier to write--you can see the
`filtered_streams` logic goes away in this diff.

I also make the tests a bit more thorough by using
combinations of Cordelia/Othello and Verona/Denmark
to try to find multiple possible flaws.

And I make the time intervals longer than 1s to
avoid false negatives from slow CI boxes.
2020-11-05 17:42:43 -08:00
Suyash Vardhan Mathur
6b48fb3c08 streams: Fix live update of stream name when renamed.
Currently, the Stream Name change isn't reflected in the streams
sidebar when a stream is renamed if the order of streams in the
sidebar remains unchanged, because the optimization to avoid
rerendering when nothing changes about the order prevents the
rerendering code from running.

We fix by this adding a flag in build_stream_list and
update_streams_sidebar functions to force a rerender, and pass that
when a stream is renamed.

Fixes #16026.
2020-11-05 11:45:44 -08:00
Tim Abbott
2d847728a3 message_fetch: Fix node tests for new on-load query.
This adds missing unit test updates from
ebea17b9a6.
2020-11-05 11:03:35 -08:00
Tim Abbott
ebea17b9a6 message_fetch: Add an extra fetch for Recent Topics.
The comment explains the problem statement in some detail, but
basically this algorithm ensures that the top items in "Recent Topics"
on page load are always the very most recent topics the user has
received messages in (well, ignoring muted topics in this iteration).
2020-11-05 10:38:45 -08:00
Vishnu KS
d6743da139 docs test: Verify zulip/zulip file and directory links locally. 2020-11-05 10:07:43 -08:00
Vishnu KS
cba292b4bb lint: Ensure that docs don't link directly to code lines. 2020-11-05 10:07:43 -08:00
Vishnu KS
f2d18010c4 docs: Don't link directly to the python file line from user doc. 2020-11-05 10:07:43 -08:00
Vishnu KS
e94989bc75 docs: Don't link directly to the python file line from integration doc. 2020-11-05 10:07:43 -08:00