This closes an open redirect vulnerability, one case of which was
found by Graham Bleaney and Ibrahim Mohamed using Pysa.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Backported for 2.0.7 security release.
For a long time, we've been only doing the zxcvbn password strength
checks on the browser, which is helpful, but means users could through
hackery (or a bug in the frontend validation code) manage to set a
too-weak password. We fix this by running our password strength
validation on the backend as well, using python-zxcvbn.
In theory, a bug in python-zxcvbn could result in it producing a
different opinion than the frontend version; if so, it'd be a pretty
bad bug in the library, and hopefully we'd hear about it from users,
report upstream, and get it fixed that way. Alternatively, we can
switch to shelling out to node like we do for KaTeX.
Fixes#6880.
Backported for 2.0.7 security release.
A bug in Zulip's new user signup process meant that users who
registered their account using social authentication (e.g. GitHub or
Google SSO) in an organization that also allows password
authentication could have their personal API key stolen by an
unprivileged attacker, allowing nearly full access to the user's
account.
Zulip versions between 1.7.0 and 2.0.6 were affected.
This commit fixes the original bug and also contains a database
migration to fix any users with corrupt `password` fields in the
database as a result of the bug.
Out of an abundance of caution (and to protect the users of any
installations that delay applying this commit), the migration also
resets the API keys of any users where Zulip's logs cannot prove the
user's API key was not previously stolen via this bug. Resetting
those API keys will be inconvenient for users:
* Users of the Zulip mobile and terminal apps whose API keys are reset
will be logged out and need to login again.
* Users using their personal API keys for any other reason will need
to re-fetch their personal API key.
We discovered this bug internally and don't believe it was disclosed
prior to our publishing it through this commit. Because the algorithm
for determining which users might have been affected is very
conservative, many users who were never at risk will have their API
keys reset by this migration.
To avoid this on self-hosted installations that have always used
e.g. LDAP authentication, we skip resetting API keys on installations
that don't have password authentication enabled. System
administrators on installations that used to have email authentication
enabled, but no longer do, should temporarily enable EmailAuthBackend
before applying this migration.
The migration also records which users had their passwords or API keys
reset in the usual RealmAuditLog table.
Backported for 2.0.7 security release.
This includes adding a new endpoint to the push notification bouncer
interface, and code to call it appropriately after resetting a user's
personal API key.
When we add support for a user having multiple API keys, we may need
to add an additional key here to support removing keys associated with
just one client.
This allows the system to get updates to the Groonga repository
signing key, so `apt update` doesn’t start failing when the key
changes (like it recently did).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
As it turns out, our rerender_the_whole_thing function (used whenever
we were adding messages and discovered that the resulting message list
would be out-of-order) was just broken and scrolled the browser to a
random location.
This caused two user-facing bugs:
* On very fast networks, if two users sent messages at very close to
the same time, we could end up with out-of-order message deliveries,
triggering this code path, which was intended to silently correct
the situation, but failed.
* In some narrows to streams with muted topics in the history but some
recent traffic, the user's browser-cached history might have some
gaps that mean the server fetch we do after narrowing discovers the
history is out-of-order, again triggering the
rerender_the_whole_thing code path.
The fix is to just remove that function, adding a new option to the
well-tested rerender_preserving_scrolltop (which has explicit logic to
preserve the scroll position) instead.
Fixes#12067. Likely also fixes#12498.
We had some excessively tight rules about what characters were
allowed, which in particular prevented using `?foo=bar&baz=quux`
structures in the realm filters URLs.
Fixes#12239.
Any regex including a match-everything subpattern (.*, .*?, .+, or
.+?) is almost automatically wrong because it fails to disambiguate
when one subpattern should end and another should begin. Among other
bugs, these kind of regexes tend to be especially prone to denial of
service vulnerabilities through catastrophic backtracking on strings
that fail to match in a large (in this case, exponential) number of
ways.
Lacking a specification to say what characters should actually be
allowed in these subpatterns (this syntax is too different from
CommonMark to be able to precisely apply those rules), I’ve tried to
make reasonable guesses and avoid changing much else.
Because Zulip doesn't store messages until they have successfully been
processed by the Markdown processor, this is not a stored DoS issue.
In general, Zulip protects against the broad category of DoS issues in
Markdown rendering via a timeout managed by another thread. However,
details of Python's regular expression implementation mean that this
particular issue could prevent the timeout thread from being
scheduled, resulting in this being a DoS issue.
This was fixed in master a few months ago as a side effect of
abe2dab88c (#12979).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
* Whitelist a small number of image/ types to be served as
non-attachments.
* Serve the file using the type that we validated rather than relying
on an independent guess to match.
This issue can lead to a stored XSS security vulnerability for older
browsers that don't support Content-Security-Policy.
It primarily affects servers using Zulip's local file uploads backend
for servers running Ubuntu 16.04 Xenial or newer; the legacy local
file upload backend for (now EOL) Ubuntu 14.04 Trusty was not affected
and it has limited impact for the S3 upload backend (which uses an
unprivileged S3 bucket domain to serve files).
This was fixed in master via 780ecb672b.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
One of smtpd_relay_restrictions or smtpd_recipient_restrictions is
required by postfix ≥ 2.10 (see
http://www.postfix.org/SMTPD_ACCESS_README.html).
This is important for using the email mirror on Ubuntu Bionic.
Long-term, we want this flow to do a full re-authentication, but this
makes the frontend consistent with the backend and fixes a confusing
bug where submitting the form ended up adding a weird `?password=`
thing to the URL, in addition to not working.
Fixes#12200.
Previously, if you restored onto a different production system from
the one where you took the backup, backup restoration would fail
because the generated rabbitmq passwords for the two systems would be
different, and we didn't update the restored system to use the
password from the original system.
Fixes#12114.
This should ensure that we apply any special configuration for the
database system (e.g. installing `pgroonga`) before we try to restore
the database contents from the archive.
For pgroonga in particular, this is important so that we can preserve
the configuration of the extension in the `pg_restore` process.
Fixes#12345.
With the S3 file upload backend, we don't store uploads locally, so
the `uploads` directory in the backup will be empty, and more
importantly, LOCAL_UPLOADS_DIR will be None, which the previous code
crashed on.
This is a small patch to fix the error message an admin would receive if
they tried to change bot info and owner from the "bots" setting of the
organization settings panel.
It's not required on Ubuntu Xenial (having been replaced by systemd)
and causes problems when installing/upgrading other packages; this
change matches a similar block of code in our installer.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
Fixes permission errors when running restore-backup on a tarball
inaccessible to the zulip user.
Fixes#12125.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
When passing arguments with the `-d` syntax, which is convenient for
command-line examples, one needs to specify `-X GET` for curl to work
properly.
Fixes#12116
One longstanding gap in our production documentation is how to
properly do an upgrade to the operating system on which Zulip is
installed.
This adds that documentation.
Ideally, we'd get a few folks to test this procedure over the next few
days to make sure it's bulletproof.
Fixes#1705.
Fixes#10796.
Removed the preview tag from the css rule, reduced the undo preview tag
to a font-size of 15px.
The preview tag being attached to the rule proved unnecessary. The icon
for reverting back to an editing state also dipped below the horizontal
level of the icon row.
Apparently, our use of JavaScript string `.replace()` here was buggy,
because replace() has several special escape sequences starting with
`$` if they appear in the replacement content string. We can work
around this through something of a hack, which is to pass a function
as the second argument to replace, which seems cleaner than replacing
all $s with $$s.
Thanks to Shreya for the report.
The construction `su postgres -c -- bash -c 'psql …'` didn’t behave the
way it reads, and only worked by accident:
1. `-c --` sets the command to `--`.
2. `bash` sets the first argument to `bash`.
3. `-c 'psql …'` replaces the command with `psql …`.
Thus, `su` ended up executing `<shell> -c 'psql …' bash`, where
`<shell>` is the `postgres` user’s login shell, usually also `bash`,
which then executed 'psql …' and ignored the extra `bash`.
Unconfuse this construction.
Note from tabbott: The old code didn't even work by accident, it was
just broken. The right fix is to move the quoting around properly.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.
As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:
>>> exec('a = 1; print([a])', {}, {})
[1]
>>> exec('a = 1; print([a for b in [1]])', {}, {})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
File "<string>", line 1, in <listcomp>
NameError: name 'a' is not defined
>>> exec('a = 1; print([a for b in [1]])', {})
[1]
Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.
Fixes#12030.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
These previously lived in Optional settings, which generally caused
users to not read it.
(Also do a bit of reorganization of the "optional settings" area).
The docs specify passing hostname with the --hostname flag, which
doesn't match the actual usage in scripts/setup/setup-certbot. This
change fixes the docs to match the actual usage.
Apparently, this has been broken since
dee4e3fb89, due to the beforeSend code
here overriding the default beforeSend function that sets the CSRF
token. The correct fix was actually to just run the relevant code
directly before the channel.patch call.
Fixes#11938.
It is observed in Mozilla margin was considered from other side of thumbnail
due to some special padding issues observed in mozilla.
To fix this top and left value are assigned to 0 so that it automatically
takes its correct position in all browsers
Fixes#11867.
We never intended to render them for this use case as the result would
not look good, and now we have a convenient bugdown option for
controlling this behavior.
Since we're not storing the markdown rendering anywhere, there's
conveniently no data migration required.
Fixes#11889.
When we try to hover over Open or Download they were not highlighted
in night mode, because of incorrect specificity. This commit adds
highlighting in night mode (possibly fixing a regression when we made
night mode less aggressive about hover).
Fixes#11887.
Apparently, our invalid realm error page had HTTP status 200, which
could be confusing and in particular broken our mobile app's error
handling for this case.
This logic for passing through whether the user was logged in never
worked, because we were trying to read the client.
Fix this, and add tests to ensure it never breaks again.
Restructured by tabbott to have completely different code with the
same intent.
Fixes#11802.
Previously, the LDAP authentication model ignored the realm-level
settings for who can join a realm. This was sort of reasonable at the
time, because the original LDAP auth was an SSO solution that didn't
allow multiple realms, and so one could fully configure authentication
settings on the LDAP side. But now that we allow multiple realms with
the LDAP backend, one could easily imagine wanting different
restrictions on them, and so it makes sense to add this enforcement.
Apparently, our new validator for stream color having a valid format
incorrectly handled colors that had duplicate characters in them.
(This is caused in part by the spectrum.js logic automatically
converting #ffff00 to #ff0, which our validator rejected). Given that
we had old stream colors in the #ff0 format in our database anyway for
legacy, there's no benefit to banning these colors.
In the future, we could imagine standardizing the format, but doing so
will require also changing the frontend to submit colors only in the
6-character format.
Fixes an issue reported in
https://github.com/zulip/zulip/issues/11845#issuecomment-471417073
According to GitHub's webhook docs, the scope of a membership
event can only be limited to 'teams', which holds true when a
new member is added to a team. However, we just found a payload
in our logs that indicates that when a user is removed from a
team, the scope of the membership is erroneously set to
'organization', not 'team'. This is most likely a bug on
GitHub's end because such behaviour is a direct violation of
their webhook API event specifications. We account for this
by restricting membership events to teams explicitly, at least
till GitHub's docs suggest otherwise.
This fixes the bug where the `Saved` state button faded out almost
instantly (that is actually 300 ms) and `Discard` button fades out
along with `Saved` state button; the key problem here was that the
setTimeout intended to fade was actually delaying the transition from
"saving" to "saved".
Now, first of all, we use `setTimeOut` function to fadeout elements giving
fadeout_delay time as `800 ms` and we hide discard button during `saving`
state. Also, when `Discard` button is selected, `Save changes` and `Dicard`
fade out simultaneously.
Fixes: #11737.
This was introduced in e0236646
For 1.5 years we did not find a case that needed it (besides the
`a` tag hover state, that is not obvious if it was needed or it was
used as an example)
It is not obvious if this solution was a good idea. The concern was
that `body.night-mode` is more specific than `body` and some styles
might override others less specific in cases we might not want that.
Of course, we want that in the majority of cases, and css-specificity
rules are not simple to comprehend.
Good further reading:
http://cssspecificity.com/https://specificity.keegan.st/
The added complexity of the resulting styles and the added code that
might not serve any practical purpose seem to not be worth it.
This fixes some annoying copy-paste issues we've seen with users
accidentally getting a weird invisible unicode character in their URL
format string when trying to copy-paste an existing linkifier to
use for a new linkifier.
Fixes#10828.
It was impossible to search people in mobile browsers because virtual
keyboard used to fire resize event and the function call that we used
to handle this event caused the input field to loose focus and this
made it impossible to type in the people search bar.
The code in this commit fixes this by simply ignoring the resize
events when the user wants to search.
Fixes#11795.
This reverts commit ff90c0101c but keeps
the test cases added for reference.
This was reverted because it was both not a clean solution and created
other realm filters bugs involving dashes (etc.).
Apparently, our testing environment for this configuration was broken
and didn't test the code we thought it did; as a result, a variable
redefinition bug slipped through.
Fixes#11786.
Earlier the behavior was to raise an exception thereby stopping the
whole sync. Now we log an error message and skip the field. Also
fixes the `query_ldap` command to report missing fields without
error.
Fixes: #11780.
Apparently, while upgrade-zulip-from-git always ensures that zulip
deployment directories are owned by the Zulip user, unpack-zulip (aka
the tarball code path) has them owned by root.
The user ID detection logic in su_to_zulip's helper get_zulip_uid was
intended to support both development environments (where the user ID
might vary) and production environments. For development
environments, the existing code is fine, but given this unpack-zulip
permissions issue, we need to have code to fallback to 'zulip' if the
detection logic detects the "zulip" user has having UID 0.
It seems like 1871d00bb2 renamed `/help/at-mention-a-user` to `/help/mention-a-user-or-group` but missed this link that shows up on the "You haven't been mentioned yet!" screen. Right now it leads to a "no such article page".
The overall goal of this change is to fix an issue where on Ubuntu
Trusty, we were accidentally overriding the configuration to serve
uploads from disk with the regular expressions for adding access
control headers.
However, while investigating this, it became clear that we could
considerably simplify the mental energy required to understand this
system by making the uploads-route file be unconditionally available
and included from `zulip-include/app` (which means the zulip_ops code
can share behavior here).
We also move the Access-Control-Allow-* headers to a separate include
file, to avoid duplicating it in 5 places. Fixing this duplication
discovered a potential bug in the settings used for Tornado, where
DELETE was not allowed on a route that definitely expects DELETE.
Fixes#11758.
The night logo synchronization on the settings page was perfect, but
the actual display logic had a few problems:
* We were including the realm_logo in context_processors, even though
it is only used in home.py.
* We used different variable names for the templating in navbar.html
than anywhere else the codebase.
* The behavior that the night logo would default to the day logo if
only one was uploaded was not correctly implemented for the navbar
position, either in the synchronization for updates code or the
logic in the navbar.html templates.
After discussion, we decided that the red color is too distinct
and does not convey the idea of "almost offline".
This changes the new "unavailable" status circle's color from dark
red to grey, the same color used by the "offline" status circle.
Fixes#11589.
Adds SCSS style for the "unavailable" user status and enables its
usage in `buddy_data.js`.
The style is a red circle with a horizontal line. The values might
look a bit 'magic' but they were considered carefully ` height` of
1px was too thin, 2px was too thick, thus 1.5px was chosen.
This is important for situations such as with our Zapier app,
where the requesting user may be a bot that would like to access
its owner's subscriptions.
Tweaked by tabbott to eliminate the 2^N growth of cases in
do_get_streams.
tests now ran in 7.649s from 9.297s. And this test works just as well
with 3 bots, since only 3 database queries with 3 bots confirms we're
not doing linear queries in the number of bots in the organization.
We want to use the baseline features of bugdown, but not fancy things
like inline URL previews, since the whole structure of stream
descriptions is to have a single-line thing supporting some
formatting.
The migration part of this change fixes a bug encountered by some
organizations upgrading from older versions of Zulip.
This allows us to have some features using bugdown rendering where
inline image previews will not be rendered (which would be problematic
for e.g. stream descriptions).
Guest users will just get an empty list of default streams; we also
hide the "Default streams" organization view from the guest users UI.
This is for consistency with not providing guest users the full list
of streams in an organization.
Fixing this involves fixing the backend to handle unchanged field
submissions of the Zoom credentials without trying to re-validate the
credentials (for performance) as well as to fetch the already-sent
secret.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.
No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
This avoids a spurious permission error inside the Postgres
`resolve_symlinks` function if we don’t have access to the current
working directory (e.g. we’re running with cwd /root inside `su
zulip`).
While we’re here, add a defensive `--` argument.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This saves about 8% of the runtime of our total response middleware,
or equivalently close to 2% of the total Tornado response time. Which
is pretty significant given that we're not sure anyone is using statsd
in production.
It's also useful outside Tornado, but the effect is particularly
significant because of how important Tornado performance is.
This avoids parsing these functions on every request, which was
adding roughly 350us to our per-request response times.
The overall impact was more than 10% of basic Tornado response
runtime.
If you topic-edited a single message within a narrow, we would update
all our unreads/sidebar/etc. data structures, and would rerender the
message if appropriate. However, for the corner case of being inside
a topic narrow when you did this, we didn't have logic to remove the
message from the narrow (which is the appropriate situation when you
just topic-edited a message in a narrow).
When topic-editing multiple messages including the currently selected
message (the more common case), we would end up changing the narrow,
resulting in this issue being masked.
Fixes#11601.
The correct behavior here is that we want to ensure there is
whitespace in between the syntax being added and the content on either
side. Our smart_insert logic handled this for the cases that were
common with inserting emoji (etc.), but didn't handle the more complex
cases with "quote and reply".
Fixes#11702.
We'll use this in the push-notifications code, in a context where
there should definitely already be UserMessage rows if everything's
gone normally... but explicitly checking at the top seems like the
right pattern from a secure-coding perspective.
When a bunch of messages with active notifications are all read at
once -- e.g. by the user choosing to mark all messages, or all in a
stream, as read, or just scrolling quickly through a PM conversation
-- there can be a large batch of this information to convey. Doing it
in a single GCM/FCM message is better for server congestion, and for
the device's battery.
The corresponding client-side logic is in zulip/zulip-mobile#3343 .
Existing clients today only understand one message ID at a time; so
accommodate them by sending individual GCM/FCM messages up to an
arbitrary threshold, with the rest only as a batch.
Also add an explicit test for this logic. The existing tests
that happen to cause this function to run don't exercise the
last condition, so without a new test `--coverage` complains.
Accomplished by adding a function to clear the status message with
an empty string. The html is then updated to reflect changes without a
refresh.
Currently, it's a small hassle to clear a status message. This option
makes things a bit easier.
Fixes#11630.
help.github.com seems to have a bug where HEAD on a redirected page
returns 404. This causes tools/test-documentation to fail. Fix it by
skipping the redirects.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This new helper allows us to do the same operation
on every message in our message_store. We will
use this in a future commit to clear the `is_tall`
flags on all messages, after a resize.
We should be somewhat cautious about using this,
but simple operations should be really fast, even
if you have lots of messages in the store.
Previously, if you were in the process of editing the last message in
a narrow and a new message came in, we'd rerender that second-to-last
message, causing your editing widget to lose focus (and thus the next
few keys you typed to be interpreted as keyboard shortcuts, which
had a good chance of resulting in your navigating somewhere random).
This rerendering was essentially unnecessary; the only change to state
going into the rendering process was the next_is_same_sender CSS class
being toggled on the messagebox in the message. So, at most, we
should have been just toggling that CSS class (and this commit makes
us do precisely that).
It seems like we could further improve this code by just removing the
next_is_same_sender CSS class entirely and removing this block, but
I'm leaving that for follow-up work.
Fixes#11656.
This fixes an issue where closing stream search was not working if
user had not entered a search term and tried to close the search box
by clicking on the close icon; the problem was that we'd end up
re-opening the widget immediately after through event propagation.
Fixes: #11636.
The is_editable field includes topic edits, so we need a separate
field for whether to display these icons which are all for content
editing.
Fixes#11666.
Adblock Plus's "Block social media icons tracking" setting blocks
images with for social media platforms in their names from loading, so
we rename the Google logo to bypass this.
Adblock Plus's "Block social media icons tracking" setting blocked
integration logos for social media platforms from loading, so the logos
are renamed to bypass this.
Fixes#11590.
There’s no reason to do this unless you’re, like, trying to trip the
Let’s Encrypt rate limits (or perhaps trying to manually test this code).
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
When copying a message by clicking on "copy and close" button in
message edit box an alert appears that says "Copied!"; Background
of the message is set corresponding with the day mode but not the
night mode. This changes the background of the alert message to
the dark color in night mode.
Also adds tests to ensure that we do not accidentally overwrite
the 'beginning' variable that contains the message content upto
that point. These should prevent similar errors in the future.
The bug was added in 8119258c4d.
This change was prompted by a possible bug on Clubhouse's end. In
general, if a branch is added, it also prompts a workflow state
transition in its primary story.
However, our webhook error logs show an erroneous payload where
the same branch is added to another story, but there is no
workflow state transition. Also, both the stories are grouped in
the same payload, which confused/broke our code. Ideally, this
shouldn't happen and is most likely a bug on Clubhouse's end.
In most cases, changes included in Clubhouse payloads never
pertain to more than one parent entity (stories) simultaneously,
and we usually operate under the assumption that the changes
included therein are related to each other in terms of their
parent object (story or epic) and not a child object (the GitHub
branch).
A check suite is a collection of check runs. We care a lot more
about the outcomes of check runs in this case because check_run
payloads are a lot more informative than check_suite payloads.
(And in any case, the check_suite events are primarily for notifying
tools like CI to run checks).
We only support notifications for events where a check run has
completed. Notifications for when a check run has been queued or
is in progress are not very informative and may be too noisy.
The bug here was that when we rerendered messages following local echo
through the echo.process_from_server code path, the eventual call to
_rerender_header() made the implicit assumption that all messages in a
message group had the same date. As a result, it created a totally
new/fake message group and called the rendering logic on that group
without calling the functions for setting up recipient row dates,
which would always result in no recipient bar date being added. This
bug was latent/invisible before, because when introduced, the locally
echoed messages were always being added to a recipient group from
today, where the recipient bar's date area was by default empty anyway.
This latent bug was revealed when we modified the structure of the app
to do date dividers between individual messages within a message
group, rather than strictly between message groups.
When we're handling a single message that was locally echoed, there
will very likely be 0 messages not removed by
`echo.process_from_server`, and we can skip the unnecessary call to
`message_events.insert_new_messages`. This is a small performance
optimization and logical simplification when sending messages.
We do not anticipate our UI for showing stream descriptions looking
reasonable for multi-line descriptions, so we should just ban creating
them.
Given the frontend changes, multi-line descriptions are only likely to
show up from importing content from other tools, in which case
replacing newlines with spaces is cleaner than the alternative.
This commit achieves two things:
1. Changes the UI of the "Create stream" form to make the
textarea previously used to get the stream description
a simple input field of type text (to suggest a single
line description).
2. Adds an extra check on the frontend side to make sure that
when users create a new stream via. the "Create stream"
option in the settings panel, they can't enter any newline
characters (i.e. we disallow the enter key from being
registered when typing out the stream description).
We must also make sure that they cannot copy-and-paste over
descriptions containing newline characters.
resolves#11617
For consistency, we should keep all the code that works with
@mentions in markdown.js. In this case, message_list_view was
rewriting the contents of the mentions in cases where users'
names had been changed since we rendered their mention.
This change should help people discover to distinguish
silent mentions in text as a part of Zulip syntax while
differentiating them from regular mentions.
To test formatting we want a hard coded date, so we
can verify the date arithmetic with stable dates.
To make the test less brittle, we disable the
feature to remove old drafts.
This was an emergency fix. We should probably just
remove the last N drafts instead of having the 30-day
limit. Or we should have a better way to stub the cutoff
date.
This optimizes test-backend by skipping webhook
tests when run in default mode.
Tweaked by tabbott to extend the documentation and update the CI
commands.
The payloads for this event are missing some important details
about the Project's changes, such as the name of the project,
the card's column name, etc. Without such details, the resultant
notifications would not be useful at all!
The background color of the portico pages aren't true white,
so this commit adjusts it to match the actual portico page
background color to eliminate differences.
This is mostly adding markup, calling some convenient
functions in buddy_data.js, and adjusting CSS.
To make the circles update dynamically, I mostly
orchestrate this though activity.js for now. It's
possible we'll want to adjust that eventually to
happen through something like a `presence_events`
dispatcher, but that's essentially what
a good part of `activity.js` does now.
We're soon gonna have user circles in four different places,
and the fourth place, Private Messages, will have different
size/position CSS.
Now each component does positioning and sizing in its
main CSS file:
user info, group info -- popovers.scss
buddy list, group PMs -- right-sidebar.scss
(We also use the more explicit syntax for padding each
side.)
We now have a function get_user_circle_class
that returns one of these values:
"user_circle_green"
"user_circle_orange"
"user_circle_empty"
And we put that in the templates.
And then CSS renders the circle of the appropriate
color.
The unit tests now explicitly capture whether
we are rendering the correct kind of circle.
This is a pure code move.
We want to use user circles in the left sidebar,
so this code will no longer belong in
right-sidebar.scss.
This code is just related to drawing the circles.
We can still position in size in other CSS files
(with more context-specific selectors).
This fixes a longstanding UI issue when you have way too many recent
private message conversations, as you can now scroll down the list to
find what you're looking for.
Fixes#5384.
Date separator exists inside the message_row, which causes the
message controls to be visible even when hovering on date
separator. These two rules are redundant and cause this buggy
action. Other rules handle the behaviour of message controls
being visible on message box hover. Hence these can be removed.
Previously, if you scrolled down all the way in the left sidebar, and kept
your mouse hovered over a link, you had a feeling that there was still "more
stuff", since you could see the top of "Back to Zulip" peeking out over the
top of the URL Chrome (and maybe some other browsers) add in the bottom left
corner.
This just adds a bit of margin so that "Back to Zulip" is above that when
scrolled all the way down.
The function that was called here has no side
effects. If you don't use its value, it's just
wasted computation. The real action happens
in the subsequent calls to `rebuild_recent`.
Apparently, virtualenv-clone ends up copying the success-stamp file
that we use to track whether a virtualenv was successfully
provisioned, which results in problems if we get a network error in
the pip install stage afterwards.
The comment explains our fix, but basically we just delete
success-stamp after the clone.
Fixes#11301.
Having it say "Clear" when you delete an existing status was a nice touch,
but it's confusing when you first open the modal and the text of the button
says "Clear".
I think the right medium-term solution here is for this modal to have "Save"
and "Cancel" buttons, and for there to be a small UI element in the user
popover itself that allows you to clear your current status.
ACCOUNT_ACTIVATION_DAYS doesn't seems to be used anywhere.
INVITATION_LINK_VALIDITY_DAYS seems to do it's job currently.
(It was only ever used in very early Zulip commits).
We were using a hardcoded relative path, which doesn't work if you're
not running this from the root of the Zulip checkout.
As part of fixing this, we need to make `LOCAL_UPLOADS_DIR` an
absolute path.
Fixes#11581.
We now use `fix_positions` to avoid cropping the emoji
picker. You can see cropping pretty easily on a short
screen if you click the smiley icon for reactions on a
message. It's a bit tricky to repro, since some
of the current top/bottom placements are correct, but
it's definitely reproducible.
I think there are opportunities to both simplify
and optimize `popovers.compute_placement`, so that it
plays nicer with `fix_positions`. For example, I would
bias it even more strongly toward favoring right/left
placement. But there are complicating factors--it is
also used by the hotspot code.
And I wanted to especially preserve the current
behavior when you launch the picker from the compose
box. That's one place where it looks pretty bad if
you select "right" instead of "bottom".
The fix_positions argument here fixes the horizontal
position of the stream popover.
It also fixes the vertical position, both in the default case, and
also doing an appropriate adjustment for the case that the color
picker is open.
This contains a few changes by tabbott to, rather than hiding the
arrow unconditionally, only do so when it would no longer point at the
right part of the screen.
Fixes#2374.
Fixes#6059.
Fixes#7290.
We use the `fix_positions` options every time
we launch a user popover, whether it is from
the message pane avatar or the buddy list
chevron.
For the message pane case, we can eliminate
some complexity related to trying to put
the menu above or below the avatar. We now
always suggest "right", and if there are
constraints due to being close to the edge
of the screen, the fix_positions code
will take care of it.
The patch to bootstrap will make the position smarter, but we still
want to preserve the 100px default vertical offset we chose for visual
reasons.
Tweaked by tabbott to preserve the visual design.
Changed <h5> to <p>, and removed the special formatting of
.empty_search_text to make this more in line with the formatting we
generally use with empty narrows.
Since da8f4bc0e back in August, this control flow has caused
`flags.active_mobile_push_notification` to be cleared if we don't send
these `remove` messages at all, and if we send them directly to GCM...
but not if we send them via the Zulip notification bouncer.
As a result, on a server configured to send `remove` notification-messages
via the bouncer, we accumulate "active" messages and never clear them.
If the user then does `mark_all_as_read`, we end up sending a `remove`
for each of those messages again, and all in one giant burst. We've
seen puzzling bursts of hundreds of removals pass through the bouncer
since turning on removals on chat.zulip.org; it's likely many of them
are caused by this bug.
This issue was made more acute with f4478aad5, which unconditionally
enabled removals.
Test added by tabbott.
This removes the left border extending the stream label from the
recipient bar in from the drafts in drafts modal. Those borders are
important in the message feed for containing several messages, but
here we're only ever going to show individual drafts, and this change
avoids potential color clashes with the blue box surrounding the
recipient blocks.
This removes the change in background to a darker one for active draft,
also removes the change in recipient_row_date color to blue; adds a blue
border around the draft box.
Since the bootstrap popovers are destroyed asynchronously so opening a
emoji popover in quick succession like by clicking the reaction button
on another message was causing a race condition which was causing some
operations to be applied on a destroyed emoji popover. This commit
fixes it by making sure to apply any operations only to the currently
active popover.
Fixes: #9851.
I'm torn about this, since there is good content here. But ultimately I think
* This page is a lot of work to write and maintain.
* In most cases, the right thing is for people to find the page that
explains the full feature. E.g. if you don't know what an "administrator"
is, the page I hope you find is "Roles and Permissions". For bots, it's
"Bots and Integrations". Writing a punchy short summary for a glossary
that does better than that is possible, but not fast.
* People find things via search, e.g. by Googling "What is X in Zulip",
rather than looking for a glossary.
* This page was written more than 3 years ago, before we had 100+ help
articles. So it may have served a purpose in the past that no longer
exists.
Adds three helper functions - `row_with_focus`, `row_before_focus` and
`row_after_focus` to get the focused, previous and next to focused
draft rows respectively.
`delete_id` in `drafts.js` referred to the next draft row which was
to be focused when deleting using hot keys. The var name was absurd
and is hence renamed.
Adds a `remove_draft` function which deletes the draft and updates the ui
by removing it from the list of drafts.
Also adds comments to increase readability.
Show "sent to different narrow" notification and other such notification by
notifications.notify_local_mixes for non locally echoed message sent by
current client.
With significant new comments added by tabbott.
Fixes: #11488.
Previously, this was only available in the Zulip development
environment.
Further work is needed on documenting this and how to use it for
managing work to follow up on.
This is the only server-side change required for the FCM migration!
Optionally, at some point in the future we might choose to migrate
to the new ("v1") API which FCM also offers. Nothing revolutionary
but there are some nice things about it:
https://firebase.google.com/docs/cloud-messaging/migrate-v1
The client-side fix to make these not a problem was in release
16.2.96, of 2018-08-22. We've been sending them from the
development community server chat.zulip.org since 2018-11-29.
We started forcing clients to upgrade with commit fb7bfbe9a,
deployed 2018-12-05 to zulipchat.com.
(The mobile app unconditionally makes a request to a route on
zulipchat.com to check for this kind of forced upgrade, so that
applies to mobile users of any Zulip server.)
So at this point it's long past safe for us to unconditionally
send these. Hardwire the old `SEND_REMOVE_PUSH_NOTIFICATIONS`
setting to True, and simplify it out.
After clicking on checkbox saying "Show text only version" UI was rendered
correctly but after refreshing page keeping checkbox checked, emails were
shown without "text only version" but checkbox value remained checked.
Now after refreshing page checkbox value changes to its default value.
For users putting Zulip behind certain proxies (and potentially some
third-party API clients), buffer sizes can exceed the uwsgi default of
4096. Since we aren't doing such high-throughput APIs that a small
buffer size is valuable, we should just raise this for everyone.
The width of the messages div is set to 600px, while the
digest-email-container can be 500px at the most. Increasing the width
of the digest-email-container makes the /digest slightly more
readable.
The padding changes move the number a bit to the right and down, towards
where the bottom right corner of an unread count box would have been. This
makes the number look better aligned with the unread count boxes above it.
We've for a while had logic to set plan_type to LIMITED when importing
into Zulip Cloud; we need corresponding logic to set it to SELF_HOSTED
when importing into a self-hosted server.
Fixes#11541.
We swallow the error if our python_to_js_filter code is
unable to parse some python regex properly. This ensures
that the web app stays responsive.
We would fail to show an accurate local echo for these
regexes, however, the backend would act as the final
authority for handling the realm pattern conversion.
Since on replacing the first 'P<>' group, we remove this text from
the string, we have to make the RegExp start looking from index 0
again to properly convert later 'P<>' groups to JS regex syntax.
For Google auth, the multiuse invite key should be stored in the
csrf_state sent to google along with other values like is_signup,
mobile_flow_otp.
For social auth, the multiuse invite key should be passed as params to
the social-auth backend. The passing of the key is handled by
social_auth pipeline and made available to us when the auth is
completed.
For internal stream messages, most of the time, we have access to
a Stream object. For the few corner cases where we don't, it is a
much cleaner approach to have a separate function that accepts a
stream name than having one multi-option helper that accepts both
names and objects.
It looked kinda terrible in between the two
user lists.
There is some discussion here (I have to break the
link into two lines to make gitlint happy):
https://chat.zulip.org/#narrow/stream/
101-design/topic/user.20sidebar.20in.20left/near/697682
We want the search widget, when visible, to be
outside the scroll container for the stream list.
One obvious use case is if you start scrolling, and
then realize it might be less effort to search.
Also, for user search, it already worked this way.
We have to add a couple resizing hooks here, but
it's not necessary to change the actual resize
calculation, since we move the section inside
of #streams_header, which is already accounted
for.
The only markup change here is to add
a `stream_search_section` class. I don't
know why we use `notdisplayed` here instead of
jQuery, or what `input-append` is for, but I
considered them outside the scope of this change.
We can also remove some crufty CSS that was
compensating for it being inside the container.
The bot's details (such as ID and name) are used to format a
connection label in Zapier's UI. This allows users to distinguish
between different Zapier-to-Zulip connections set up with different
bots.
First, we are not removing Group PMs from the
right sidebar, where most people see it.
There is a setting called:
[ ] User list in left sidebar in narrow windows
There are probably very few people that turn that on,
and even when they do, the setting only takes effect
when your window is less than a certain width.
This feature bitrots very quickly, because very few
core maintainers use it.
It's already kind of broken. It gets very crowded,
and we get CSS bugs when we move the right sidebar
into the left sidebar. (We can fix those bugs, but
they crop up unexpectedly due to the nature of CSS.)
We historically tried to maintain a ratio between
stream list, single-user buddy list, and group-user
buddy list, but the group-user buddy list gets
particularly crowded out, and it's basically useless
now.
We want to revisit the entire feature eventually, but
this commit at least gives the normal buddy list some
breathing room.
Also, if you need to see the info in the group PM
list, you can basically expand "Private Messages" to
see your recent group PM conversations. And if you
want to see who's actually online, that info is
already implicit from the normal buddy list.
If users have the option to put user lists in
the left sidebar for narrow windows, they will
now get 15px more of real estate in the left
sidebar.
I just removed the `-15` fudge factor.
We were double-counting the keyboard icon's
margin (8px) when figuring out how much room
we had for the two users lists.
Now we just use the safe outer height of
the anchor tag.
This change only impacts users who have the setting
to put the user lists in the left sidebar when they
have a narrow window.
First, we move ".right-sidebar-items" as an entire
group.
Second, we append the items to "#left-sidebar"
instead of ".narrows_panel".
The name `bottom_sidebar` was misleading, because it
includes the entire "normal" left sidebar.
It includes the 4 narrow links at the top plus the
stream/topic list.
We now call is narrows_panel.
Note that the left sidebar sometimes also includes
the user list (with a display setting turned on).
And it will eventually include other views.
We also remove an intermediate value in the resize
calculations.
This adds date dividers within a single message group when the only
reason we had previously been splitting apart two message groups is a
change of date. The overall effect is a cleaner message list user
experience.
The downside of this change would be that the recipient bars no longer
will always show a new date for date changes; to fix that, we rewrite
how the floating recipient bars both set the date field on the
floating recipient bar itself, as well as ensure that non-floating
recipient bars don't show duplicate dates.
In a future design update where we modify how message recipient bars
look, we may very well be able to simplify this logic by removing some
of the dynamic nature of the recipient bar calculations. But this is
a good implementation of what remains.
Tweaked significantly by tabbott from Steve Howell's original, both to
extract these changes from a larger PR as well as to modify the
first_visible_message logic to handle some tricky corner cases.
Fixes#10171.
On clicking the edit button for a stream description, the stream's
unrendered description should be made editable as text instead of
the stream's rendered description (which would be displayed as HTML
instead of text).
This completes the effort to use backend-rendered stream descriptions
here. Fixes#11272.
Use the results of commit #73d26c8 to remove the method
`render_stream_description` in static/js/stream_data.js and instead
use the rendered_description attribute now being sent by the backend.
This will be a valuable optimization and a step towards removing the
need for the marked.js markdown parser and speeding up the client end.
Our html collects extra spaces in a couple of places. The most prominent is
paragraphs that look like the following in the .md file:
* some text
continued
The html will have two spaces before "continued".
This changes the border-radius to 6px for the tabbed display, which is not
in line with the current Zulip style for border-radius (4px). However 6px
really looks a lot better for this (possibly because it's a bigger box than
most of our other boxes?)
The message_edit_history UI was incorrectly inheriting its content
div's structure from message_edit_content, i.e. the form for editing
message content, and not message_content, i.e. the class for rendering
the content of messages in the message feed (which we also use for
drafts).
Fix this by changing the inheritance, while also adding a (currently
unused) class for any future customizations.
Fixes: #5629.
This extract functions `get_mention_candidates_data` &
`filter_mention_name` to make code reusable and cleaner and further use
the logic in silent mention syntax.
Having a tiny bit of margin below the stream list
makes it possible to see the bottom of the scrollbar.
It also makes it so that the scrollbar activates
for a tiny range of list sizes where before the
last element would have been right up against the
bottom of the page, but we wouldn't scroll.
I was hoping this would make things faster... it does, but sadly only
by about 70ms, 5% of this file's test runtime.
It sure does make this file rather less action-at-a-distance, though,
as well as fixing some duplication.
If we make a practice on the Zulip server of always explicitly setting
the desired priority, then when an old server doesn't set the priority
we can reasonably have the bouncer make a guess.
We need to move the update_group_date_divider call to run when a
message group is created. This achieves a few things:
* Fixes calling this multiple times useless for long message groups.
* It will soon no longer be correct to assume that every message
within a group has the same date, and in that case, we want to process
the date of the first message in the group, not of the last.
We only generate message_containers in one place, and that code path
already calls update_timestr. And update_timestr's effect only
depends on the message. Thus, this code was useless.
Commit 7d12e2019d (#10994) broke fresh
provisions by importing zproject.settings before we were done
modifying settings. Fix it by moving the generate_secrets invocation
to the earliest reasonable place.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
That is, this allows a Zulip server to now set the `priority`; but if
it doesn't, we use upstream's default value, which has the same effect
as we've always previously had by not setting it at all.
But when this is deployed to the push notifications bouncer server, it
does allow another server to set priority when pushing notifications
through the bouncer.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with re-fetching the Stream object by name.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with getting a Stream object back.
In small screen sizes, when the user presses shortcut `w` to search
for another user, the hide_all function calls in the search code path
would hide the right sidebar, immediately after opening it, making the
hotkey basically unusable.
We fix this by extracting a separate hide method that hides all true
popovers, but not the user list sidebar.
Fixes#11463.
The `uploadFinished` code switches on the composing mode, if we aren't
in the composing mode already. This causes the focus to be incorrect
when this code path runs due to an upload from the message edit
box. This commit fixes that logic to turn on the composing mode or
switches focus to the message edit box, depending on where the upload
was triggered.
So the top navbar is above the left sidebar
on the z-axis, not the y-axis.
So it doesn't make sense to use the top
navbar in calculating the size of the left
sidebar.
It kind of coincidentally works, since these
two numbers are closely related:
left sidebar top margin = 50
navbar height = 40
Calculating `bottom_sidebar_height` correctly
decreases its value by 10.
And then the only value that depends on it
is `stream_filters_max_height`. We were
subtracting out 10 there to make it work,
since `bottom_sidebar_height` was inaccurate
by +10. Now that's fixed.
The comment there was inaccurate--the
`stream_filter` div actually has a bottom
margin of 22px. The bottom margin does
have some consequences for scrolling,
but the main goal here is to make the
calculation return the same value but
be more accurate about what happens
toward the top of the screen.
We have always intended to have 10px of whitespace
below the navbar, and this enforces it directly
and explicitly in the CSS.
Note that the three major panels still should
have a margin of 50px, which is equal to
the safe outer height of the header (40px + 10px).
The name for_stream_name is more appropriate here. The name
for_stream is more suitable for a function that takes in a Stream
object, which we're about to add.
Our hash-naming of production assets interacted badly with the "look
at files in a directory" algorithm used to determine what sound
options exist for the "notification sound" feature. For lack of a
better solution, we fix this by excluding files with an extra `.` in
their name.
Eventually, we'll just make this the default, but for now, we should
fix the typos and use our standing approach for documenting running
management commands.
The border makes the alignment look nicer. Without
a border your eyes plays tricks on you and makes it
seem like numbers are not in the same column.
The border color is the same subtle color as the
backgrounds in others.
Because CSS is annoying, you have to tweak the padding
to make room for the border.
(It should look ok in night mode, too.)
Since the main autoscroll feature was implemeneted, the
maybe_advance_to_recently_sent_message logic had an unfortunate
structure, where the code for this potentially large scroll was
running AFTER the autoscroll decision was made, but before an actual
scroll could have occurred.
This resulted in code that was very difficult to reason about, as
there were 2 potential sources of scrolling when you send a new
message, with little connection between their implementations either
in location or implementation.
Moving this into the main autoscrolling code path clarifies the code,
with the added benefit of fixing a bug where we would report to the
user that they needed to scroll down when in fact we were just about
to scroll the bottom of the feed into view (via
maybe_advance_to_recently_sent_message).
With this change, we never display the "you need to scroll manually"
message in the cast that we just scrolled you there via selecting a
message.
When you just sent a large message, our logic for "you need to scroll"
notifications did not correctly take into account the height of the
compose box. This was easily reproduced when sending very long
messages. The correct solution requires a bit of math to compute what
the visible area will look like after the compose box is closed.
This should be the final fix to #11138.
Extracts out common tests so that future social-auth backends can
be tested without duplicating tests. I have been careful to not
change any testing logic.
This adds a function that controls the whole process of applying
markdown and displaying the markdown rendering preview on request;
This is required to avoid code duplication when adding preview feature
to message-edit UI.
Add all the stop words to page_params, reading from the
`zulip_english.stop` database, with caching to avoid loading the file
on every page load.
Part of #10592.
This causes changing the email_address_visibility field to actually
modify what user_profile.email values are generated for users, both on
user creation and afterwards as email addresses are edited.
The overall feature isn't yet complete, but this brings us pretty close.
The original commit made a number of well-meaning but suboptimal product
changes to the invoice events, such as threading them under the invoice id
rather than the customer id. It also seems to be causing 500s for
invoiceitem events, though I'm not sure why.
This reverts commit 65489b0391.
We had disabled reference style links in bugdown, however,
we hadn't disabled them in marked. This commit rectifies
that and adds test cases for the same.
Fixes#11350.
This helps keep the realm.json small and easy to process; previously,
almost the entire size of that file was the analytics data.
We implement this by refactoring the analytics Config objects into a
separate subroutine that writes to a separate file, plus the
corresponding import code.
Manual testing was performed by exporting the 'analytics' realm, and
importing back to a newly created 'test' realm. The 'test' realm was
then exported and the json files were inspected. The data appeared
consistent with no abnormalities.
Fixes: #11220.
AssertionErrors were raised when attempting to run manual comparison
tests to ensure correctness when exporting the analytics realm using
export_from_config. This was caused by this populate_analytics_db
stream being created without any subscribers, which violates an
invariant.
We fix this by simply subscribing the 'shylock' user to that stream.
The beforeSubmit function was a feature of the jquery-form plugin that
we removed months ago; the appropriate similar feature of jQuery's
built-in AJAX library is beforeSend.
We eliminated use of this function in outgoing_webhook.py in
bdc95b5d72.
Tweaked by tabbott to also eliminate code only used for that mock.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
In ebdd55814c, we added zilencer imports
without using the proper mocking procedure for when zilencer is not enabled.
This whole setup is a mess and probably we should enable zilencer
unconditionally in a future version.
On usage errors (except --help), write usage message to stderr and
exit with nonzero status.
Forbid setting the hostname and email to the example values. Those
are specifically checked for and would fail later.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This code will correctly add video call link to the message
textarea based on whether 'Add video call' was selected from
message composition form or message edit form.
The implementation was semi-rewritten by tabbott to remove an
unnecessary global variable, with fixes for the unit tests from
showell.
Fixes#11188.
This commit does the following three things:
1. Update stream model to accomodate rendered description.
2. Render and save the stream rendered description on update.
3. Render and save stream descriptions on creation.
Further, the stream's rendered description is also sent whenever the
stream's description is being sent.
This is preparatory work for eliminating the use of the
non-authoritative marked.js markdown parser for stream descriptions.
This adds a new API for sending basic analytics data (number of users,
number of messages sent) from a Zulip server to the Zulip Cloud
central analytics database, which will make it possible for servers to
elect to have their usage numbers counted in published stats on the
size of the Zulip ecosystem.
This is primarily a feature for onboarding, where an organization
administrator might send a bunch of random test messages as part of
joining, but then want a pristine organization when their users later
join.
But it can theoretically be used for other use cases (e.g. for
moderation or removing threads that are problematic in some way).
Tweaked by tabbott to handle corner cases with
is_history_public_to_subscribers.
Fixes#10912.
This replaces the current usage of stream names with stream ids.
This commit also removes the `traditional` attribute from the invite
form as now we are sending stream_ids as an argument; this was the
only place in the codebase we used traditional=true, and it's great to
have it removed.
This function unlike `invite_streams()` returns an array of objects having
various info (name, stream_id, invite_only, default_stream) related to
streams rather than an array of names of streams.
We now compute the class that drives the tiny
green/orange/empty dot in the user popover using
the same logic as the buddy list.
This was broken in the early implementation of
set/clear-away, but it was never released.
Fixes#11413
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py). This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Ever since we implemented support for stream IDs in Addressee,
Addressee.stream_name() can now return None. This commit ensures
that _internal_prep_message only calls ensure_stream when
Addressee.stream_name() is not None.
This commit also contains the following auxiliary changes:
* Adds a custom exception, StreamWithIDDoesNotExist for when
a stream with a given ID does not exist because the error
message returned by StreamDoesNotExist only makes with stream
names, not IDs.
* Adds a new helper, get_stream_by_id_in_realm, which is similar
to get_user_profile_by_id_in_realm (introduced in #10391).
* Adds a helper, validate_stream_id_with_pm_notification, which
returns the Stream object associated with a given ID and also
handles PM notifications to the bot owner if the message was
sent by a bot and if the stream does not exist or has no
subscribers.
* Modifies the message sent by send_pm_if_empty_stream to
accommodate stream IDs.
Note that all of the above changes are required before check_message
can be modified to support stream IDs.
We've had this code oscillated a few times; the original comparison
was added as part of Stride import but broke HipChat import.
c34a8f2e69 fixed HipChat import but
regressed Stride.
This change fixes this for both HipChat + Stride.
This makes it possible to mention a user with a name like Gaël that
contains diacritics by typing e.g. "Gael", significantly reducing the
need to use a special keyboard to mention other users.
Fixes#11183.
This reverts commit ec9f6702d8.
Now that pika 0.13.0 has merged our PR to not import twisted unless it
is needed, we don't need to use this performance hack in order to
avoid wasting time importing twisted and all its dependencies.
The only time we set the `home` flag to true
is when it's the last (and only) item in the
list, in which case we flip `hash` to false
at the end of `make_tab_data()`.
So the section of code where both `home` and
`flag` were true is dead code.
Also, we can use `else` instead of `unless`.
The following elements in the top left corner
are major components of our app:
All messages
Private messages
Starred messages
Mentions
We can now find them directly:
$('.top_left_all_messages')
$('.top_left_private_messages')
$('.top_left_starred_messages')
$('.top_left_mentions')
Before this, we had to build up complicated selectors
like below:
exports.get_global_filter_li = function (filter_name) {
var selector = "#global_filters li[data-name='"
+ filter_name + "']";
return $(selector);
};
I don't think any newbie would know to grep for "global_filter",
and I've seen a PR where somebody added specific markup here
to "Private messages" because they couldn't grok the old scheme.
Another thing to note is that we still have a "home-link"
class for "All messages", which overlapped with portico
code that had the same name. (There were some inaccurate
comments in the code relating to the tab bar, but we don't
actually have a way to click to the home view in the tab
bar any more.) I'll eliminate that cruft in another commit.
For this commit the four elements still have the
"global-filter" class, since there's some benefit to being
able to style them all as a group, although we should give
it a nicer name in a subsequent commit.
Most of this PR is basic search/replace, but I did add a
two-line helper: `top_left_corner.update_starred_count`
This seems to be a common enough pitfall to justify
a bit of extra handling. Example output:
$ ./tools/run-dev.py
Clearing memcached ...
Flushing memcached...
OK
Starting Zulip services on ports: web proxy: ...
Note: only port 9991 is exposed to the host in a Vagrant environment.
ERROR: You probably have another server running!!!
Traceback (most recent call last):
File "./tools/run-dev.py", line 421, in <module>
app.listen(proxy_port, address=options.interface)
File "/srv/zulip-py3-venv/lib/python3.5/...
server.listen(port, address)
File "/srv/zulip-py3-venv/lib/python3.5/...
sockets = bind_sockets(port, address=address)
File "/srv/zulip-py3-venv/lib/python3.5/...
sock.bind(sockaddr)
OSError: [Errno 98] Address already in use
Terminated
Apparently, the "continue to registration" flow used a subtly invalid
way of encoding the full name. We put in the query part of the action
URL of the HTML form, but apparently HTML forms with a `GET` type will
ignore the query part (replacing it with any input values), which
makes sense but doesn't do what we want here. There are a few sane
ways to fix it, but given that the encoding logic we had before for
including the name in the URL was ugly, I'm pretty happy with just
adding a hidden input to the form for the name.
As part of Google+ being removed, they've eliminated support for the
/plus/v1/people/me endpoint. Replace it with the very similar
/oauth2/v3/userinfo endpoint.
This additional logic to prevent resizing is certain circumstances
(file size, dimensions) is necessary because the pillow gif handling
code seems to be rather flaky with regards to handling gif color
palletes, causing broken gifs after resizing. The workaround is to
only resize when absolutely necessary (e.g. because the file is larger
than 128x128 or 128KB).
Fixes#10351.
We had initially designed the poll widget like a blog
post with comments beneath it but it makes more sense
to think of it as just a simple poll with options.
We add a new syntax which converts the messages like the following:
```
/poll Who do you support?
Nadal
- Djokovic
```
to a poll with the two names as options. The list syntax is optional
since anyone making a poll is likely to want to create a list anyway.
This doesn't have any security impact, since we overwrote any other
fields in any case, and also this step happens before the security
part of input validation for stream creation. But this does improve
error messages if one tries to specify other arguments, and also makes
more clear that the `description` argument is supported here.
Refactor the potentially expensive work done by Beautiful Soup into a
function that is called by the alter_content function, so that we can
cache the result. Saves a significant portion of the runtime of
loading of all of our /help/ and /api/ documentation pages (e.g. 12ms
for /api).
Fixes#11088.
Tweaked by tabbott to use the URL path as the cache key, clean up
argument structure, and use a clearer name for the function.
In very old Slack workspaces, slackbot can appear as "Slackbot", and
the import script only checks for "slackbot" (case sensitive). This
breaks the import--it throws the assert that immediately follows the
test. I don't know how common this is, but it definitely affected our
import.
The simple fix is to compare against a lowercased-version of the
user's full name.
Earlier, our realm filters didn't render for languages that do not
use spaces (eg: Japanese) since we used to check for the presence
of an actual space character. This commit replaces that logic with
a complex scheme to detect word boundaries.
Also, we convert the RealmFilterPattern to subclass InlineProcessor
and make use of the new no-op feature in py-markdown 3.0.1 where we
can tell py-markdown that our pattern didn't find a match despite
the initial regex getting matched.
Fixes#9883.
Since we are building our parser from scratch now:
1. We have control over which proccessor goes at what priority number.
Thus, we have also shifted the deprecated `.add()` calls to use the
new `.register()` calls with explicit priorities, but maintaining
the original order that the old method generated.
2. We do not have to remove the processors added by py-markdown that
we do not use in Zulip; we explicitly add only the processors we
do require.
3. We can cluster the building of each type of parser in one place,
and in the order they need to be so that when we register them,
there is no need to sort the list. This also makes for a huge
improvement in the readability of the code, as all the components
of each type are registered in the same function.
These are significant performance improvements, because we save on
calls to `str.startswith` in `.add()`, all the resources taken to
generate the default to-be-removed processors and the time taken to
sort the list of processors.
Following are the profiling results for the changes made. Here, we
build 10 engines one after the other and note the time taken to build
each of them. 1st pass represents the state after this commit and 2nd
pass represent the state after some regex modifications in the commits
that follow by Steve Howell. All times are in microseconds.
| nth Engine | Old Time | 1st Pass | 2nd Pass |
| ---------- | -------- | -------- | -------- |
| 1 | 92117.0 | 81775.0 | 76710.0 |
| 2 | 1254.0 | 558.0 | 341.0 |
| 3 | 1170.0 | 472.0 | 305.0 |
| 4 | 1155.0 | 519.0 | 301.0 |
| 5 | 1170.0 | 546.0 | 326.0 |
| 6 | 1271.0 | 609.0 | 416.0 |
| 7 | 1125.0 | 459.0 | 299.0 |
| 8 | 1146.0 | 476.0 | 390.0 |
| 9 | 1274.0 | 446.0 | 301.0 |
| 10 | 1135.0 | 451.0 | 297.0 |
We avoid re-computing the regex string here, and we
also avoid re-compiling the regex itself.
I decided to put the "one_time" decorator in the
bugdown file itself, just to reduce friction in
folks reading the "buyer beware" comments.
Unfortunately, we can't use this for the
get_web_link_regex() function due to testing concerns,
so that continues to do an inelegant cache-with-global-var
scheme.
We use early-exit to flatten the code.
I also tweaked the comments a bit based on some recent
profile findings. (e.g. reading the file isn't actually
a big bottleneck, it's more the regex itself)
Instead of custom stubs, we now use zjquery. We also
limit a couple checks to the first call to
`show_empty_narrow_messages`, since it's the same
logic every time.
When you hover over a user that has set a user
status, we now show something like "out to lunch."
You can test this in the console by doing:
user_status.server_update({status_text: 'out to lunch'})
And then hover over your name in the buddy list.
The stubs here were kind of unnecessary, as the
real people module is lightweight and data setup
is pretty easy.
In passing I also removed the unnecessary `sed`
abbreviation.
The name `insert_user_into_list` is sort of misleading,
since we are often just redrawing the user's existing
item in the buddy list.
I chose `redraw_user` over `update_user` to emphasize
that we're just going to redraw it with whatever data
has been updated by the callers.
This fixes an annoying bug where clicking to subscribe to a stream
would change the color shown in the "manage streams" UI immediately
after you click.
Fixes#11072.
This adds a setting under "Notification" section of
"Organization settings" tab, which enables Organization administrator to
control whether the missed message emails include the message content or
not.
Fixes: #11123.
The commit f863a9b567 had modified
jquery.filedrop's paste method to exit early if any of the items in the
clipboardData is of the string kind. The early exit was added to prevent pasting
an image thumbnail for text copied from software like MS Word, instead of
pasting the actual copied text content. When copying an image in a (modern?)
Browser, though, the clipboard seems to contain a html `img` tag item, along
with the actual image file. This resulted in pastes being broken.
This commit modifies the condition checked for the early exit. We now actually
look at the html content in the clipboard to see if it is an `img` tag, in which
case we upload the image, instead of exiting early.
Closes#7130.
This is likely not the "right" fix in that it involved a negative
margin, but this does eliminate an annoying visual glitch where the
scrollbar overflows above its container in the left sidebar, without
creating other apparent problems.
Fixes#8731.
User was able to click delete button multiple time which could cause
multiple delete requests. This commit disables and hides the delete
message button after the first click and shows a spinner until http
the delete request responds.
Also adds a casperjs test to ensure that spinner becomes visible and
delete button becomes invisible after clicking on delete button for
first time and hides spinner and show delete buttton when message is
deleted.
Fixes: #11219.
Multiple delete message requests for the same message sometimes caused
a 500 error. This happened via the normal IntegrityError being thrown
by delete message/archiving code.
This was manually reproduced by adding latency in function
move_messages_to_archive() in retention.py and
delete_message_backend() in views.py. This addresses the problem by
adding code to handle the exception and throw JsonableError to convert
500 to 400 errors, with an automated test.
This fixes a section of code that hasn't really
been turned on yet. We decided to rename
"info" to "status_text", and I apparently missed
this. We don't have any UI to set these yet,
so it was a harmless bug.
I'll try to get some better test coverage on this
when I tweak the buddy list to show user status.
Apparently, Clubhouse has two different payloads for story label
changes, one where the label name lives inside the "references"
object, and the other where it lives inside the "actions" object.
Their webhook API is still in beta, so this could just be a bug.
Anyhow, we should support both.
This commit takes away the ability for non-admin members to create
streams where only admins can post messages by hiding the option from
them.
Fixes#11290.
This a check on server side to verify whether the user sending request
to create stream where only admins can post is an admin or not; Raises
a JsonableError when the user is not the realm admin.
You can now pass in an info field with a value
like "out to lunch" to the /users/me/status,
and the server will include that in its outbound
events.
The semantics here are that both "away" and
"status_text" have to have defined values in order
to cause changes. You can omit the keys or
pass in None when values don't change.
The way you clear info is to pass the empty
string.
We also change page_params to have a dictionary
called "user_status" instead of a set of user
ids. This requires a few small changes on the
frontend. (We will add "status_text" support in
subsequent commits; the changes here just keep
the "away" feature working correctly.)
We now have single function that handle both away
and not-away.
This refactoring sets us up to piggyback "info" more
easily onto status updates.
The only thing that changes here is that we don't
delete database rows any more when users revoke
their away status. Instead we just set the status
to NORMAL.
When I was initially writing the tests to solve issue #10131 in PR
2 schema checkers as I modified the code to send the rendered_value
only when required.
When I was using just 1 schema checker shared between two code paths,
we needed _allow_only_listed_keys. But after shifting to 2 schema
checkers for the two different cases, we no longer needed that flag,
and it's better to remove it for a stronger check.
We had a bug where if you started typing a message
and then used quote/reply (after the fact), we
would overwrite the user's original message.
The bug was kind of subtle--the internal call
to "respond" to the message would select the message
text, and then `smart_insert` would replace the
selection, unless it was Firefox.
Note that we now also allow you to cross-post
replies, which is a plausible scenario, although
possibly unintentional at times, too. I'm erring
on the side of giving the user control here, but
I'll add a warning in the next commit. Our compose
fade feature should also prevent unintentional
mixes here, too.
We often need to go to the server to get raw content.
The exceptions are messages for which we've already
fetched the raw content for some other reason (maybe
a previous quote-and-reply) or which are locally echoed.
Whether we can get the raw content locally or from
the server, the replace_content() logic is the same.
NOTE: If you revert this commit, you want to revert
the immediately prior commit as well. The history
is that Ishan made some improvements to the widget,
but there were some minor bugs. I decided not
to squash the commits together so that the git
history is clear who did what. (In particular, I
want questions about the JS code to come to me if
somebody does `git blame`.)
Anyway...
This is a fairly significant rewrite of the polling
widget, where I clean up the overall structure of
the code (including things from before the prior
fix) and try to polish the prior commit a bit as
well.
There are a few new features:
* We tell "other" users to wait for the poll
to start (if there's no question yet).
* We tip the author to say "/poll foo" (as
needed).
* We add edit controls for the question.
* We don't allow new choices until there's
a question.
This also fixes few unusual UI issues like an invitation got failed when
certain emails can't be invited then the error box is left with "warning"
even when next request got succeed and another case when invitation got
succeed after failing it's still reported with "alert-error" class alert
banner.
It's no longer used, as can be seen in
2d52463b61, in past we use `type` for
specifiying whether status is 'subscriptions-status' or else, which isn't
used now, hence `type` is removed here.
This reverts the temporary fix done in commit
46f4e58782 and replaced it with the fix that
non-admins should be able to see a dropdown to select a non-admin type of
invited user i.e. normal member or guest user.
This commit fixes a bug that caused:
1: A valid full name on an onboarding form to be cleared after an
invalid submission.
2: Incorrectly cleared name populated from LDAP which was janky from
UX perspective.
Ideally we should disable name change for LDAP as next login
will overwrite any changes but I think that can be done in a
separate PR.
Fixes: #10867.
`fakeldap` assumes every attribute to be a multi-value attribute
while making comparison in `_comapare_s()` and so while making
comparisons for password it gives a false positive. The result
of this was that it was possible to login in the dev environment
using LDAP using a substring of the password. For example, if the
LDAP password is `ldapuser1` even entering `u` would log you in.
On the backend, we extend the BlockQuoteProcessor's clean function that
just removes '>' from the start of each line to convert each mention to
have the silent mention syntax, before UserMentionPattern is invoked.
The frontend, however, has an edge case where if you are mentioned in
some message and you quote it while having mentioned yourself above
the quoted message, you wouldn't see the red highlight till we get the
final rendered message from the backend.
This is such a subtle glitch that it's likely not worth worrying about.
Fixes#8025.
These mentions look like regular mentions except they do not
trigger any notification for the person mentioned. These are
primarily to be used when you make a bot take an action and
the bot mentions you, or when you quote a message that mentions
you.
Fixes#11221.
Apparently, zoom's API will (sometimes?) return a 201 (not 200)
created in response to the API request to create a call. We fix this
by using the proper requests check for whether or not the request
failed.
Users can preview their profile from user settings. If user
open preview profile modal from user settings, then closing
preview profile modal should redirect them to settings modal
again (since probably they want to keep editing).
This commit fixes above issue.
This commit fixes an error in the logic for allowing admins to edit any
user's CPF (custom profile field) values. The logic allowing users to
edit their own CPF values is however sound. What happens is that of all
the CPF types, for "choice fields" as well as "URL" and "date fields",
when the value is reset/deleted/cleared by the admin in the Admin UI
(organization settings), the frontend would send a null (empty string)
value to the backend for that custom profile field (as this is, after
all, the new value in this case). This would then triggers the backend
validators to return an error message.
We fix this by using the method check_remove_custom_profile_field_value,
that both code paths (user editing their own CPFs and admin editing a
user's CPF) can call.
This moves the logic for deleting the user's custom profile field
value in the remove_user_custom_profile_data view function to a method
named check_remove_user_custom_profile_value in actions.py, so that we
can reuse it in the next commit.
This adds the same style of "Saving"/"Saved" loading spinners we use
elsewhere in our settings.
Tweaked significantly by tabbott to fix issues with the notifications
being on the wrong screen for reactiving/deactivating users; this was
done by introducing the get_status_field helper function and using it
everywhere.
The legacy "Updated Successfully" message shown after saving changes,
is removed, and replaced with our standard "Saving" spinner and
animation.
Fixes: #11177.
This code will correctly render emoji to the message textarea based on
whether emoji was selected from message composition form or message
edit form.
Fixes part of #11188.
AFAIK I should this never fail, hence the blueslip.error line. But it
is failing in practice when rendering user groups after looking them
up by ID, and the error handling should definitely be softer.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.
Fixes: #10900.
Our recent work on inviting users as guests accidentally set the
invite_as argument in a way that would fail for non-admin users.
Fixes#11283, fixes#11255.
We make this change because setting up reminders in PM's didn't
play really well with our current infrastructure. Basically the
reminder messages from the bot can't appear in the same narrow as
that of a PM between two people and therefore we disable it.
Though we make an exception here where a person wants to set up
reminder for himself.
We do this because now we send a message to stream if a reminder
is set and won't need the notification above the compose box saying
that we set a reminder. We would still need that notifications for
the send later feature so we make the construct conditional.
We do this since we are yet to figure out how the entire realm
internal bots scenerio should work and therefore for the timming
we will use notification bot to deliver the reminders.
Previously, the subscription color attribute had a validator of
check_string, but this is insufficient. Hence this commit update the
validator used to check_color. Fixes#11268.
For unsupported or invalid payloads, we should just raise the
UnexpectedWebhookEventType exception and let our logging system
take care of recording the payload that caused the error.
This commit improves a couple of things:
* All of the message templates are now at the top, a convention
we follow in a lot of our webhooks.
* Messages are not prefixed with any emojis. We don't do this in
any of our other webhooks. Plus, the emojis were outdated.
* Remove some superfluous code.
* Use ```quote <quote goes here> ``` style formatting for
quoted text instead of the `>` character.
This commit only adds support for the four events that have sample
payloads provided for them on the Pagerduty developer website.
Support for the remaining events will be added in subsequent
commits, as we get access to more sample payloads.
Apparently, the missed-message templates have a slightly different
structure from our other email templates, which triggered a latent,
subtle bug in inline-email-css's effort to remove duplicate <html>
blocks from emails that had been generated by premailer. Fix this
bug, and add appropriate assertions to prevent similar issues in the
future.
Fixes#11249.
Previously, zerver.views.registration.confirmation_key was only
available in development; now we make that more structurally clear by
moving it to the special zerver/views/development directory.
Fixes#11256.
Some urls are only available in the development environment
(dev_urls.py); Corresponding views (here email_log.py) is moved to the
new directory zerver/views/development.
Fixes#11256.
This is a major upgrade, and requires some significant compatibility
work:
* Migrating the pattern-removal logic to use the Registry feature.
* Handling the removal of positional arguments in markdown extensions.
* Handling the removal of safe mode.
This adds language paramater to send_future_email. As a result, this
properly internationalizes invitation reminder emails, by passing
correct language into send_future_email.
Fixes#11240.
The site_packages variable points to (e.g.)
zulip-py3-venv/lib/python3.4/site-packages. If that doesn’t exist,
we’re probably running the wrong Python version.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Now, if you pass an api_key, we'll initialize the public room
subscribers to be whatever they were at the time the import happened.
Also, document the situation on the caveats section.
We had an inconsistent behavior when `LDAP_APPEND_DOMAIN` was set
in that we allowed user to enter username instead of his email in
the auth form but later the workflow failed due to a small bug.
Fixes: #10917.
This now sets the user-agent to something like:
ZulipOutgoingWebhook/2.0
(It uses the current ZULIP_VERSION.)
Before this change, the user-agent would be
something like `python-requests/2.18.4`.
Fixes#10741
Closes#11195. We add a management command to allow us to send emails
to the email mirror directly. The command doesn't require any
configuring of email sending or receiving for the email mirror,
it passes the emails directly using the process_message function.
In between releases, the following commit introduced
a bug where we agressively scroll to the top every
place we call `ui.update_scrollbar`:
092b73d0b7
The main symptoms were that the left and right sidebars
would go to the top for things like selecting a topic,
getting activity updates from the server, and resizing
the window. It was very jarring.
The recent commit looked innocuous--the root of the problem
was the original API expressed an intent to scroll to the
top, but didn't actually do it, so it was a bug in hiding.
There are **some** occasions where it's actually appropriate
to scroll to the top, mostly around search filtering, and
in those places we now call the new `ui.reset_scrollbar`
function.
This is a bit of an emergency fix, so particularly with
the settings stuff, we may get more reports of glitches here.
The important thing here is that you almost never want to
reset the scrollTop for sidebars.
We need to explicitly check for empty recipient lists in
send_message to ensure that internal_send_huddle_message doesn't
call Addressee.for_private with an empty recipient list.
The check for empty recipient lists (the "message_to" argument)
does not belong in check_message. As we implement support for
sending messages by user IDs (see #9474), we will be extending
much of the existing code in Addressee.legacy_build that validates
recipient lists. Therefore, Addressee.legacy_build is a much more
apt area to check for empty recipient lists.
Also, Addressee.for_private and Addressee.for_user_ids also need
to do their own validation, since not everything goes through
Addressee.legacy_build. It is okay to simply throw a 500 in these
cases because we expect that callers will be doing their own
validation for calls that don't go through Addressee.legacy_build().
This commit is a part of our efforts surrounding #9474.
Update the list of ciphers that nginx will use to the current
Mozilla recommended ones.
These are Intermediate compatibility ones suitable for clients
running anything newer than Firefox 1, Chrome 1, IE 7, Opera 5
and Safari 1. Modern compatibility is not suitable as it excludes
Andriod 4 which is still seen on ~1% of traffic.
More info: https://wiki.mozilla.org/Security/Server_Side_TLS
This seems like a small change (apart from all the
test changes), but it fundamentally changes how
the app finds "topic" on message objects. Now
all code that used to set "subject" now sets "topic"
on message-like objects. We convert incoming messages
to have topic, and we write to "topic" all the way up
to hitting the server (which now accepts "topic" on
incoming endpoints).
We fall back to subject as needed, but the code will
emit a warning that should be heeded--the "subject"
field is prone to becoming stale for things like
topic changes.
Feature of sending notification to the stream using notification bot
is added. user_profile is also passed to do_rename_stream for using
the name of user who renamed the stream in notification.
Notification is sent to the stream using
internal_send_stream_message in do_rename_stream.
Fixes#11034.
We recently added a feature to warn users that they
may need to scroll down to view messages that they
just sent, but it was broken due to various complexities
in the rendering code path.
Now we compute it a bit more rigorously.
It requires us to pass some info about rendering up
and down the stack, which is why it's kind of a long
commit, but the bulk of the logic is in these JS files:
* message_list_view.js
* notifications.js
I choose to pass structs around instead of booleans,
because I anticipate we may eventually add more metadata
about rendering to it, plus bools are just kinda brittle.
(The exceptions are that `_maybe_autoscroll`, which
is at the bottom of the stack, just passes back a simple
boolean, and `notify_local_mixes`, also at the bottom
of the stack, just accepts a simple boolean.)
This errs on the side of warning the user, even if the
new message is partially visible.
Fixes#11138
We now have two functions:
add_new_messages
add_old_messages
This is a lot easier on the eyes, and it will also
prevent us from exceeding line length in future commits.
We also remove an unneeded stub in the narrow_activate
tests.
This commit makes it a bit more explicit about
why we're updating 2 or 3 message lists every time.
It looks funny now to repeat the home-list updates
in both sides of the conditional, but this will be
more obvious in a subsequent commit, where we want
to capture return values from rendering.
In a recent commit we allowed for `scroll_amount`
to be zero (as an indirect consequence of letting
`scroll_limit` be zero without early exiting).
See 0f75be3e8e
We want to short circuit the call to
`system_initiated_animate_scroll`, partly to save
unnecessary computation, but in particular to avoid
invoking the suppress-pointer-update logic.
It's convenient to have visible_bottom as well
as top/height, and the extra computation is
trivial (it's just arithmetic, no extra jQuery
involved).
There's some minor cleanup here too.
Previously, this wasn't an explicit feature of the export tool.
Note that the current version still includes metadata on private
streams and private message recipients, just not their messages.
This adds a proper template for the /digest page, making it a
reasonable way to view the digest email content for development and
debugging.
Fixes: #11016.
Previously, we had some hand-written logic for parsing the subject
line of the email's headers and turning it into a Python string using
each of the valid encodings for an email. That logic was buggy, and
sometimes resulted in a bytes object being passed into the
`send_zulip`, which would eventually throw an exception.
The fix for this is to use the Python standard library make_header
method for handling internationalized email headers.
https://stackoverflow.com/questions/7331351/python-email-header-decoding-utf-8
Since the Zulip API runs on both developement and production
servers, it is misleading to mention "dev servers" when discussing
zuliprc files.
Also, note that it is better to manually edit all of our JS
examples than to implement macro-like functionality that we use
for our Python examples. For our current purposes, it would be
too much work to build a full-blown testing framework for our
JS code examples just so that we can fix a minor wording issue.
Fixes#10672.
Guest users can not add subscribers to subscribed or unsubscribed
streams. Therefore hide add-subs html element if current user
is guest user.
Tweaked by tabbott to use the early-return pattern.
Add explanation in popover on disabled add-subscriptions input elements,
admin can't add subscribers to non subscribed private streams, only
subscribed users can.
Fixes#10593
Fixes part 1 of #10612. We use a regex to remove RE:, FWD: (and similar
variations) from email subjects. Unit test is included, we add
subjects.json in fixtures containing various subjects to try the
stripping on.
There's a subtle change here in how we handle the
hypothetical case that the selected message is above
the top of the feed. Instead of early-exiting
from _maybe_autoscroll(), we just treat the limit
as zero, which will have the same effect.
We also change a var name be just `scroll_limit`
instead of `available_space_for_scroll`. A longer
name would be valuable if it were somehow more
specific, but it was needlessly verbose.
The idea is to use this field for storing the best matching alias
to be displayed in search results. In subsequent commits I will
replace the search and rendering logic to use this field instead
of creating new objects on each search.
Since we have already added the `invite_as` field to models, we can now
replace usage of `invite_as_admin` properly with its equivalent `invite_as
== PreregistrationUser.INVITE_AS['REALM_ADMIN']`.
Hence, also removed now redundant `invite_as`.
This should make it easily for mobile/terminal apps to handle
situations like the user's API key changing.
Also fix the fact we were incorrectly using a 400, not 401, status
code for this case.
It hasn't been working for years, but more importantly, it spams up
root's mail queue so that one can't find important things in there
(e.g. the fact that the long-term-idle cron job was failing).
Refactoring in 4e1c058 was not correct since recipient_block
and message_content checked for if not condition while
recipient_header checked for if.
The naming of classes in 6077a33 was also not correct
semantically.
It looked like I got some hex values wrong during the
hsl to hex conversions. I used the built in vscode color
converter for this so not really sure how this happened.
The colors are now equalent to the original hex colors
before the conversion to hsl was made in
5869cc3b6d
The logic for flushing the API key has been broken every since we
added the cache, since we were incorrectly flushing the new API key,
not the old API key, from the cache after regeneration.
This should eliminate the need to do manual analytics work when
importing organizations imported/exported using the zulip -> zulip
import/export tools.
This adds reference for reloading nginx when the certificates are
replaced so that the server works with the new certificates instead of
the old ones.
Fixes: #4849.
This couples the behavior of Enter to Save with Enter to Send.
Extracts and exports functions responsible for checking if enter
does send/edit and handling default behavior otherwise.
Changes made in static/js/message_edit.js and
static/js/composebox_typeahead.js
Fixes#10320.
This adds two functions to static/js/composebox_typeahead.js.
should_enter_send(event) checks if enter should actually send.
handle_enter(textarea,event) emulates browser's default behavior
if enter doesn't send.
These two are just straight extractions of portions of the compose box
logic.
The slim_mode setting had been incorrectly configured to skip
"deleted" users, resulting in bugs where private messages with deleted
users would not be imported.
It's a bit dangerous for the user to hit escape
to close the feedback widget, since it can
disappear suddenly, but users will try it, and
we should just close the widget.
(Hitting escape should be a noop if the box
is closed, but now it goes to "All Messages".)
We only need these once, not during every show()
call. We actually were only setting up the
click handlers one time, but we had redundant
mouse handlers.
More importantly, we stop a runaway timer
that tries to fade out our feedback widget
every 100ms or ten times per second!
A few things are still hard coded, but the class exposes
show() and dismiss() now.
The show() method is configured with callbacks for
populate() and on_undo().
This endpoint serves requests which might originate from an image
preview link which had an http url and the message holding the image
link was rendered before we introduced thumbnailing. In that case
we would have used a camo proxy to proxy http content over https and
avoid mix content warnings.
In near future, we plan to drop use of camo and just rely on thumbor
to serve such images. This endpoint helps maintain backward
compatibility for links which were already rendered.
This setting splits away part of responsibility from THUMBOR_URL.
Now on, this setting will be responsible for controlling whether
we thumbnail images or not by asking bugdown to render image links
to hit our /thumbnail endpoint. This is irrespective of what
THUMBOR_URL is set to though ideally THUMBOR_URL should be set
to point to a running thumbor instance.
We used to add sharpen filter for all the image sizes whereas it was
intended for resized images only which would have been smoothened
out a bit by the resize operation.
This unnecessary use of the filter used to result in weird issues
with full size images.
For example: Image located at this url:-
http://arqex.com/wp-content/uploads/2015/02/trees.png
When rendered in full size would have just boundaries visible.
As part of this change, we port into the .messages class the work in
4e8e7348da to change overflow-y to auto,
not scroll (skipping that would result in a regression).
We calculate a few values higher up in the function.
This reduces a bit of code duplication and removes
a somewhat janky expression that happens against
a mutated list.
I use shorter var names in frb_botom() and do early
return in obscured_recipient_bar().
Also, we always call exports.hide() if we don't find
a valid recipient bar to "float."
There is no reason not to initialize people.js
super early in the process, as it only reads
data from page_params. Also, other modules
are likely to want its data during their own
initialization steps.
This is the preferred way to check that a user
id belongs to the current user.
We have a recent bug where the current user's
circle doesn't turn green right away. It's not
clear this is the fix, though. (It's hard to
repro locally.)
When trying to find the email gateway address, use the
`email.util.getaddresses` function to deal with cases
where multiple recipients are included in the email header
or the stream address appears as an angle-addr with a
name given (e.g. if someone added it to their address book).
Added some other headers where the required address may
appear: "Resent" headers are sometimes used for forwarding,
and streams may also be found in CC. There is no way to find
the address if the email was recieved as a BCC.
This should hopefully be the last commit of this form; ultimately, my
hope is that we'll be able to refactor the semi-duplicated logic in
this file to avoid so much effort going into keeping this correct.
Apparently, our thumbor logic assumed S3_REGION would be present in
prod_settings.py. While it is in newer Zulip releases, upgraded
servers won't have a copy of this field.
This now checks if the user is zulip, and if not, switches to the
zulip user, making it possible to run it as root.
Significantly modified by tabbott to not break existing behavior.
This is mostly for testing purposes. The code
structure here is pretty stable--we will probably
always use level() here to either sort or
group users, and being able to test it directly
is nice, rather than bringing in all the other
machinery.
For many actions, we make a single call to
send_event, and it's kind of heavy now to
properly assert we made one call, and we
don't need to exercise all the tornado code
to prove that the action was written correctly.
This makes it possible it include our standard markdown formatting in
one's custom profile fields, allowing for links, emphasis, emoji, etc.
Fixes#10131.
While we're at it, we remove the JSON parsing that was part of the
user field code path, since this function isn't responsible for
rendering user fields.
Apparently, our custom profile fields feature was parsing the "user
list" field type in multiple places, and ignoring the results in one
of them. That code had been causing some confusion; the correct
solution is to just delete it, since the template rendering process
ignores that value for this field type.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
This adds a model field `invited_as` to PreregistrationUser model class
which will replace `invited_as_admin` in future. Intentions behind adding
this are that we can specify "types" of users for an invited person other
than admin or regular that is, guest user or maybe many others in the
future.
The octet-stream content type is potentially under-specified, but it's
better than potentially submitting None and increases consistency of
this part of the codebase.
The boto library's s3 interface allows setting only string-format
metadata keys. So we need to cast the last_modified floating-point
timestamp into a string before storing on the S3 object.
This bug mostly broke uploading avatars when using the S3 storage backend.
Apparently, hc-migrate can generate emoticons.json files with a
somewhat different format. Assuming that other files are in the
normal format, we should be able to handle it like this.
See report in #11135.
Apparently, some methods of exporting from HipChat do not include an
emoticons.json file. We could test for this using the
`include_emoticons` field in `metadata.json`, but we currently don't
even bother to read that file. Rather than changing that, we just
print a warning and proceed. This is arguably better anyway, in that
often not having emoticons.json is the result of user error when
exporting, and it's nice to flag that this is happening.
Fixes#11135.
Our logic for doing pageup/pagedown calculation inside compose was
written too tightly, and ended up breaking the keys inside message
editing.
Fix this by using generic selectors that don't hardcode compose.
Previously, messages with more than one line did not parse '/me' at
the beginning of the message. Since there's a reasonable way to
render multi-line messages, this commit adds support for doing so.
This change does potentially break with the expected behavior of other
slash commands, but it seems worth providing useful functionality over
a blind focus on consistency.
Fixes#11025.
If branch for showing the profile details would
not have executed if the subdomain was root ("").
The check was changed to check for select input
instead of checking for subdomain.
This avoids a bunch of potential confusion around users trying to
interact with these UI in situations that don't make sense.
(E.g. showing a menu to start editing the message when the menu is
already open).
Fixes#3802.
We instead get the specific fields from message
that we use. This is particularly helpful
for subject -> topic migration; we no longer
have to account for "subject" fields in
client-side templates.
This IntegrityError has been happening occasionally in production due
to races, likely due to some sort of mobile app double-post bug.
Handle this by avoiding a 500, and returning the same 400 we would do
if there hadn't been a race.
This continues the effort to isolate "subject" references
to util calls.
Also, we fix a comment.
Finally, we use canonicalized operators in a switch
statement.
This adjusts the spacing so that the out-of-view notifications for
group PMs (which have particularly long text) don't end up with the
"x" to close the notification overlapping the text.
Fixes#11058.
This section is largely unnecessary, doesn't convey any useful
information, and is probably a remnant from an older version of
this doc that we forgot to remove.
The crawler used to be called directly for checking external links.
Now the scrapy command calls the crawl_with_status wrapper.
Crawl_with_status has been modified to pass the external parameter in
the previous commit, so we can now use this simpler approach.
This commit adds a custom Markdown include extension which is
identical to the original except when a macro file can't
be found, it raises a custom JsonableError exception, which
we can catch and then trigger an appropriate test failure.
Fixes: #10947
Our HipChat conversion tool didn't properly handle basic avatar
images, resulting in only the medium-size avatar images being imported
properly. This fixes that bug by asking the import tool to do the
thumbnailing for the basic avatar image (from the .original file) as
well as the medium avatar image.
The `assert_message_groups_list_equal` and
`assert_message_list_equal` helpers were
always returning `true`, as they were doing
a bogus traversal of the data structures and
always comparing empty arrays, even when there
was real data.
To prevent this pitfall in the future, we assert
that the extracted data is truth-y, and for the
empty cases we just directly assert deep equality
to `[]`.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.
To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
A lot of the seemingly unrelated test fixture changes are because we're
removing a query to stripe in the upgrade path, in cases when the user's
realm has an existing Customer object.
As part of giving the stream/topic fields in the
compose box longer ids, I broke the autocomplete
code that handles re-focusing the cursor after
a user hits enter. The worst symptom of this was
that we tried to send a message before compose
finished (although it wouldn't fully deliver the
message).
The new code should be a bit easier to grep for
if we rename these fields again, as we explicitly
use selector syntax.
The link is broken, and I'm not sure we want this
paragraph in there, as it's somewhat speculative
(we don't know how extensively Slack uses PHP any
more, nor do we know how much of a factor it is in
any security issues) and is probably mostly
preaching to the choir. The "bounty" link should
suffice.
Note that a pretty common use case for this is a realm admin sending this to
everyone after an import from HipChat or Slack. So this adds the realm_name
to the title (so that there is something they might recognize) and kept the
wording generic enough to accommodate the user not having clicked anything
to get this email.
Also strengthens the tests a bit to better test the complicated template
logic.
This is somewhat hacky, in that in order to do what we're doing, we
need to parse the HTML of the rendered page to extract the first
paragraph to include in the open graph description field. But
BeautifulSoup does a good job of it.
This carries a nontrivial performance penalty for loading these pages,
but overall /help/ is a low-traffic site compared to the main app, so
it doesn't matter much.
(As a sidenote, it wouldn't be a bad idea to cache this stuff).
There's lots of things we can improve in this, largely through editing
the articles, but we can deal with that over time.
Thanks to Rishi for writing all the tests.
This change lets us eliminate the need for new authentication backends
to edit get_auth_backends_data, since we're just computing it from the
official registry in zproject/backends.py. Should save a few lines of
work whenever we add a new auth backend, and make that more accessible
to new contributors.
Previously, we were searching the whole message_row object for emoji,
mentions, etc., which has a bunch of UI elements that can't contain
the syntax we want to modify. This should be a slight improvement in
the performance of message post-processing, which runs a lot of times
and thus is fairly important.
This check caused us to only run the code inside that block if the
message mentioned the current user (since that's when the `mention`
class is added to the main message row).
While this was a useful performance optimization, it probably was a
small one, not worth it for the correctness cost.
This adds a new realm_logo field, which is a horizontal-format logo to
be displayed in the top-left corner of the webapp, and any other
places where we might want a wide-format branding of the organization.
Tweaked significantly by tabbott to rebase, fix styling, etc.
Fixing the styling of this feature's loading indicator caused me to
notice the loading indicator for the realm_icon feature was also ugly,
so I fixed that too.
Fixes#7995.
Logic for checking if the last message in the current table is visible was
already written in message_viewport.js; Code in notifications.js is changed
to reduce redundancy.
We are trying to carve room for a more specific
"user_status" concept, which refers to statuses
that users specifically set, like "I'm away".
So we call this function "update_presence_info",
which reflects that it's more about actual
"presence"--i.e. the user really is present
in the browser, even though the actual human
may not want to be disturbed.
The current user gets excluded from all non-empty
searches, even ones that match the user, since
it can look funny when the user's at the top of a
search, and you'd never need to search for yourself
(again, since the current user is at the top of
the buddy list).
Apparently, we didn't have one of these, and thus had a moderate
number of generally very old violations in the codebase. Fix this and
clear the ones that exist..
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).
This discussion was split from #11004.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Support for extended mention syntax was added as a part of
commit fbe99b812ee8fbca7257a5b7156c57a6cd74195b in the
python-zulip-api repository. The relevant function,
extract_query_without_mention now relies on the client's ID
in order to check for the extended syntax. Since the
EmbeddedBotHandler has no user_id attribute, the latest
python-zulip-api release broke a test in the main repo.
This is a simple change to our validation, to allow multiple copies of
the main linkifier syntax, which lets us support things like generic
GitHub URLs.
Fixes#10914.
This is a common bug that users might be tempated to introduce.
And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
We move all of its logic into settings_sections.
Note that this is slightly more than a refactor.
We are slightly more aggressive about resetting
sections. For example, if you go into Settings,
then exit the overlay, then go into Manage
Organization, we will now reset sections for both
groups.
We now rely on set_up() methods to call their
own module-specific versions of maybe_disable_widgets()
in the codepath for admin_sections.load_admin_section().
And then for live updates, we just explicitly call
all four modules that support maybe_disable_widgets().
This should make switching between sections slightly faster,
and it also reduces the risk of module A messing with
module B's state. (Granted, we have lots of other ways
that modules can mess with each other's state.)
Apparently, when we renamed these files to no longer have a .txt
extension, we accidentally removed them from the set of strings for
translation, because `manage.py makemessages` by default only
processes .txt and .html files under the templates/ directory.
Fix this by adding a .txt extension.
It appears that our i18n logic was only using the recipient's language
for logged-in emails, so even properly tagged for translation and
translated emails for functions like "Find my team" and "password
reset" were being always sent in English.
With great work by Vishnu Ks on the tests and the to_emails code path.
The previous version was also doing almost the same thing.
But checking for DEVELOPMENT_LOG_EMAILS would allow us
to control the call of send_email by altering the value
of DEVELOPMENT_LOG_EMAILS in tests.
Bootstrap's typeahead is the main part of the project that we've
forked, and moving it to its own module should help unlock our ability
to upgrade bootstrap itself.
This form isn't actively used, which is how it ended up broken, but it
basically didn't display its content properly at all.
Convert it to use our standard white-box framework.
This still doesn't look great in various ways, but it's at least not
obviously totally busted now.
To support this, we add a pass_targets option to the main linter
library, because with current mypy, it's generally counterproductive
to pass the list of files in (can produce spurious errors; isn't
faster).
The testing section is more appropriate, since it's fundamentally part
of our CI system.
While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
The previous logic for soft deactivation ended up doing a giant
transaction in the case that there were thousands of users to
deactivate; this was messy and potentially buggy.
The batched transactions were useful for RealmAuditLog management,
however. So the right solution is to do reasonably sized batches
(e.g. 100 users).
Apparently, our do_batch_update method (used, e.g., in a pgroonga
migration) was using semi-invalid syntax that was removed in postgres
10.
Thanks to Ilya Evseev for the report.
Fixes#11063.
Since we're adding this to a large number of portico pages, there's no
guarantee that these pages actually have a CSRF input.
Though given that the logout template contains a CSRF input,
realistically it should always be present.
We weren't using the actual sorted data to find
the last element. This probably worked by accident
in some cases, but this commit makes it more
deterministic.
We want the return value from this for the next
commit, so we no longer call `format_drafts` indirectly
from `populate_and_fill`, and we rename the latter
to `render_widgets`.
We had an anonymous callback for drafts that was
hard to read. It's much easier to flatten the code,
give functions actual names, and stub them as needed
in the unit tests.
This should make it possible for blueslip error reports to be sent on
our logged-out portico pages, which should in turn make it possible to
debug any such issues as they occur.
Since we're adding this to a large number of portico pages, there's no
guarantee that these pages actually have a CSRF input.
Though given that the logout template contains a CSRF input,
realistically it should always be present.
If a user has an old mention and has since been renamed, there's
really nothing for us to do to render it; we should just return as
though we have no data.
Fix an issue that when a message is being edited, sometimes compose
options are hidden if there is no time limit. Also, move the options
further from the time limit to make them more noticeable.
Fix#11056.
This changes a few things:
* Deplicates deps_to_install logic.
* Has a retry flag, under which we can guard the apt retry print statements.
* Makes the install_system_deps flow more parallel.
The `replying_to_message` field was used in some
early versions of compose fade, but it has no more
use in the current code.
The drafts implementation didn't really make any sense,
anyway, as we were claiming to reply to the same
message we were drafting.
A common source of confusion for new users is sending a message when
you're scrolled up in the message feed; in this case, it's nice to
communicate to the user why the message is not in view.
Fixes#10792.
Restructured by tabbott to replace overly complex logic for getting
the position of the new message with a `message_list.get_row()` call.
Now, we correctly avoid calling various password quality/strength
functions in the registration flow in the event that there isn't a
password form on the current page.
Before, some code wasn't inside a block at all, while other code was
using an incorrect check (an empty jQuery object is not falsey).
The overall result was that this would often crash on certain
pages/flows, stopping JS execution and causing various secondary
problems.
The first bug fixed here has been around for a long
time--we were redundantly updating unread counts
indirectly via muting_ui.initialize(). The
unread counts also get updated in
unread_ui.initialize(), when we have more valid
state. (And it's worth noting here that the unread
counts get updated yet again once message fetches
complete.)
The second bug was a very recent regression from
my recent stream name -> stream id cleanup in the
muting system. We now depend on stream_data to
initialize muting data, so we need to initialize
muting.js slightly later in the process.
These fixes are intertwined, because they were both
somewhat caused by the anti-pattern of having
muting_ui.js initialize unread_ui.js and muting.js,
instead of doing more direct, fine-grained initialization
from ui_init.js.
Essentially we replace this code:
exports.update_muted_topics = function (muted_topics) {
muting.set_muted_topics(muted_topics);
unread_ui.update_unread_counts();
};
with this:
exports.initialize = function () {
exports.set_muted_topics(page_params.muted_topics);
};
And the modules load like this:
stream_data
...
muting
...
unread_ui
And we don't need any page-load initialization for muting_ui,
which is mostly used for Settings/Muted topics.
This function used to be called initialize_from_page_params(),
and we called it indirectly through `subs.js`.
Now we call it directly from `ui_init.js`, which gives us a
bit more control over how things are initialized. In fact,
this sets us up for the next commit, where I fix a recent
regression I introduced.
This is mostly about cleaning up the naming convention
for streams and topics, but it also adds a test that
specifically tests the muted-topic case (without any
other factors that would prevent a notification).
Before this commit, it was possible to change the
API for muting topics and get false positives, even
when the test setup was clearly broken.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.
This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.
This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.
Tweaked by tabbott to fix a few small issues.
Fixes#10895.
While reviewing #11012, I discovered a nondeterministic result for
test_signup, which I tracked down to specifically this triple of tests
failing when run in this order:
test-backend GCMSuccessTest \
zerver.tests.test_push_notifications.TestAPNs.test_get_apns_client \
zerver.tests.test_signup.LoginTest.test_register
with a query count mismatch like this:
expected length: 73
actual length: 79
Comparing the list of queries, it's clear that test_register was
seeing `push_notifications_enabled()` returning True in this test order.
It's not clear why GCMSuccessTest was required here (it was!), but
further debugging determined the problem was that
`test_get_apns_client` left the _apns_client initialization system in
a state where get_apns_client would return a non-None value, resulting
in push_notifications_enabled() returning True for future tests.
The immediate fix is to just reset the `_apns_client` and
`_apns_client_initializedstate` state properly after the test runs;
but arguably we should do a larger refactor to make this less
fragile.
The data attribute here has some value if you're
inspecting the HTML in the browser, but it's not
worth the extra code.
All the list items have data-stream-id, so there's
no need for the parent to have it.
The stream_list test that was fixed here was sort of
broken. It accomplished the main goal of verifying
what gets rendered, but now the data setup part is
more like the actual app code (and simpler, too).
This fixes the most core data structures inside of
muting.js. We still use stream names for incoming
data to set_muted_topics and outgoing data from
get_muted_topics.
This will make us more resilient to stream name changes.
Before, if you were logged on when a stream rename
occured, topics that were muted under that stream would
appear to be unmuted. (You could fix it with a reload,
but it can be jarring to have a bunch of unread messages
appear in your feed suddenly.)
Fixes#11033
The previous migration code path was broken in two ways:
* ScheduledEmail objects generally contain a `None` value for
whichever of `to_user_id` and `to_email` isn't in use; this could
result in us sending a [None] to send_email(), which doesn't make
sense.
* We were calling handle_send_email_format_changes in the wrong order
with respect to the JSON loading process.
Thanks to Tom Daff for the report!
Our list of allowed characters in realm filter patterns has long been
too string; fix this by extending the pattern.
Also, extend the tests to have examples of actual strings one would
use with the patterns, for clarity.
Fixes#10953, fixes#6835.
The fixture changes are because self.upgrade formerly used to cause a page load
of /billing, which in turn calls Customer.retrieve.
If we ran the full test suite with GENERATE_STRIPE_FIXTURES=True, we would
likely see several more Customer.retrieve.N.json's being deleted. But
keeping them there for now to keep the diff small.
Technically, we will only need to process deactivated users for the
purpose of reactivating them (and can ignore, e.g., name changes).
But it's simplest to just process them unconditionally.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.
This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.
Fixes#286.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.
Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
This makes it possible to add --skip-purge-old-deployments in the
deploy_options section of /etc/zulip/zulip.conf, and control whether
old deployments are purged automatically on a system.
We still need to do https://github.com/zulip/zulip/issues/10534 and
probably also to add these arguments to be directly passed into
upgrade-zulip, but that can wait for future work.
Fixes#10946.
Like the other similar commits, we were doing the same work in all
code paths, just with a much more error-prone approach.
We can also now remove the now-unused finish_initial_narrow function.
Like the other commits in this series, we were already doing this in
all of the callers of load_messages; this centralizes that logic in a
less ad-hoc feeling way.
We no longer use or need the start_initial_narrow function.
Previously, each individual caller of load_messages that passed
num_before > 0 would do its own manual management of fetch_status;
now, we just do it inside load_messages.
Apparently, the older side of the FetchStatus object for home_msg_list
was incorrectly not being maintained. We got away with this, because
the do_backfill code path (which runs after we're done with the
load_more cycle) will correct the error for found_oldest. But we
didn't have proper handling for history_limited here.
When we're doing the load_more frontfill, we were not correctly
declaring that we were in the process of doing a fetch. Because the
next load_more call clears this state anyway, this was generally a
short race, off-screen, but it is still a data flow bug.
See the upcoming commits for a refactor that will eliminate the
possibility of this sort of bug.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).
We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.
Fixes#11014.
When a user's name is edited, currently we still show the old name is
mentions (though clicking on the item does the right thing).
However, at present, it creates a new problem in search results, where
the highlighting is removed by this substitution.
The Slack import process would incorrectly issue
CustomProfileFieldValue entries with a value of "" for users who
didn't have a given CustomProfileField (especially common for the
"skype" and "phone" fields). This had no user-visible effect, but
certainly added some clutter in the database.
We should rate-limit users when our rate limiter deadlocks trying to
increment its count; we also now log at warning level (so it doesn't
send spammy emails) and include details on the user and route was, so
that we can properly investigate whether the rate-limiting on the
route was in error.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.
Fixes#7423.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).
Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).
So we skip running our current rate limiting algorithm for these views.
Now that we've styled this feature properly, this makes it possible to
copy various user-preferences type profile data in production when
making a new account with the same email address as an existing
account.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.
Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.
Fixes#11005.
A bug caused background links to open even when a modal in the user
settings overlay was active in the foreground. This commit fixes this
by disabling mouse events for the background when the modal is active,
and restoring them as soon as the modal starts closing.
Fixes#10654.
This optimizes tools/provision by not running
`tools/update-authors-json --use-fixture` unless either the script
itself or its fixtures file (zerver/tests/fixtures/authors.json) was
changed.
Fixes#10991.
Broaden the type of the AbstractEnum __reduce_ex__ parameter to object; this
matches the parameter type specified in the latest enum.pyi file in typeshed.
Fixes#10996.
This adds a line to static/js/hotkey.js for focusing the "Close"
button. Tweaked by tabbott to make more clear that we don't expect
there to ever be both a close button and a save button, since in that
case this code would be busted.
Fixes: #3830.
The stream/topic edit areas now have these ids:
#stream_message_recipient_stream
#stream_message_recipient_topic
They are pretty verbose, but being able to grep
for these without noise does have some value.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.
Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
notifications settings are requested.
Fixes#8051.
A bunch of details were out of date about the current state of Zulip
and thus what projects make sense; while this isn't likely to be our
final 2019 ideas list, this should help avoid confusion about what's
happening.
Since yarn has a package.json conveniently available, we can parse
that with jq, saving the expensive operation of starting up yarn.
This saves ~300ms in a no-op provision.
This should be a nice performance improvement for browsers that
support it.
We can't yet enabled this in the Zulip on-premise nginx configuration,
because that still has to support Trusty.
The Casper code that I eliminated here seems to be
bogus, in that I don't think it really waited for
all the clicks.
I **think** the intent of the test was to verify that
when you leave settings and go back into it, it remembers
the panel. I was able to verify this manually.
We have an upcoming change that lets us use the
back button after going arrowing through multiple
settings pages.
Without first adding this commit, we would have an
infinite loop when you came back to '#settings' and
then '#settings' would rewrite the url with the current
hash.
Just replacing the browser state allows the browser
to do the right thing.
The history protocol is pretty well supported:
https://caniuse.com/#search=history
We can eliminate the janky `setup_page` methods
and just pass in section from `hashchanged`.
This sets us up to handle browser history more
nicely when you load '#settings' and we could essentially
redirect you to '#settings/your-account' (or similar
things). A future commit will address that.
We also use `launch` as the new entry point, which
is more consistent with other modules.
The prior name of this was a bit inaccurate, as we no
longer ever hide the menu item for non-admins. Also,
it belongs more naturally in `gear_menu.js` at this point.
Also, we remove one call to this, which was in a place
where it was no longer necessary.
We now run the code to disable widgets every time
we reload a section, which was the original intention
of the code, but the call to it only happened when
you first launched the page.
We also continue to run this logic for live updates
of is_admin, although it's worth noting that the
code still only handles the "demotion" case of going
from admin to non-admin. (If somebody makes you an
admin, you continue to need to reload to get
widgets enabled.)
If you go to "Manage streams" in the gear menu,
it's now a direct link to '#streams/subscribed'
instead of just '#streams'. This makes our
back button behavior more predictable.
This ensures the "account settings" UI for managing a user's own email
address uses the delivery email, since that's what users care most about.
Eventually, we'll need to add support for at least viewing both email
addresses in "account settings", but this is the right long-term
behavior.
A key part of this is the new helper, get_user_by_delivery_email. Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.
Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
This new setting is still hidden in the UI when not in the development
environment, because the feature isn't ready for production, but
merging this will help simplify future work on the feature.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
This makes it possible for the Puppet codebase to access the path to
the relevant /home/zulip/deployments type directory that puppet was
run from, which in turn makes it possible to safely call scripts from
here.
Based on work by Rein Zustand.
When the list of recipients of a group private message conversation is
truncated due to being long in the "Private messages" area, we didn't
provide easy access to the full list of people. Fix this by adding a
title attribute for the area.
Fixes#10978.
This correctly shows guest users that they cannot interact with
anything in the "Organization" tab, though we preserve the read-only
access to it so that they can (e.g.) browse the organization's custom
emoji.
We've had a long stream of bugs existed because only one of these two
code paths was tested (usually the local uploads backend). By
deduplicating these functions, we ensure that this category of bugs no
longer happens.
Following my recent refactor, this is just a straightforward merge,
with code for one or the other backend ending up inside an if
statement.
Previously, we were incorrectly importing avatar PNGs to a filename
without the .png extension, resulting in them effectively not being
imported.
This was mitigated by the fact that we imported the originals and ran
the appropriate `ensure_` functions, but still a bug.
Apparently, there was a bug in notify_bot_owner_on_invalid_json, where
we didn't reraise the JsonableError.
We fix this with a refactoring that makes the exception layering
clearer as well.
In a quick scan of today's nginx logs on chat.zulip.org, there
were 20 distinct user-agents that begin with 'ZulipMobile/'.
Here's a representative sampling of them, such that the rest
were all boringly similar to one of these.
First, to make room for these without an excess of copy-paste and
overlong lines, convert this test to a data-oriented style. The
existing, synthetic cases appear in the new data followed by the
seen-in-the-wild cases.
Happily, the code being tested passes all these new cases unchanged.
This release is from 2018-08-22, a little over 100 days ago.
It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null". We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.
The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
This makes it more convenient for developers to set very short values
for this (e.g. 1 minute) for the purposes of testing/debugging; there
aren't obvious problems with letting users set short values for this.
This option causes test-documentation to only verify internal links
that we control and are important to be correct. This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.
Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.
Fixes#10942.
This works by yielding messages sorted based on timestamp. Because
the Slack exports are broken into files by date, it's convenient to do
a 2-layer sorting process, where we open all the files for a given
day, and then sort their messages by timestamp before yielding them.
Fixes#10930.
If a user deletes message between when it triggered a potential push
notification for another user, and when that notification was actually
sent, we'd end up with a situation where the `Message` table didn't
have an entry for the requested message ID.
The clean fix for this is to still throw an exception in the event
that the message doesn't exist at all, but if it exists in
ArchivedMessage, don't throw a user-facing exception.
This seems to happen when Apple is having a partial outage on some of
their APNS shards; it should be treated like other networking errors
connecting to APNS (with an automatic retry).
Previously, Topic editing was offered in the UI even to message
senders and organizations admins only if the message was no more than
one day old. This was correct for the "community topic editing" case,
but not for message senders and organization admins.
While we're at it, this also centralizes some previously haphazard
logic to always call message_edit.is_topic_editable().
Tweaked significantly by tabbott to fix the logic.
Closes#10568.
The goal here was to enforce 100% coverage on
parse_narrow, but the code has an unreachable line
and is overly tolerant of bogus urls. This will
be fixed in the next commit.
Apparently, we have a second code path where we might try to call
send_email library functions on old data, namely in the
queue_processors codebase. So we apply the same migration logic here.
Also adds relevant tests and documentation. We currently
do not narrow to a new topic, and instead just narrow to
the stream. Similarly, we do not narrow to a PM if any of
the recipients are invalid.
We stopped setting this nearly five years ago, as part
of bd9cccffce
The big conditional that I removed here should have
always evaluated to false, as I understand the code.
Presumably either the browsers handle # -> '' redirects
better now, or we address this somewhere else in our
codebase.
We ignore keystrokes like alt-left-arrow and alt-right-arrow,
so that the browser can do back/forward.
We may need to refine the handling of ctrl/alt/shift in the
future, but now we only support single-key operations.
This change removes all the complexity around
get_hash_group(), and we now only go into the
"same overlay" logic within Settings or within
Manage Organization, but not between them.
This means if you're in Settings but hit the back
button to something under "#organization" we now
do "more stuff", since we want to err on the side
of reloading sections, etc.
There's not much flicker in my testing, and
this is not a super common transition, anyway.
This code brings the focus to the first input field with errors rather
than just the first input field present in the form after the sign up
form is rendered again after invalid data is submitted.
Note from tabbott: This still doesn't handle the ToS checkbox being
the source of the error, but that's an independent issue.
Fixes#10869.
Add a lock icon to the right of tabs on which nothing is editable
for normal users. Add lock next to Custom emoji option if only admin
can edit them.
Tweaked by tabbott to use title for the lock icons, rather than
aria-hidden, since they do convey useful information.
Fixes: #10893.
Positioning using flexbox makes life much easier for everyone. With
this change we make positioning of icon relative to the label in the
dropdown menu much easier to do and alter if required. We now no
longer need to fiddle with tedious pixel measurements for placing the
icon in the right place.
As a result of this commit we had to change a click event binding
back to be associated with .dropdown-toggle class rather than being
associated with the h3, i because of the re-arrangement of the
dropdown configs.
The integration categories dropdown was too wide in comparison to
the category lozenges and hence this commit attempts to smooth up
the UI a bit on that front.
Here we just fix the behaviour of angle icon which is present
in the integration categories dropdown. It used to change direction
from down to right only if "All" options from the dropdown was
selected (which is also the initial and default option). This behaviour
was pretty inconsistent and looked odd. Rather than having a direction
changing icon here, it migth be just better to stick with just the
down facing angle arrow. Arrow direction in general represents in
which direction the dropdown is gonna open up (in addition to the
fact that a dropdown exits here).
We make the integration categories dropdown gradually slide down/up
rather than appearing instantenously. I believe this gives a better
look to the dropdown and how it behaves.
We also fiddle a bit with the code relating to angle icon in the
dropdown. Basically though its behaviour was already buggy and
will be addressed in an upcoming commit, we try to maintain whatever
behaviour it had before introduction of the annimation effect.
The issue here was that if we opened up integrations page in
responsive mode (so the integrations category sidebar turns into a
dropdown) and click a few centimeters outside the actual dropdown
or perhaps the dropdown menu when its open, it is possible to toggle
or select a integration category.
What this essentially means is that clicking in blank area outside
visible boundaries of dropdown menu its possible to interact with it.
Fix: We change elements on which the click event is tied to and
adjust a bit of CSS for relevant elements so things look as they
used to but function in correct or better manner.
What is the buggy behaviour?
Before this commit if you were to open the integrations docs page
in a smaller window so that the integrations categories sidebar
changed into a dropdown (so that our page is responsive to
screen size), one would notice that selecting a category from the
dropdown menu didn't make dropdown to auto collapse. This feels very
uncomfortable from users prespective since an ugly dropdown with all
the categories sticking around uncollapsed kind of defeats the purpose
of having a dropdown.
Fix: We make the categories dropdown toggle/auto collapse upon
selection of a category.
Fixes part of #10026.
Adds additional option to typeahead:
`tabOpensEmptyTypeahead`(default: false):
tabOpensEmptyTypeahead overrides helpOnEmptyStrings.
This commit sets helpOnEmptyStrings to false and
tabOpensEmptyTypeahead to true. Now typeahead will
open on an empty string only if Tab has been pressed.
Fixes part of #10026.
NOTE: The Tab key will select option from typeahead if the typeahead
is already open i.e the same behaviour as Enter.
NOTE: This behaviour applies irrespective of search pills are enabled
or not.
f52e9d1 ended up not going far enough. Keeping f52e9d1 in place in case we
ever want to go back to that sort of solution.
Also removes the keep argument from test_billing_quantity_changes_end_to_end,
since that test is actually testing the arguments to
stripe.Subscription.save(), not what is returned by Stripe.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.
We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.
Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.
Fixes#10896.
While this would never happen for a real article, this prevents a 500
in this case for a situation which is definitely user error and should
be a 40x (in this case, 404).
As part of this, we refactor the main view code to do validation in a
single code path, since the semi-duplicated-in-3-places logic was
getting pretty buggy.
This prevents the warning about push notifications not being
registered for from being printed in development environment startup
by default. In development, that's the expected state, and we don't
need to spam up the output with that notice.
Apparently, we were incorrectly expressing the paths in the
caches_in_use data structures for these two cache-cleaning algorithms,
resulting in the default threshhold_days algorithm controlling which
caches could be garbage-collected. While the emoji one was just a
performance optimization for upgrade-zulip-from-git, it was possible
for the main `node_modules` cache in use in production to be GCed,
resulting in LaTeX rendering being broken.
This makes it possible to still run the deliver_scheduled_messages
queue worker, even though we're not creating reminder-bot by default
in new organizations.
Previously, we frequently accessed user_profile.realm from outside the
loops that interact with UserProfile objects. This variable reuse
outside the loop could be confusing and should be a style/lint
violation.
While in this case, the behavior was correct (in that all users in the
loops were within the same realm), extracting a separate `realm`
variable significantly clarifies what's going on here.
Even prior to my recent change in settings_panel_menu.js,
we were assigning window.location.hash a value that doesn't
have a '#' prefix. This probably doesn't matter too much
for the browser, but it does confuse our own checks about
whether we're redundantly updating browser history.
Now we prefix the settings hash with '#' and we encorce
this convention with a blueslip error.
Just calling update_browser_history is sufficient
here, and we end up short-circuiting some code
in hashchanged():
* we don't need to set state.old_hash, because
that's what update_browser_history does
* we bypass the is_overlay_check, which is always
false in this context
If `/etc/zulip/zulip.conf` exists in development, `manage.py` fails
unnecessarily. Instead, check if `deploy_type` is set, as done in
`settings.py`.
Fix#10931.
For stream links inside messages (like "#social") we
now use these functions:
hashchange.go_to_location:
We don't need to set href. Relative paths
are more standard, and the url is already
encoded.
hash_util.by_stream_uri:
This saves a step in building the URL.
We call hashchanged.update_browser_history() when
we switch panels. This API short circuits the
hashchanged callback and avoids code churn.
(We weren't actually double rendering, as the downstream
code does nothing at this point, so this is more
just preventig a pitfall and moving to a consistent
API.)
Before this commit, we would sometimes have
the toggler handle clicking or arrowing to
the All tab, but then also rewrite the hash
which caused us to re-process the event.
Now we only call update_browser_history()
in the callback handler from the toggle widget.
There's a bit of refactoring to make this happen,
but the call stacks end up being this:
call toggler.goto(...)
# callback is dispatched
call subs.switch_stream_tab
actually_filter_streams
update_browser_history
While they can share some code, opening the edit panel
for a stream and clearing the panel are pretty different
actions, so we simplify the API for each thing.
You no longer have to pass in booleans, and for the clear
case, you don't have to pass in a bogus node that just
gets ignored.
This fixes a bug where hitting the "n" hotkey was
causing double work related to the hashchange system.
The code is now organized like this:
do_open_create_stream() does the GUI piece
We call the above directly for hash changes.
For in-app actions, whether clicks or hotkeys,
we call open_create_stream(), which delegates
most of the work to do_open_create_stream() but
also updates the hash.
This diff looks a bit more complicated than it really is.
We had a bug where we'd call subs.change_state for
non-streams-related changes. The bug probably barely
impacted customers, since it's hard to get into that
situation unless you're in "Settings", and then the
code mostly did nothing. There's still a deeper issue
of what we actually do want to for settings changes,
but this fix does not address that.
We invert the conditionals related to internal state
changes, so that we can handle internal state changes.
And we make sure to only call subs.change_state if our
"base" is "streams".
This is mostly extracting the code within the `if`
block, as well as setting `base`, which wasn't used
elsewhere.
Also, the `else` no longer calls `is_overlay_hash`,
which was a redundant check.
The previous implementation used run_parallel incorrectly, passing it
a set of very small jobs (each was to download a single file), which
meant that we'd end up forking once for every file to download.
This correct implementation sends each of N threads 1/N of the files
to download, which is more consistent with the goal of distributing
the download work between N threads.
Reran every test with GENERATE_STRIPE_FIXTURES = True, which also caused a
few fixtures to get updates unrelated to these changes (likely due to API
updates that hadn't been previously applied).
There are (at least) two types of objects that could be sent with a
charge.succeeded event, a Charge (e.g. for credit cards) or a Payment (if
they pay by ACH). We were handling the first but not the second.
This commit also updates the fixture for the existing charge.succeeded event
to the latest API version.
This reverts commit 2fa77d9d54.
Further investigation has determined that this did not fix the
password-reset problem described in the previous commit message;
meanwhile, it causes other problems. We still need to track down the
root cause of the original password-reset bug.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
This isn't super required, in that we add these repositories via
`setup-apt-repo` in any case, but the previous code was wrong and
worth fixing in any case.
Apparently, on Debian stretch, the gnupg package isn't installed by
default, which means that our `apt-key add` commands were failing with
these errors on an ultra-minimal Debian installation:
+ apt-key add ./scripts/setup/packagecloud.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
+ apt-key add ./scripts/setup/pgroonga-debian.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
Fixes#10480.
This is definitely a micro-optimization, but avoiding
creating an extra object speeds up page loads by about
20ms per 1000 streams.
It's slightly sketchy to mutate the value in place, but
the original value never gets used again.
We now let color_data keep its own state for
unused_colors, so that we longer have to pass in
a large list of unused_colors every time we want
to assign a new stream color.
This mostly matters at startup, where we might
be cycling through 5000 streams. We claim all
the unused colors up front.
Each operation now has an upper bound of expensiveness,
where the worst case scenario is basically popping
off the first element of a list of <= 24 colors.
The algorithm is now deterministic, too, to make
it easier to test. It's unclear whether random color
assignment ever had much benefit, and it made unit
testing the algorithm difficult. Now we have 100%
line coverage.
Fixes part of #10902.
When there is some error in connecting to server(more specifically to the
tornado server) the "Unable to connect to Zulip" connection error message
gets cleared as Django server could send the response of "get" request of
old messages and hence get_old_messages_success hides the error message
even though the connection is not properly established.
Fixes: #5599.
This provides a nice user experience for folks where we do know what
their LDAP credentials are.
Though we need to fix#10917 before the content in the email with be
correct.
This is the first step of letting users use Zulip markdown in their
SHORT_TEXT and LONG_TEXT custom profile fields, so that they can
include emphasis, links, etc.
This doesn't include any frontend logic yet, however.
When we started the billing system we started by following conventions used
in the Stripe documentation, but in hindsight it makes more sense to follow
conventions used in the Zulip codebase.
The original code was actually broken, in that it checked the wrong
path, but it didn't matter because it used `ln -nsf`.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This commit fixes bug: When user clicks on remove-user-pill-btn,
it closes the parent modal instead of removing user pill from input.
This happens because button has class `exit` and there is click
event listener on all `exit` class buttons, which closes modal.
Fix this by adding `e.stopPropogation` to remove-user-pill listener.
While we don't actually need another tooltip on /stats right now, this
provides a clear approach for how to do that. We've since added
tooltips in various other parts of the webapp, and that code is pretty
copy-pasteable, so I think it's reasonable to say this closes#4612.
Cleaned up by tabbott to remove a bunch of unnecessary changes.
Recent changes merged in #10877 didn't handle these events
correctly. The linkified_id function breaks for the `discount`
object in the JSON payload. A cursory glance at Stripe's docs
tells me that since a discount is associated with a customer
or a coupon, it makes sense for a `discount` object to not have
an ID that can necessarily be linked to. So, we can just link
to the associated coupon instead.
This commit changes the return type of get_possible_mentions_info to a
list instead of a dict, thus disposing off the hacky logic of storing
users with duplicate full names with name|id keys that made the code
obfuscated.
The other functions continue to use the dicts as before, however, there
are minor variable changes where needed in accordance with the updated
definition of get_possible_mentions_info.
This is required in some AWS regions.
The right long-term fix is to move to boto3 which doesn't have this
problem.
Allows us to downgrade the priority of #9376.
This code prevents the password bar from being incorrectly clear after
the sign up form is rendered again after invalid data is submitted
(generally due to forgetting to agree to ToS).
Fixes#10868.
This used to be a more likely codepath, before we introduced
Customer.has_billing_relationship. It is no longer literally impossible to
hit this race condition, so I'm not deleting the code, but it's unlikely
enough that it's not worth figuring out how to test it.
Already better tested by the upgrade and downgrade tests using mock_stripe.
Note that the line that was removed is actually not possible to reach, since
canceled subscriptions aren't shown on the Customer object.
We've had a few unpleasant bugs with real documentation links being
broken, so we're going to make this more aggressive for now.
I think we instead want a more subtle option for suppressing failures
in some places but not others.
I think this will fix a Casper flake where there was a race
window with multiple temp DOM elements holding copied text.
I also add a comment to the code I think causes this race
for the tests.
Previously, we unconditionally tried to restart the Tornado process
name corresponding to the historically always-true case of a single
Tornado process. This resulted in Tornado not being automatically
restarted on a production deployment on servers with more than one
Tornado process configured.
Previously, because the parens were added via CSS, copy-pasting the
EDITED notices resulted in junk like this:
Iago 3:51 PMEDITED
edited message content
Now, you get:
Iago 3:51 PM (EDITED)
edited message content
Status messages were incorrectly not selectable, due to a bug in how
we setup the no-select hierarchy (for making copy-paste not have weird
whitespace issues).
Fixes#10456.
For many years we have been excluding the current user
from the buddy list, since their presence is kind
of implicit, and it saves a line of real estate.
This commit removes various user-is-me checks
and puts the user back for the following reasons:
* explicit is better
* newbies will be less confused when they
can see they're actually online
* even long-time users like myself will
feel more comfortable if it's just there
* having yourself in the buddy list facilitates
things like checking your presence or sending
yourself a message
* showing "me" reinforces the meaning of the
green circle (if my circle is green and I'm
active, then others with green circles must
be active too)
* If you're literally the first user in the
realm, you can now see what the buddy list
looks like and try out the chevron menu.
The biggest tradeoff here is the opportunity cost.
For an org with more people than fit on the screen,
we put the Nth person below the fold to show "me".
I think that's fine--users can still scroll or
search.
This commit doesn't do anything special with the
current user in terms of sorting them higher in the
list or giving specific styling.
Fixes#10476
We reduce nesting of code by just early-exiting
for the `is_current_user` check.
This also forces us to be a bit more thorough
with our tests if we want to maintain line
coverage.
This function is equivalent to recipient_for_emails, but fetches
user_profiles by IDs, not by emails.
This commit is a part of our efforts surrounding #9474, but is
more primarily geared towards adding support for sending typing
notifications by user IDs.
Previously, get_user_profiles() was split into two functions:
* user_profiles_from_unvalidated_emails, which raised a
ValidationError upon encountering a non-existent user email.
* get_user_profiles, which caught the ValidationError raised
by user_profiles_from_unvalidated_emails and raised a
JsonableError instead.
According to Steve Howell, this complexity is partly a relic
of past refactoring and is unnecessarily heavy. It is better to
just raise JsonableError directly.
recipient_for_emails is used by our typing notifications code.
user_profiles_from_unvalidated_emails is used by our typing
notifications code *and* for sending messages.
user_profiles_from_unvalidated_emails is a part of a larger
framework used by Addressee to validate recipient emails when sending
messages and will eventually need to be removed as we move forward
with #9474. So it makes sense to just inline this function within
recipient_for_emails so that we don't break our typing notifications
code in the future.
This commit is a part of our efforts surrounding #9474.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.
Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.
Fixes#10807.
This adds a web flow and management command for reactivating a Zulip
organization, with confirmation from one of the organization
administrators.
Further work is needed to make the emails nicer (ideally, we'd send
one email with all the admins on the `To` line, but the `send_email`
library doesn't support that).
Fixes#10783.
With significant tweaks to the email text by tabbott.
The previous content made it sound like we were actually sending a
push notification, which could be confusing/alarming in some cases
(see e.g. 9c224ccdd3). Instead, we make
clear that we're sending it to all clients (which one might correctly
suspect is vacuous in the development environment).
This fixes a lot of spammy output of the form:
2018-11-27 17:46:48.279 INFO [zerver.lib.push_notifications] Sending push notification to user 46
when running populate_db, which is both confusing (since we're not
actually sending push notifications here) and spammy.
While it could make sense to print these logging statements at WARN
level on server startup, it doesn't make sense to do so on every
message (though it perhaps did make sense to do so before more recent
changes added good ways to discover you forgot to configure push
notifications).
Instead, we now just do a WARN log on queue processor startup, and
then at DEBUG level for individual messages.
Fixes#10894.
For messages with strange senders, we don't import
messages. Basically, we only import a message if
it has sender with an id that maps to a non-deleted
user.
We now account for streams having users that
may be deleted. We do a couple things:
- use a loop instead of map
- only pass in users to hipchat_subscriber
- early-exit if there are not users
- skip owner/members logic for public streams
Change the truncation marker from `...` to `\n[message truncated]`
when receiving messages from the API or through e-mail. Also, update
tests to account for the new change.
Fix#10871.
Also fixes a bug in process_initial_upgrade. If you have a card on file
(e.g. from a previous subscription), and try to upgrade by billing by
invoice, neither the if nor the elif condition applies.
There are only a handful of non-JSON webhooks that wouldn't
benefit from the notify_bot_owner_on_invalid_json feature.
Specifically, these are the webhooks where the third-party product
uses another format, whether it be HTML form-encoded, XML, or
something else.
Tweaked by tabbott to correc the list of excluded webhooks.
Previously, the Stripe webhook code was riddled with implicit
assertions that there were exactly N event types within a given
category, and we handled the last one in a final `else` clause in the
block. This was likely to cause confusing problems in the event that
we're missing an event type (or Stripe adds a new one).
We fix this by just adding a few more conditionals and raising the
standard "unexpected event type" exception for the others.
There is no attribute called plan_interval. The reason
this was missed by mypy is moost likely due to us using
Any instead of Subscription in extract_current_subscription.
Our recent change in 2fa77d9d54 to
disable the cached_db cache backend broke upgrade-zulip-from-git with
an attributeerror; we fix that by checking the session engine before
trying to access its cache-related attributes.
Was hoping to do this after adding timestamp normalization to
normalize_fixture_data, which would have turned this into a <10 line
diff. There is a potentially material change in this API upgrade though
(around how invoices are handled), so just doing it now.
Removes email_not_verified option. That option was used to assign
email_data a different set of emails for a test. Instead of that,
this refactor allows to specify the email_data itself in the function
which calls github_oauth2_test. Flags like email_not_verified are
generally used in one test. This is a preparatory refactor for
choose email screen which may have introduced multiple flags otherwise.
Realm object is not json-serializable; store the realm id instead
and retrieve the realm in social_auth_finish using
`Realm.objects.get(id=return_data["realm_id"])`.
The email_list returned has the primary email as the first element.
Testing: The order of the emails in the test was changed to put a
verified email before the primary one. The tests would fail without
this commit's change after the changes in the order of test emails.
See https://github.com/zulip/zulip/issues/10856 for details on the bug
here; but basically, users who reset their password were unable to
login until the next time we flushed memcached. The issue disappeared
after stopping using the Django cached_db session engine, so it's
pretty clear that some sort of bug with that session engine
interacting with our password reset logic is the root cause.
Further debugging is required to understand this fully, but for now,
it seems wise to just disable the backend.
The cost of doing so is a small performance decrease, which is likely
acceptable until we can resolve this (it's certainly a more minor
problem than the "Can't login" bug that disabling this removes).
This fixes a bug where our API routes for uploaded files (where we
need to use a consistent URL between session auth and API auth) were
not properly configured to pass through the API authentication headers
(and otherwise provide REST endpoint settings).
In particular, this prevented the Zulip mobile apps from being able to
access authenticated image files using these URLs.
For message groups, I just changed the internal name
to "topic_links".
For uses of "subject_links" that are tied to how the
server names fields, I introduced these wrappers:
* util.set_topic_links(obj, topic_links)
* util.get_topic_links(obj)
These can be used for either messages or events.
Previously, messages were a string of disconnected regions. Modeling them as a list brings several benefits:
* Quickly jump to the message list by using a screen reader's list navigation hotkey.
* Quickly jump between messages by using a screen reader's list item navigation hotkey.
* Quickly jump to the beginning or end of message lists in screen readers that support it.
This is initial work, which will help us establish habits of using a
well-tested approach for renaming a Zulip organization (since as part
of https://github.com/zulip/zulip-mobile/issues/3142, we'll likely
need to make this function do more).
This is a nice performance optimization for the rare case where the
user does quote-and-reply on a message, aborts the compose, and then
re-does the quote-and-reply.
This seems like kind of a silly function to extract
to topic.py, but it will theoretically help us sweep
"subject" if we change the DB.
It had test coverage.
We use the message a lot for the query modified
here, so I think it's worth taking the up-front
hit of getting bulkier objects to avoid O(N)
hops back to the database.
Even though you'd think these regexes would be
cached, compiling the regex outside of looping
through lines makes a difference.
My timings are 8.4s -> 6.0s. (You need to hack
on the linter to isolate the custom checks.)
Normal hipchat exports use integer ids for their
users and "rooms," which we just borrowed during
conversion.
Atlassian Stride uses stride UUIDs for these instead, but otherwise
has the same export format.
We now introduce IdMapper to handle external ids
that aren't integer. The IdMapper will map UUID
ids to ints and remember them. For ints it just
leaves them alone.
Fixes#10805.
We split out two new functions and call them
everywhere that we used to call add_display_time():
- `update_group_time_display`
- `update_timestr`
We also make some of the local vars more consistent,
as well as doing more explicit clearing of vars than
`delete`.
Splitting these functions will allow us to muck with date
dividers without affecting the `update_str` functionality.
Our webhook-errors.log file is riddled with exceptions that are
logged when a webhook is incorrectly configured to send data in
a non-JSON format. To avoid this, api_key_only_webhook_view
now supports an additional argument, notify_bot_owner_on_invalid_json.
This argument, when True, will send a PM notification to the bot's
owner notifying them of the configuration issue.
Until we resolve https://github.com/zulip/zulip/issues/10832, we will
need to maintain our own forked copy of Django's SessionMiddleware.
We apparently let this get out of date.
This fixes a few subtle bugs involving the user logout experience that
were throwing occasional exceptions (e.g. the UpdateError fix you can
see).
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:
https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/Fixes#10674.
This was generated by:
* `rm -f requirements/*.txt`
* `./tools/update-locked-requirements`
and passes tests. The effect is to unpin ourselves from old versions
of a bunch of recursive dependencies of our third-party dependencies.
Without the cast mypy raises the following error:
Incompatible return value type (got "Callable[..., Any]",
expected "CallableT")
This is a known issue: https://github.com/python/mypy/issues/1927
This makes a few other changes to the fixtures as well. Most are from API
updates, though I'm not sure why "Zulip Cloud Standard" got changed to
"Zulip Cloud Premium".
[Substantial edits by Rishi Gupta]
Change wording of public stream description to
"Any member of the organization" from "Anybody"
to indicate that guest users can't subscribe even
public stream of organization.
We (lexically) remove "subject" from the conversion code. The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.
There was good code coverage on `build_message`.
We're trying to sweep "subject" out of the codebase,
even when it has nothing to do our legacy "subject"
field. The rewording here will prevent some linter
noise.
By far the dominant cause of errors when installing apt packages is
not having the Universe repository enabled in Ubuntu bionic (this
seems to have started happening a lot recently; I wonder if Ubuntu
changed the defaults for new server installs or something?).
In any case, providing that suggestion in the error output should help
reduce these a lot.
Previously, when a new stream was created on a client other than the
current one, the browser would first receive the "stream_created"
event, and make up a client-side display color at that time to use in
the "stream settings" view (it doesn't yet know the color that was
selected when the user was actually subscribed, because it doesn't
even know yet that the user is being subscribed to this stream), and
then moments after it'll receive a "susbcribe" event letting the
client know that the user is subscribed (and specifying the color to
use).
However, due to an argument not being passed through properly and a
missing rerender, we were not properly updating either the data
structures or doing a stream colors rerender in order to show the new
color.
This fixes the issue reported in
https://chat.zulip.org/#narrow/stream/48-mobile/subject/stream.20colors/near/660170
This moves the deactivate account which was previously placed under
"SETTINGS / YOUR ACCOUNT" -> "Deactivate account" to "SETTINGS / YOUR
ACCOUNT" -> "User settings" for making it more visible.
Previously, a string ending in "... 😄" was reported as an
error and the linter complained that there should be a space
after the last ':'. This commit changes the pattern so that the
linter only checks for colons that are preceded by an opening
double-quote (").
This was a pretty nasty error, where we were accidentally accessing
the parent list in this inner loop function.
This appears to have been introduced as a refactoring bug in
7822ef38c2.
We now prevent adding "subject" to any code in
zerver/lib, unless you specifically exempt it.
The new set called `FILES_WITH_LEGACY_SUJECT`
is also has comments that give a roadmap of what
to fix.
The various vars here that had recipient_subject
in the name now have either bucket or bucket_tup
there.
The shorter names are a bit easier to read, and the
original names were misleading for the PM case.
This was basically two search/replaces, and we have
good test coverage here, so it's pretty low risk
despite the messy diff.
While from my anecdotal experience, it is accurate that at many SaaS
providers, dozens or even hundreds of engineers have access to
production infrastructure, the "essentially all" statement isn't true
(not sure how that got added), and in any case, there aren't great
public sources we can cite to educate folks who might think Silicon
Valley startup practices are better than they actually are.
We now attach zulip_db_data to the markdown engines
for classes that need it. This was the last remaining
global we had, so we remove `arguments.py` here.
The Markdown processor makes it fairly simple for
the helper classes to access the `md` engine. We
now write `_md_engine.zulip_message` to avoid having
the current message in the global namespace.
Note that we do reuse engines for multiple messages,
but each engine is specific to a realm. And we therefore
avoid even the theoretical possibility of leaking message
data between realms.
This makes us consistent with how we import codehilite.
Using Python's normal import mechanism avoids some overhead
with Markdown having to parse dotted notation.
These modules are tiny, so they shouldn't impact startup
too much. Also, by explicitly importing them, we avoid
the pitfall of having a sucessful startup and a broken
renderer.
We were building the same link regex every time
we build a Markdown engine, which happens twice
per realm. It's an expensive operation due to
the complexity of the regex and us reading a file.
Nested classes are kind of expensive in Python,
particularly when you throw in mypy annotations.
Also, flatter is arguably better, although it is
kind of a pain here not to have closures.
This change avoids hitting the Django ORM when
we don't find any possible group mentions in
the message content.
Django doesn't necessarily actually hit the database,
but it's still slow and shows up in profiles.
This commit speeds up the import by avoiding
sender lookups and instead using the data
for users that we already have in memory.
This avoids a few DB hops, many hops to memcached,
plus some object construction.
We now call do_render_markdown() directly. This
also makes it more explicit that the import has
never rendered alert words.
For the import-data codepath, we will call
the extracted function directly in a
subsequent commit.
The do_render_markdown() function has more
required parameters, which allows for more
explicit code and also allows us to flatten
out some logic related to alert words. (We
just pass in empty sets/dicts as needed).
We can rely on `message_realm` being the same
as `message.sender.realm`, which allows us to
skip two queries to the database for the rare
Zephyr mirroring case.
This function requires a message object, whereas
we want to work with JSON data to avoid necessary
queries when we import data. Inlining the function
sets us up for a subsequent refactoring.
We change the way we deal with theoretical return
values of `None` to use an assertion; otherwise,
we would have to loosen up a bunch of mypy types
from `str` to `Optional[str]`. It's not clear `None`
is even possible--we've moved toward throwing exceptions
there instead of silently failing.
This is somewhat hairy logic, so it's nice
to extract it and not worry about variable leaks.
Also, this moves some legacy "subject" references out
of actions.py.
We start by including functions that do custom
queries for topic history.
The goal of this library is partly to quarantine
the legacy "subject" column on Message.
Use the placeholder `[Quoting…]` when quoting and replying before the
quote has been added to the message. Also, add tests to the
`compose_actions` Node tests for the new behavior.
Fix#10705.
A recent change to check_send_webhook_message allows webhooks to
unescape stream names before sending a message. This commit adds
a test for the edge case where the webhook URL is escaped twice by
a third-party.
Apparently, we can use the process group naming style of having dashes
in the names without using the explicit nun_procs feature of
supervisord configuration.
The new configuration is perfectly satisfactory, so there's no real
reason to prefer the old approach.
This allows our Tornado monitoring to correctly report whether
multiple configured Tornado processes are running.
This setup isn't ideal, in that it can't detect cases where the wrong
set of Tornado processes are running, but it's nice and simple and
should catch most actual problems.
Recently, one of our users reported that a JIRA webhook was not
able to send messages to a stream with a space character in its
name. Turns out that JIRA does something weird with webhook URLs,
such that escaped space characters (%20) are escaped again, so
that when the request gets to Zulip, the double escaped %20 is
evaluated as the literal characters `%20`, and not as a space.
We fix this by unescaping the stream name on our end before
sending the message forward!
The previous logic was incorrect, in that if `content_type` was set to
None (which happens with Slack/HipChat export, among other things),
then we wouldn't run the `guess_type` logic to auto-detect the
Content-Type to send to S3.
Guest users can't access subscribers of any(public or private)
non-subscribed streams. Therefore, hide subscribers list
of all non-subscribed streams from guest users in UI.
Fixes#10749 (the previous parts were fixed already).
For the PHP link, I just found an equivalent link that detailed
some well-known PHP security vulnerabilities.
As for the now defunct RequestBin, another instance is hosted
by another domain, so we can recommend using that for now.
The last_modified field is intended to support setting the
orig-last-modified field in the S3 backend when importing, basically
to keep track of this bit of pre-export data for debugging. In the
event that it isn't available, the correct thing to do is not write
out an invalid `last_modified` field; we should just not write it out
at all.
This fixes the fact that these emoji were sometimes not displaying
properly (because of changes in the emoji names used in the codebase),
while also making this integration more standard (since it was the
only one with such an aggressive use of emoji).
The UserMessage table can be huge, so creating a
bunch of entries in `ID_MAP` can overflow memory.
We don't have any tables that depend on `UserMessage`,
and we don't send the 'id' fields from `zerver_usermessage`
to the database, so re-mapping them was just busy-work.
"description":"Additional license (Apr 11, 2012 - Jan 2, 2013)",
"discountable":false,
"id":"ii_1DxdeVGh0CmXqmnwf6cLlEA7",
"invoice":null,
"livemode":false,
"metadata":{},
"object":"invoiceitem",
"period":{
"end":1357095845,
"start":1334113445
},
"plan":null,
"proration":false,
"quantity":3,
"subscription":null,
"unit_amount":5814
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
