This commit upgrades 0015_clear_duplicate_counts migration to remove
duplicate count in StreamCount, UserCount, InstallationCount as well.
Fixes https://github.com/zulip/docker-zulip/issues/266
Fixes#15904.
settings is supposed to be a proper OneLogin_Saml2_Settings object,
rather than an empty dictionary. This bug wasn't easy to spot because
the codepath that causes this to demonstrate runs only if the
SAMLResponse contains encrypted assertions.
This commit rewrites the way addresses are collected. If
the header with the address is not an AddressHeader (for instance,
Delivered-To and Envelope-To), we take its string representation.
Fixes: #15864 ("Error in email_mirror - _UnstructuredHeader has no attribute addresses").
This commit re-adds the integration for canarytokens.org, now separate
from the primary Thinkst integration.
Signed-off-by: David Wood <david@davidtw.co>
This commit fixes the Thinkst Canary integration which - based on the
schema in upstream documentation - incorrectly assumed that some fields
would always be sent, which meant that the integration would fail. In
addition, this commit adjusts support for canarytokens to only support
the canarytoken schema with Thinkst Canaries (not Thinkst's
canarytokens.org).
Signed-off-by: David Wood <david@davidtw.co>
On calling `loading.make_indicator` for the second
time or more no spinner is being displayed.
This bug can be viewed on visiting a `near: 1` narrow
and the spinner for the newer messages is displayed
only once (i.e. the first time it is rendered), while
the logo is displayed every time.
This happens because `loading.destroy_indicator` sets
the css of that container to display: "none". This can
be removed as we are emptying the container just above.
Introduced in 953d475274.
Zulip Desktop version 5.3.0 only supported adding custom certificates
inside the application. Starting in version 5.4.0, it also supports
reading from the system certificate store; in the next release (likely
5.5.0) the support for the internal store will be removed.
Document the change, and add explicit instructions on how to add
certificates into the system store on each of the operating systems.
Co-authored-by: Manav Mehta <tmanavmehta@gmail.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A few major themes here:
- We remove short_name from UserProfile
and add the appropriate migration.
- We remove short_name from various
cache-related lists of fields.
- We allow import tools to continue to
write short_name to their export files,
and then we simply ignore the field
at import time.
- We change functions like do_create_user,
create_user_profile, etc.
- We keep short_name in the /json/bots
API. (It actually gets turned into
an email.)
- We don't modify our LDAP code much
here.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.
An upcoming commit will eliminate it from the
database.
This fixes up some complex helpers that may
have had some value before f-strings come along,
but they mostly obscured the logic for
people reading the tests.
We still keep really simple helpers for the
common cases, but there are no optional
parameters for them.
One goal of this fix is to remove the
short_name concept, and we just explicitly
set senders everywhere we need them.
We also now have each test just explicitly set
its reaction_type.
For cases where we have custom message ids
or senders, we just inline the simple call
to api_post.
We generally want to avoid having two sibling test
suites depend on each other, unless there's a real
compelling reason to share code. (And if there is
code to share, we can usually promote it to either
test_helpers or ZulipTestCase, as I did here.)
This commit is also prep for the next commit, where
I try to simplify all of the helpers in EmojiReactionBase.
Especially now that we have f-strings, it is usually
better to just call api_post explicitly than to
obscure the mechanism with thin wrappers around
api_post. Our url schemes are pretty stable, so it's
unlikely that the helpers are actually gonna prevent
future busywork.
It's not clear to me why these passed mypy
before, given this:
def assert_realm_values(f: Callable[[Realm], Any], ...
But this is clearly more accurate.
This issue isn't something a system administrator needs to take action
on -- it's a likely minor logic bug around organization
administrators moving topics between streams.
As a result, it shouldn't send error emails to administrators.
It's unclear what the purpose of this logic was, but testing confirms
that the text color is as expected without this in the day theme (so
it's likely a relic of an old design) and removing it fixes the hover
text being overridden to white in the night theme.
This is a hacky fix to avoid spoiler content leaking in emails. The
general idea here is to tell people to open Zulip to view the actual
message in full.
We create a mini-markdown parser here that strips away the fence content
that has the 'spoiler' tag for the text emails.
Our handling of html emails is much better in comparison where we can
use lxml to parse the spoiler blocks.
We hide the spoiler content in browser/desktop notifications.
Note: its not worth adding zjquery tests for this bit of code because
the tests do not operate on the actual data and are likely to get stale
if we change the syntax for spoilers.
We include tests for the new implementation to avoid churning the
codebase too much so this can be easily reverted when we are able to
re-enable the feature.
This handler adds a neat little effect whereby hovering over the
clickable region to open the navbar triggers the search_icon hover
effect and is a neat little visual cue about what happens onClick.
The previous implementation was slightly messy because it fetched the
color and applied it via ".css(". This commit cleans it up by creating
and using the class "search_icon_hover_highlight" instead. We also
make the selectors more specific, ensuring they target children of
"#tab_bar", this was so because it was reasonable to expect someone to
define eg `search_closed` elsewhere and we wanted to prevent bugs when
that happened.
In 9046fc1032 we updated the navbar.html
file so that our css selectors did not override each other and cause
annoying problems.
Unfortunately this caused a regression in night mode where the
search_icon didn't have the correct hover effect.
This fixes the regression by adding the selectors.
The page_params.timezone feature is perhaps a misfeature, but
importantly it's not what is used to display the time in the message
feed (it's mainly used to show others your timezone).
Given that reality, we shouldn't use it for a feature whose whole
purpose is to display the time using the same timezone we use in the
message feed.
Fixes#15790.
We were not passing any arguments to needs_subscribe_warning
when testing the function itself.
This commit changes the code to pass the user-id and stream-id
to needs_subscribe_warning. We also remove the stubs for
get_by_user_id and is_user_subscribed and do these tests by
calling the original functions, because passing the arguments
(user-id and stream-id) only makes sense if we use original
functions for them rather than stubs.
Delete stored topic data in `recent_senders` and `recent_topics`
about the message's topics and re-render them. The process is similar
to topic editing. See `recent_senders.process_topic_edit` for
logical details.
The tests had a bunch of different ways to create
users; now we are consistent. (This is a bit of
a prep step, too, to allow us to easily clean
Hamlet's existing words before each test.)
The prior version clobbered all flags, which means
we had unrealistic values for is_private.
Now we only touch the unread flag, which
also means when we go next to create alert words,
those will now work.
In 9648e64d23, we added a clear outline
around focused link elements in popovers.
This was a good fix for popovers, but it's distracting for the
experience of clicking links in the sidebars and message feed.
We could certainly do better with the handling here, but using the raw
string that the user gave us is okayish for now.
Proper formatting of timestamps requires handling locales and timezones
of the receiver as well which is a larger project.
We now do something sensible for spoilers in notifications. A message
like:
```spoiler Luke's father is
Vader. Don't tell anyone else.
```
would be rendered as:
Luke's father is (...)
We have changed our all instances of list_render to use
simplebar and thus, we will now use simplebar container
to track scroll event for all the lists created by
list_render.
This fixes the bug of new subscribers not rendering on
scrolling at the end of subscriber list in stream settings
and similar bug in some other lists also.
This commit also removes scroll_util.get_list_scrolling_container
function as this is no longer used.
Fixes#15637.
This commit fixes the dropdown_list_widget to use simplebar for
scrolling.
It was not being used because data-simplebar should not be inserted
to the element being rerendered. This commit adds a new element
wrapping 'dropdown-list-body' which was being rerendered and
data-simplebar is added to that new element 'dropdown-list-wrapper'.
Also, there should always be a max-height property on data-simplebar
element and it is also added in this commit.
There is also a change to set margin of 10px only on the first div
element and which is direct children of organization-settings-parent
element. This is correct because we only want margin to add some space
between the heading of subsection and the first setting of that
subsection. Previously, the margin was being added to first div of all
the other child containers also and this was adding unnecessary margin
to the first div element of different simplebar containers.
We do not need to use list_render for displaying list of streams specific
notification settings, as this is not scrollable and we do not provide
option to sort or filter this list as well.
After this change, all our list_render instances will be using simplebar
and we can change the code accordingly to fix the behaviour of scrolling.
We remove the action column and show action buttons next to topic
after unread count (if present). This save us a lot of extra space
on small window sizes.
For a:focus Bootstrap sets the following rules:
outline: thin dotted #333;
outline: 5px auto -webkit-focus-ring-color;
Firefox does not know -webkit-focus-ring-color and falls back to the
previous rule, making the outline invisible in darkmode.
Chromium has a bug[1] that makes outline: auto invisible when focussing
elements programmatically (which we do for the up & down arrow keys).
[1]: https://bugs.chromium.org/p/chromium/issues/detail?id=1105822Fixes#15768.
jQuery's fadeOut() sets display: none using inline CSS.
This was overriden by .alert-notification since it used !important
to override the display: block set in loading.js. Removing the latter
allows us to remove the !important, and doesn't seem to break anything.
Fixes#15759.
Prior to PostgreSQL 12, the `recovery_target_timeline` setting is only
valid in a `recovery.conf` file, as that file has its own
configuration parser. As such, including it in `postgresql.conf`
results in an error, and PostgreSQL will fail to start.
Remove the setting, reverting bff3b540b1. This fixes PostgreSQL 9.5,
9.6, 10, and 11; while the setting is not an error in a PostgreSQL 12
configuration file, it is unnecessary since `latest` is the default.
7d4a370a57 attempted to move the replication check to on the
PostgreSQL hosts. While it updated the _check_ to assume it was
running and talking to a local PostgreSQL instance, the configuration
and installation for the check were not updated. As such, the check
ran on the nagios host for each DB host, and produced no output.
Start distributing the check to all apopdb hosts, and configure nagios
to use the SSH tunnel to get there.
OpenAPISpec is our main class for accessing OpenAPI objects and
has the capability of creating the OpenAPI objects only once, thus
saving time. Since the openapi-core request validator object is going
to be accessed for considerable time during testing, hence add it to
the class for faster testing.
We extract OptionalContent and RequiredContent since some endpoints
require it and other don't.
In an ideal world, we'd have a better way to express these small
variants.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/users/{user_id}` can match
with `users/me`. So change the position of`/user/{user_id}` after all
such possible matches to avoid errors.
See https://github.com/p1c2u/openapi-core/issues/226 for details.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/messages/{message_id}`
can match with `messages/matches_narrow`. So change the position of
of `message/{message_id}` after all such possible matches to avoid
errors.
See https://github.com/p1c2u/openapi-core/issues/226 for details.
This reverts commit 63643c9d9d.
As the commit mentions, it makes a UI change for legacy search which
has largely been considered a regression. We've been running with
this reverted in zulip.com essentially since it was first merged.
Apparently, our scrollbar logic crashed with an invalid URL fragment
(hash), which resulted in initialization not completing and thus the
logic failing.
In my view the root issue here is that we're not doing a good job of
catching JavaScript exceptions in portico pages.
Fixes#15706.
wal-g was used in `puppet/zulip` by env-wal-g, but only installed in
`puppet/zulip_ops`.
Merge all of the dependencies of doing backups using wal-g (wal-g
installation, the pg_backup_and_purge job, the nagios plugin that
verifies it happens) into a common base class in `puppet/zulip`, since
it is generally useful.
In 42f20e81be I fixed an edge case but
also accidentally made clicking on reactions open the compose box.
This commit adds back the e.stopPropagation(); and explicitly hides the
emoji picker popover, to address the inconsistency fixed in the previous
commit.
Previously, we were experiencing a bug that caused the left border of
the searchbox/tab_bar to disappear when the searchbox was opened. This
bug was a result of the following changes:
- 4cdd7aed2b accidentally added this line
as right: 2; instead of right: 2px;
- 46c966576d fixed this line to be
right: 2px; but caused the regression.
This commit fixes the bug by deleting this line.
If the push_notification for the UserMessage is already active,
we don't send any push notification to the user. This may
happen due to race conditions.
Added and fixed test cases affected by this.
Previously user have to click pixel perfect on the message controls
icons to achieve the click action.
This commit will uniformly increases the click target
area for the icons.
Tweaked by tabbott to avoid some weird glitches.
Throughout the codebase we use <i> tag for icons.
This commit will add <i> tag inside the starred message
div and fa classes are now used with this <i> tag.
The starred message div is now consistent with other
message_controls divs.
Tweaked by tabbott to use the name star_container for better
readability.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base.
It would have been nicer if we could simply purge tab_bar from the
codebase and rename "#tab_list" so that we have an anchor and wrapper
structure in the html, but dropping the float: left on tab_bar causes
some confusing problems such as causing the horizontal border to
disappear and the search_box to shift out of its intended position and
so its simpler to get rid of tab_list from our code base first.
This commit:
- Removes the #tab_list wrapper div from tab_bar.hbs.
- Removes any #tab_list selectors from night_mode.scss so that they
simply target based on "#tab_bar" instead of "#tab_bar #tab_list".
- Removes tab_list selectors from zulip.scss, so that #tab_list
attributes now apply to the #tab_bar, in the process we drop the
duplicated width property and reorder the attributes.
- Replaces all mention of #tab_list with #tab_bar in JS files.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base. This commit pushes us towards that goal by simplifying
the future merge of the two CSS labels.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base. This commit pushes us towards that goal by simplifying
the future merge of the two CSS labels.
The letter-spacing attribute was set to its default value and so we
remove it and rely on the default.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base. This commit pushes us towards that goal by simplifying
the future merge of the two CSS labels.
Currently the #tab_bar is just a container for the #tab_list and the
#tab_list handles the overflow attributes and so these do not serve
any purpose.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base. This commit pushes us towards that goal by simplifying
the future merge of the two CSS labels.
We have the same selector within #tab_list and the duplication is
unnecessary.
Ideally we would not be relying on something that requires a 4 line
comment, and also makes it harder to add more static elements at the
end of the navbar, but this block should be acceptable for now.
One alternate would be a "grow-1" class or similar but we might need
to think that through.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base. This commit pushes us towards that goal.
Previously, this element was part of both searchbox and
searchbox_legacy of which, only one would render based on the flag on
search pills. This wasn't great because:
* it made it likely that someone would change only one of the two and
unintentionally introduce regressions.
* it meant that search_icon selectors within the searchbox would mess
with the search_icon elements within the tab_bar, leading us to rely
on messy CSS overriding.
Since there doesn't seem to be a strong reason to have this be the way
it previously was, this commit extracts "#tab_bar".
It's worth keeping in mind that we use the "#tab_bar" element as an
anchor to append the #tab_list onto.
Previously, we had the entire div within the conditional, instead of
just the contents, which were the only variable elements.
This change moves the conditional over just the contents of the div
and improves readability.
Previously, the navbar sub count would not live update as users
subscribed or unsubscribed, this commit adds the relevant calls in
stream events.
It would have been better to just have a single call within
server_events_dispatch but it seems difficult due to the way of
mark_subscribed and mark_unsubscribed are structured.
stream_events.mark_unsubscribed conditionally calls
subs.update_settings_for_unsubscribed which calls
subs.rerender_subscriptions_settings and as such handles the update
for the subscriptions modal on its own. Hence, we simply rely on the
stream_data.update_calculated_fields to ensure the subscriber counts
are updated and make a call to
tab_bar.maybe_rerender_title_area_for_stream(sub).
stream_events.mark_subscribed is similar.
Previously, we were overriding narrow_state.is_for_stream_id() to make
sure we test the functions we intend to.
In this commit we:
* zrequire('narrow_state')
* set the filter to "stream:frontend" before the test cases which
were overriding is_for_stream_id to return true (and remove the
overrides).
* reset_current_filter() at the end of the above cases (and remove
the lines overriding is_for_stream_id to return false)
This is a prep commit to adding live update for sub_count in the
navbar.
Previously, we had the lines from this block being duplicated in all
the stream update paths, which is a little brittle.
Hence, in this commit, we extract it out with a comment explaining
what it does and call it in all the duplicated spots.
My previous message_header fix
0b4568d249
accidentally changed the alignment of dates in private messages (so that
it was inconsistent to the alignment in other narrows).
We use narrow_state.stream_sub instead of narrow_state.stream to directly
get the sub object instead of stream name, while subscribing to a stream
from the stream narrow.
This commit changes stream_data.is_user_subscribed to use stream id
instead of stream name.
We are using stream ids so that we can avoid bugs related to live
update after stream rename.
Fixes#12868.
We now also include python version in the format
'major.minor.patchlevel', when generating hash for a
requirement file. This was necessary since packages tend to
break on different versions of python, so it is important to
track the version on which the venv was setup.
WARN: This commit will force all zulip venvs to be recreated.
We were already using packages names along with their versions
to generate hash for the requirement file, as we were passing
the `.txt` files to the hash_reqs file instead of intended `.in` files
for which the functions in this file was originially designed.
Changed the expand_reqs_helper function to adapt for the `.txt` files.
No plugins are installed inside the /usr/local/munin/lib this creates
in munin-node, nor are they symlinked into /etc/munin/plugins, so
non-default plugins are added by this.
The contents in the database are unchanged across the PostgreSQL
restart; as such, there is no reason to invalidate the caches.
This step was inherited from the general operating system upgrade
documentation. When Python versions change, such as during OS
upgrades, we must ensure that memcached is cleared. However, the
`do-release-upgrade` process uninstalled and upgraded to a new
memcached, as well as likely restarted the system; a separate step for
OS upgrades to restart memcached is thus unnecessary.
pg_upgradecluster has two possibilities for `--method`: `dump`, and
`upgrade`. The former is the default, and does a `pg_dump` of all of
the databases in the old cluster and feeds them into the new cluster.
This is a sure-fire way of getting the same information in both
databases, but may be extremely slow on large databases, and is
guaranteed to fail on servers whose databases take up >50% of their
disk.
The `--method=upgrade` method, by contrast, uses pg_upgrade to copy
the raw database data file over to the new cluster, and then fiddles
with their internal structure as needed by the upgrade to let them be
correct for the new version[1]. This is slightly faster than the
dump/load method, since it skips the serialization step, but still
requires that there be enough space on disk for both old and new
versions at once. `pg_upgrade` is currently supported for all
versions of PostgreSQL from 8.4 to 12.
Using `pg_upgrade` incurs slightly more risk, but since the it is
widely used by now, using it in the relatively-controlled Zulip server
environment is reasonable. The expected worst failure is failure to
upgrade, not corruption or data loss.
Additionally passing `--link` uses hardlinks to link the data files
into both the old and new directories simultaneously. This resolve
both the runtime of the operation, as well as the disk space usage.
The only potential downside to this is that as soon as writes have
occurred on the upgraded cluster, the old cluster can no longer be
started. Since this tooling intends to remove the old cluster
immediately after the upgrade completes successfully, this is not a
significant drawback.
Switch to using `--method=upgrade --link`. This technique spits out
two shell scripts which are expected to be run after completion of the
upgrade; one re-analyzes the statistics, the other does an `rm -rf` of
the data where it is still hardlinked in the old cluster. Extract the
location of these scripts from parsing the `pg_upgradecluster` output;
since the path is not static, we must rely on it being relatively easy
to parse. The risk of the path changing is lower, and has more
obvious failure modes, than inserting the current contents of these
upgrade steps into the overall `upgrade-postgres`.
[1] https://www.postgresql.org/docs/12/pgupgrade.html
We have logic in place to update the ui for re-sending messages
on recieving the acknowledgement from the server on that API call.
However, if the acknowledgement is recieved through the get events
request before the `on_success` of `resend_message`, the message
gets re-rendered allowing the failed message actions to be clickable.
Now, we update the ".message_failed" ui for both cases. This helps
in preventing the "Trying to get local_id from row that has reified
message id" exception.
Fixes#15351.
Previously, we rendered the twitter previews outside of a
spoiler block at the end of the message. The commit series
ending with this commit fixes that by inlining twitter
previews instead of appending them all at the end. As a
consequence of the inlining, we have fixed the issue here.
This commit just adds a test to assert that.
Fixes#15518.
This is similar to our behavior with image previews, and helps
reduce clutter in the final rendered html.
We add the string 'Tweet: ' to our existing tests so those tests
remain the same.
This commit makes our handling of twitter previews consistent with
how we handle our inline images so that tweets render next to the
paragraph that links to the tweet.
We decouple the logic of insertion rules for inline links from
image preview logic. Now, we can use this same logic for other
kinds of link previews as well.
fadeTo is not a good method to hide elements since it sets
opacity to 0 in which the element still can consume space and
be clickable. We set it's display to None using fadeOut method.
Also, allow this method to be called via ui_report.error.
The one complexity is that hosts_fullstack are treated differently, as
they are not currently found in the manual `hosts` list, and as such
do not get munin monitoring.
check_memcached does not support memcached authentication even in its
latest release (it’s in a TODO item comment, and that’s it), and was
never particularly useful.
Although mktemp is deprecated due to security issues, this is not a
security issue.
The security problems with mktemp happen when you open the resulting
filename (without O_EXCL) in a publicly writable directory, because
then someone else might have predicted the filename and created or
symlinked or hardlinked something there between the mktemp and the
open, causing you to write to a file you didn’t expect.
Here we don’t open the resulting filename, we symlink to it. symlink
will refuse to clobber an existing file, and we handle the error that
arises from this case. This is the normal way to atomically create a
symlink.
We should still replace mktemp because it’s deprecated, but we can’t
replace it with a function that creates the temporary file. Instead
we build a random filename ourselves.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously .recipient_row_date was positioned absolutely, allowing it to
overlap with the topic name on narrow screens. This can be solved by
using flexbox. To implement the empty space between the bar controls and
the date on wider screens we move the date outside of the bar controls
(which also makes more sense semantically since the date isn't a
control).
Fixes#15501.
The stream notification settings checkboxes were not checked
even when the notifications were turned on for the stream.
This was happening because we were passing stream name to
receives_notifications instead of stream id.
This commit fixes the bug by passing stream id to
receives_notifications. This change should have been done
in f3604fb while refactoring receives_notifications to use
stream id instead of name.
As a consequence of commit 1113589b9d the backspace key and some other
keys did not respond if the search yielded no results.
This change fixes that bug.
We also remove the post_process_data option.
The sanity check is just overkill at this point,
since the mechanism to find streams is very
direct due to a recent commit.
Before this change we would only export streams
that had actual subscribers, which is usually
harmless, but it was mostly a relic of a one
time migration that we did when we were cleaning
up some dirty data in some of our very early
databases (circa 2016).
Now we work down the table hierarchy in a
more natural way:
- get Streams in Realm
- get Recipients matching above Streams
- get Subscriptions matching above Recipients
Note that for per-user exports, I kept the
same logic (users -> subscriptions -> recipients ->
streams) we had before.
One subtle detail here is that we make our
final Config blocks--which build the final
version of Recipient/Subscription--now hang
off of realm_config.
Fixes#15146.
This fixes the bug of extra topics not being rendered on scrolling.
list_render uses `max-height` to determine which container is being
scrolled upon. Set the `max-height` on the scrolling container of
recent topics to help list_render identify it.
This is useful for the emoji-picker where holding down shift lets you
select multiple emojis. Otherwise when selecting multiple emojis with
the keyboard shortcuts you would need to release shift everytime you
wanted to navigate with the arrow keys.
Previously clicking on an existing message reaction (outside of the
emoji picker) while having the emoji picker open, removed the reaction
without updating the highlighting of the reacted emoji in the emoji
picker.
The emoji picker already is already closed when clicking outside.
The message reaction click handler however previously stopped the event
propagation, leaving the picker open, allowing the inconsistency.
Since we already run this on every push we don't need to run it as a
cron job every week for no reason. While we are touching this code
block, we convert it to on: [push, pull_request] since the previous
format felt weird. It was only written that way because we had the
cron job declared there.
There is a bug when clicking on the stream row in stream settings page
subscribes/unsubscribes the user from stream. The ideal behaviour
should be that user should be subscribed/unsubscribed only
when user clicks on tick icon. The buggy behaviour is visible only after
clicking the tick icon once.
This is because clicking on tick is adding sub_unsub_button class to the
stream row, while it should be added only to the ".check" element to
re-enable the button to handle requests.
This commit fixes the bug by adding the "sub_unsub_button" class to
".check" element only and not to the stream row and same change
is done for removing the "sub_unsub_button" class also.
Subscribe/Unsubscribe button in the right section of stream settings
page is not working because the target element in click handler was
changed to 'check.sub_unsub_button' in c234b4f2 and the button in
the right section with class 'sub_unsub_button' also uses the same
click handler.
This commit reverts c234b4f2 and the bug of subscribing/unsubscribing
the user by clicking on stream row in the stream settings page, which
c234b4f2 intended to fix will be fixed in next commit.
As we add more features where rendered_markdown.update_elements does
something useful, it'll become important to run this code everywhere
we render markdown in the DOM.
One can see in this case that we had actually copied one hunk of
rendered_markdown.update_elements years ago, before we extracted it as
an independent function; we get to delete that copy.
Fixes#15500.
success-http-headers-bionic.txt and success-http-headers-focal.txt
differ only in the nginx version so this substitution will allow
us to have single file for both of them. Also this change helps
to avoid CI failure if Nginx version is updated in the OS.
We do not show the warning while sending messages to announce
stream if there is a wildcard mention in the message (i.e.
when wildcard_mention != null)
There are two cases where we should ideally show the warning
but we don't-
- When there is no wildcard mention in the message and
wildcard_mention is set to undefined (initial value of
wildcard_mention).
This is because "wildcard_mention != null" returns true for
this case and thus the warning is not shown, assuming the
message to have wildcard mention.
- When previous message had a wildcard mention and now a message
is being sent with no wildcard mention.
This is because the condition "wildcard_mention != null" is
checked with the previous value of wildcard_mention and not
with the value according to current message content, and thus
the warning is not shown, assuming the message to have wildcard
mention.
This commit changes the code to set wildcard_mention from the
latest message content before performing other validations and
thus solves the problems described above.
This particular commit has been a long time coming. For reference,
!avatar(email) was an undocumented syntax that simply rendered an
inline 50px avatar for a user in a message, essentially allowing
you to create a user pill like:
`!avatar(alice@example.com) Alice: hey!`
---
Reimplementation
If we decide to reimplement this or a similar feature in the future,
we could use something like `<avatar:userid>` syntax which is more
in line with creating links in markdown. Even then, it would not be
a good idea to add this instead of supporting inline images directly.
Since any usecases of such a syntax are in automation, we do not need
to make it userfriendly and something like the following is a better
implementation that doesn't need a custom syntax:
` Alice: hey!`
---
History
We initially added this syntax back in 2012 and it was 'deprecated'
from the get go. Here's what the original commit had to say about
the new syntax:
> We'll use this internally for the commit bot. We might eventually
> disable it for external users.
We eventually did start using this for our github integrations in 2013
but since then, those integrations have been neglected in favor of
our GitHub webhooks which do not use this syntax.
When we copied `!gravatar` to add the `!avatar` syntax, we also noted
that we want to deprecate the `!gravatar` syntax entirely - in 2013!
Since then, we haven't advertised either of these syntaxes anywhere
in our docs, and the only two places where this syntax remains is
our game bots that could easily do without these, and the git commit
integration that we have deprecated anyway.
We do not have any evidence of someone asking about this syntax on
chat.zulip.org when developing an integration and rightfully so- only
the people who work on Zulip (and specifically, markdown) are likely
to stumble upon it and try it out.
This is also the only peice of code due to which we had to look up
emails -> userid mapping in our backend markdown. By removing this,
we entirely remove the backend markdown's dependency on user emails
to render messages.
---
Relevant commits:
- Oct 2012, Initial commit c31462c278
- Nov 2013, Update commit bot 968c393826
- Nov 2013, Add avatar syntax 761c0a0266
- Sep 2017, Avoid email use c3032a7fe8
- Apr 2019, Remove from webhook 674fcfcce1
This commits adds the steps for configuring message retention policy
for an organization and for individual streams in message retention
policy docs.
Fixes#15495.
The installer does not adjust the node name if the rabbitmq already
exists, and the default node name bakes in the
`zulip-install-bionic-base` hostname. As such, the resulting LXC
image does not properly start rabbitmq.
Remove rabbitmq, allowing the installer to install and configure it
with a nodename of `zulip@localhost`. This also lets the installed
image be successfully copied and booted under a new hostname without
breaking rabbitmq.
Previously, the function update_table_stream_color did not update the
color of the date_row when it appeared within a topic. This was
incorrect because it meant that a small piece of the border to the
left of the message box would be left to the previous color when the
stream color was updated. This commit resolves the issue by adding an
update.
Fixes#15197.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.
This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.
In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration. Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.
This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
This is a fine solution short-term until github implements the
yaml anchors support. The limitation of this method is that we
cannot re-use most of the steps again for production install test
builds.
Thanks, Anders for this solution.
Verifying everything is migrated correctly is a pain. This script
ensures everything is done correctly (previous commit message
contains explainations for the steps being ignored if; in case
of github-actions steps they are ignored because they are actions
specific):
"""
This script prints out the ignore steps first. Then
prints out each step of both circle and actions side-by-side.
One step is out of order for bionic but verfying correction is
still easier. Format:
Actions: Install dependencies
Circle CI: install dependencies
....
"""
import yaml
with open('.circleci/config.yml') as f:
circleci_config = yaml.safe_load(f)
with open('.github/workflows/zulip-ci.yml') as f:
actions_config = yaml.safe_load(f)
circle_bionic_steps = []
circle_focal_steps = []
actions_bionic_steps = []
actions_focal_steps = []
"""
We ignore casper artifact upload, save_cache, and
store_tests_reports steps.
"""
def get_circleci_steps(job, arr):
for step in circleci_config['jobs'][job]['steps']:
if isinstance(step, str):
arr.append(step)
continue
step_name = step.get('run', {}).get('name', False)
if not step_name:
if step.get('restore_cache'):
key = step['restore_cache']['keys'][0].split('.')[0]
step_name = f'<restore-cache> {key}'
elif step.get('store_artifacts', False):
destination = step['store_artifacts']['destination']
step_name = f'<store-artificats> {destination}'
if destination == 'casper':
\# This is no longer needed
print('Ignoring step:')
print(step)
print()
continue
else:
"""
We don't care about save_cache; github-actions
does this automatically, and store_tests_reports
is circelci timing specific.
"""
print('Ignoring step:')
print(step)
print()
continue
if step_name != 'On fail':
arr.append(step_name)
get_circleci_steps('bionic-backend-frontend', circle_bionic_steps)
get_circleci_steps('focal-backend', circle_focal_steps)
""" We ignore there steps specific to github-actions"""
for step in actions_config['jobs']['focal_bionic']['steps']:
BOTH_OS = 'BOTH_OS'
if_check = step.get('if', BOTH_OS)
step_name = step.get('name')
if step_name is None:
step_name = step['uses']
if (
step_name == 'Upgrade git for bionic' or
step_name == 'Add required permissions' or
step_name == 'Move test reports to var'
):
print('Ignoring step:')
print(step)
print()
"""These are github-actions specific; see comments"""
continue
if if_check == BOTH_OS:
actions_bionic_steps.append(step_name)
actions_focal_steps.append(step_name)
elif 'is_bionic' in if_check:
actions_bionic_steps.append(step_name)
else:
actions_focal_steps.append(step_name)
bionic = zip(circle_bionic_steps, actions_bionic_steps)
focal = zip(circle_focal_steps, actions_focal_steps)
print('Bionic steps:')
for (circle_step, actions_step) in bionic:
print(f'CircleCI: {circle_step}')
print(f'Actions: {actions_step}')
print()
print('Focal steps:')
for (circle_step, actions_step) in focal:
print(f'CircleCI: {circle_step}')
print(f'Actions: {actions_step}')
print()
Doing service memcached start instead of restart fixed an issue on
focal build in GitHub actions, where it exits with code 1 when it
is done twice.It is done first in Install Dependencies step and then
again in last step where we call tools/ci/setup-backend again which
runs provision.
Furthermore, I don't belive there is a technical reason we use
restart over start; rather I think it was just a random choice with
the intend to just start the services in CI. I traced the code back
to commit 1f2f497cab if it helps.
Looking at the source code of memcached, the step that's failing is:
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
which is equivilent to: service memcached stop, we can rule out the
service memcache start since it works. Ideally, we do figure out and
solve the issue of why memcached fails when executing service
memcached stop but I am not equipped with debugging it. And this
workaround seems reasonable rather than a "hacky" solution.
For the relevant code in memcached see:
https://github.com/memcached/memcached/blob/master/scripts/memcached-init.
Finally, the change to the rest of services is for consistency.
Some noteable diffrence from circleci:
- We upgrade git to newer version (reason explained in comments)
- We set HOME to /home/github (also explained in comments)
- Adjust permissions (... comments)
- Minor changes to step names and cache keys.
- We don't need to port the save_cache steps they are done
automatically in actions. And, we did not port the
store_test_results step which is circleci specific.
- We didn't port the notify_failure step yet (see the TODO).
Log RealmAuditLog in do_set_realm_property and do_remove_realm_domain.
Tests for the changes are written in test_events because it will save
duplicate code for test_change_realm_property.
Added new Event Type in AbstractRealmAuditLog STREAM_CREATED.
Since we finally create streams in create_stream_if_needed function
in zerver/lib/streams.py so logged realm_audit there.
Passed acting_user when create_stream_if_needed or ensure_stream
function is called.
Added tests in test_audit_log.
This commit moves out the SoftDeactivationMessageTest out of
test_messages.py (which at the moment have mixed category of tests) into
a more logical file, test_soft_deactivation.py.
This commit moves TestMessageForIdsDisplayRecipientFetching class which
have tests regarding display_recipient filled in by MessageDict to
test_message_dict.py.
This commit moves InternalPrepTest test class to test_message_send.py
because it tests internal_send_* and internal_prep_* functions which are
used for internal message sending in zulip.
This commit moves few tests related to testing proper sending of private
messages from PrivateMessagesTest class in test_messages.py to a new class
in test_message_send.py.
The commit moves, test_create_mirror_user_despite_race which is not related
to message sending from MessagePOSTTest class in test_message_send.py to
test_mirror_users.py.
This commit extracts out MessagePOSTTest class from test_messages.py
intially.
In future commits other related message sending tests will be moved from
test_messages.py to test_message_send.py.
Starting with extracting out MirroredMessageUsersTests as it is related to
mirror users than anything message-specific.
In a future commit, may extract out some tests from MessagePOSTTest as well
but still deciding on those.
To make the typeahead code more readable, we extract this function to
timerender. We also improve the logic to be more readable, and add tests
to confirm its validity.
We have moved our invalid timestamp logic to use timestamp-error class,
however, if there are any valid outputs by the backend markdown that
the frontend considers invalid, we want to debug them. This commit
adds tooling to ensure we log those error messages.
We had been using !time() syntax for timestamps so far. Since its
an unreleased feature, we can make changes without affecting many
people.
Fixes#15442.
This commit fixes a bug where clicking on a stream row on the left
in the subscriptions table called the ".stream-row" as well as the
".sub_unsub_button" click events in `stream_edit.js`.
This caused the stream subscription to toggle everytime the row was
clicked. Also, this bug is only observed if the ".sub_unsub_button"
had been clicked first.
The previous element passed was a simplebar container which calculated
the incorrect height, thus `scroll_util.scroll_element_into_container`
had no effect.
There is a bug and race issue that occurs when a message is selected
while we are in the process of reifying a locally echoed message,
raising the "Selected message id not in MessageList" error.
The code flow to get the exception is as follows:
* A user sends a message to the current narrow we are in.
* Before the new message event is received, we sent a message to
the same message list which renders it with a locally echoed id.
* One of the ways of getting the exception is to already have the
locally sent message selected, before receiving an acknowledgment
from the server.
* Thus the Message List Data's `selected_id` now points to the new
message id. The exception is raised on entering the `was_selected`
if block inside `message_list_view` which tries to re-select the
message.
Updating the `_rerender_message` code for this special case won't fix
the entire bug because, as mentioned above there are other ways of
getting the exception:
Ideally, after all our synchronous work (`echo.process_from_server`)
has completed we would expect the re-order and re-render work of the
`change_message_id` would occur first, due to the timer of the
setTimeout being set to 0.
However as evident from the race condition existing, this isn't always
the case. `change_message_id` function is responsible for 3 things:
updation, re-ordering and re-rendering.
The first one which is responsible for updating the message list's
local cache, occurs synchronously while for the latter two, they both
occur asynchronously.
Before the setTimeout which is responsible for the latter two actions,
is encountered the user might select the message by clicking or more
commonly by scrolling, which causes this message selection event to be
ahead of the setTimeout in the callback queue.
During this time frame, our race condition takes place.
And even though the message id is updated it's Message List is not
in the correct sort order, which leads to `closest_id` !== `id` in
`MessageList_select_id` being true and raising the exception.
Now, we only asynchronously call the re_render function, to guarantee
the data is always correct and UI updates should be done at the end.
Extended by tabbott to comment the setTimeout call.
Fixes#15346.
We change validate_stream_message to check the existence of stream from
the stream name in compose box early and we then pass stream_id or the
obtained sub objects accordingly to other validate functions.
Passing stream_id or sub objects to these functions, enables us to use
stream_id instead of stream name in stream_data.get_subscriber_count.
stream_data.get_stream_post_policy is also removed as we only used it in
validate_stream_message_policy, but we do not need it now as we can get
stream_post_policy directly from sub object obtained by early check of
valid stream name.
This commits add data-stream-id attribute to the compose_invite_users
template. This helps in avoiding the error that occured if user
clicked the link after renaming of stream.
As a result of above changes, the checks for empty and invalid stream
name in compose box are done in warn_if_mentioning_unsubscribed_user
function instead of needs_subscribe_warning function.
Prior to this commit whenever convert was imported from zerver.lib.markdown
it was aliased as markdown_convert for readability.
This commit rename convert function to markdown_convert so that it can be
directly import it without aliasing and without compromising readability.
If a channel has a thousand subscribers this commit results in the count
being displayed with a thousands separator, e.g. with English locale you
get 1,000 instead of 1000.
Since we migrated to `image_upload_widget.hbs`
for upload widget's so we have to access those widget's elements
according to the `image_upload_widget.hbs` new CSS class names.
We need to access delete button element with
`#user-avatar-upload-widget .settings-page-delete-button`
not with old CSS id `#user_avatar_delete_button`.
When we extract common HTML template (`image_upload_widget.hbs` )
for user avatar, realm day/night logo and realm icon widget's
lot of new CSS are created to match image_upload_widget.hbs and
old CSS are preserved in `settings.scss`. This commit removes all
unwanted or unused CSS in `settings.scss`.
Without this change, pyflakes reports this exception:
pyflakes | zerver/worker/queue_processors.py:152:9 local variable 'e' is assigned to but never used
pyflakes | zerver/worker/queue_processors.py:155:81 undefined name 'e'
We use the EMAIL_TIMEOUT django setting to timeout after 15s of trying
to send an email. This will nicely lead to retries in the email_senders
queue, due to the retry_send_email_failures decorator.
smtlib documentation suggests that socket.timeout can be raised as the
result of timing out, so in attempts I'm getting
smtplib.SMTPServerDisconnected. Either way, seems appropriate to add
socket.timeout to the exception that we catch.
Added -d Flag in do-release-upgrade for Bionic to Focal upgrade.
The -d switch is necessary to upgrade from Ubuntu 18.04 LTS
as upgrades have not yet been enabled and will only be enabled
after the first point release of 20.04 LTS.
Source https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
Otherwise, the useradd command will fail during the DigitalOcean
1-Click App installation because the install script is called
twice during the whole process. Plus the Zulip install script
is designed to be idempotent and this bug compromises that.
For the email and full name modals we simply change width to max-width.
The password modal used a flex-row class for no apparent reason,
the class wasn't used anywhere else and removing it fixes the UI bug.
Fixes#15311.
The forms to change plan_type, add discount, scrub_realm etc
all post to the same endpoint.
Our frontend code is written so that only one form posts at a time.
But there should be no harm in enforcing the same in backend as well.
While highlights was the only case where this came up in the normal
message feed, we might in the future have other elements inside a
link, and we don't want those to ever eat the click.
I don't think there's a real performance concern here; fundamentally,
this is the handler for the user clicking, it can traverse a few DOM
elements.
Adds support for clicking anywhere on the header (except in a link) to
expand a spoiler block. Also fixes jumpy animation glitch on spoiler
collapse.
Fixes#15414.
Co-authored-by: Sara Gulotta <sgorecki@gmail.com>
The value is a holdover from when it controlled runtime behavior,
which it no longer does.
Stop taking a DEPLOYMENT_TYPE, which is unused; the python code only
care about if the option exists, not its value.
These are more correct to the sense of "is this a service we
configured for Zulip", and removes potential confusion around the 0/1
values being backwards from how binary is usually interpreted.
Using checks of `,$PUPPET_CLASSES,` is repetitive and error-prone; it
does not properly deal with `zulip_ops::` classes, for instance, which
include the `zulip::` classes.
As alluded to in ca9d27175b, this can be fixed by inspecting the
classes that would be applied, using `puppet --write-catalog-summary`.
We work around the chicken-and-egg problem alluded to therein by
writing out as complete `zulip.conf` as would be necessary, before
running puppet and removing the sections we then know to not be
needed.
Unfortunately, there are two checks for `$PUPPET_CLASSES` which cannot
be switched to this technique, as they concern errors that we wish to
catch quite early, and thus before we have puppet installed. Since we
expect failures of those to only concern warnings, and only be
mistakenly omitted for internal `zulip_ops::` classes, this seems a
reasonable risk to admit in exchange for catching common errors early.
This fixes our triggering a RabbitMQ event to send a push notification
to remove the empty set of push notifications, resulting from not
using the correct data structure to determine which message IDs to look at.
This was causing a lot of visible exceptions when running
the `test_messages.py` test suite.
For users who are unsubscribed from the new stream but are in
the old stream, we delete the UserMessage.
We send the delete_message event only to guest users,
who have completely lost asses to the moved messages, for other
users we send the normal update_message event which moves
the messages to the new unsubed stream which
otherwise would look broken to the
user without reloading to the webpage.
Our previous OpenAPI schema validator that we implemented ourselves
was useful training wheels for our understanding OpenAPI properly, and
was mostly correct. But given that we've finally reached the point
where our OpenAPI file accurately describes the API, it makes sense to
switch to use an official OpenAPI validator. We lose some ability to
do exclude rules for particular elements, but those were primarily
important for us when we had a lot of them.
As part of this change, we need to add `additionalProperties: false`
for all of our dictonaries/objects where we've documented every
parameter; otherwise the OpenAPI schema checker won't know that we
expect every parameter to be documented.
This was hiding an actual type error in test_cache: a mismatch between
the object ID type, which is str, and the default id_fetcher, which
returns int.
Mypy’s insufficient support for default generic arguments basically
means we can’t use them without a lot of overloading, and there are
not enough callers here to justify that.
https://github.com/python/mypy/issues/3737
We avoid this being super messy where the code calls this by adding
some less generic wrappers for generic_bulk_cached_fetch.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When supervisor is first installed, it is started automatically, and
creates the socket, owned by root. Subsequent reconfiguration in
puppet only calls `reread + update`, which is insufficient to apply
the `chown = zulip:zulip` line in `supervisord.conf`, leaving the
socket owned by `root` and the last part of the installation unable to
restart `supervisor` services as the `zulip` user. The `chown` line
in `scripts/lib/install` exists to paper over this.
Add a separate exec target for changes to `supervisord.conf` itself,
which restarts the full service. This leaves the default `restart`
action on the service for the lightweight `reread + update` action,
which is more common.
We use `systemctl` only on redhat-esque builds, because CI runs
Ubuntu, but init is not systemd in that context. `systemctl reload`
is sufficient to re-apply the socket ownership, but a full `restart`
and not `reload` is necessary under `/etc/init.d/supervisor`.
According to @showell:
> All the slow decorators can die. That was a failed experiment of
> mine from 2014 days. I have meaning to kill them for a couple years
> now. I wrote this with the best of intentions, but I believe it's
> now just cruft. We never made a "fast" mode, for one. And we kept
> writing more and more slow tests, haha.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
I checked that this does not interfere with the MRO of the auth
backends:
In [1]: import zproject.backends; zproject.backends.GitHubAuthBackend.__mro__
Out[1]:
(zproject.backends.GitHubAuthBackend,
zproject.backends.SocialAuthMixin,
zproject.backends.ZulipAuthMixin,
zproject.backends.ExternalAuthMethod,
abc.ABC,
social_core.backends.github.GithubOAuth2,
social_core.backends.oauth.BaseOAuth2,
social_core.backends.oauth.OAuthAuth,
social_core.backends.base.BaseAuth,
object)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes the click target for the gear icon by providing
a proper rectangular area around it.
Minor UI adjustments of gear icon and expanded
navbar-search for small size devices.
Fixes: #15222.
When the keyboard focus is not on search box and user clicks on
search box, the focus will go move away from the search box to
the `current_focus_elem` on any rerender. To avoid that, we
move `current_focus_elem` to the search box when user clicks on it.
Since we had extracted `image_upload_widget.hbs` HTML for
image upload widget's like user avatar, realm logo, realm icon
we can also extract `image_upload_widget.scss` SCSS file
from settings.scss file with all the CSS related
to image upload widget's.
This change will also help us to keep `settings.scss` cleaner.
We remove support for the old clients which required an event for
each message to clear notification.
This is justified since it has been around 1.5 years since we started
supporting the bulk operation (and so essentially nobody is using a
mobile app version so old that it doesn't support the batched
approach) and the unbatched approach has a maintenance and reliability
cost.
wal-g has a slihghtly different format than wal-e in its `backup-list`
output; it only contains three columns:
- `name`
- `last_modified`,
- `wal_segment_backup_start`
..rather than wal-e's plethora, most of which were blank:
- `name`
- `last_modified`
- `expanded_size_bytes`
- `wal_segment_backup_start`
- `wal_segment_offset_backup_start`
- `wal_segment_backup_stop`
- `wal_segment_offset_backup_stop`
Remove one argument from the split.
Previously, this was unconditional wrt local echoe which caused a bug
where by the pencil edit icon was visible first, then got hidden when
the chevron and star became visible.
This commit resolves the above bug, but this area still behaves
slightly weirdly in that, if, within a PM narrow, the user's mouse
cursor is over the spot where the message edit controls will appear
after sending, and then a message is sent by the user, only the
chevron and star appear at first. On cursor exit and reenter, all
three controls reappear ie the final state is correct.
Similar to the other "unless locally echoed" cases here, this will
probably prevent unexpected bugs caused by eg trying to enter edit
state before the message was received by the server, and then
redrawing later, etc.
This commit changes the name of fixture that uses reference to bugdown.
Word backend in backend_markdown is important so to make it clear that
it is backend markdown. These test fixtures are also used in frontend,
so highlighting this is useful.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
This commit removes bugdown alias and do proper imports from markdown
module. Also remove bugdown word and replace it with markdown in
comments.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
In Bionic, nagios-plugins-basic is a transitional package which
depends on monitoring-plugins-basic. In Focal, it is a virtual
package, which means that every time puppet runs, it tries to
re-install the nagios-plugins-basic package.
Switch all instances to referring to `$zulip::common::nagios_plugins`,
and repoint that to monitoring-plugins-basic.
Frontend hosts in multiple-host configurations (including docker
hosts) need a `psql` binary installed. ca9d27175b switched to not
setting `postgresql.version` in `zulip.conf`, which in turn means that
`$zulip::base::postgres_version` is unset. This, in turn, led to the
frontend hosts installing `postgresql-client-`, whose trailing dash
causes apt to _uninstall_ that package.
Unconditionally install `postgresql-client` with no explicit version
attached. This is a metapackage which depends on the latest client
package, which currently means it will install `postgresql-client-12`.
On single-host installs which have configured `postgresql.version` in
`zulip.conf` to be a lower version, this will result in
`postgresql-client-12` existing alongside another
version (e.g. `postgresql-client-10`); `psql` will give the most
recent. This is acceptable because the semantic meaning of the
postgresql version in `zulip.conf` is about the database engine
itself, not the command-line client.
This changes the notification messages for events that currently just
include the string `"the repository"` to also include the full (`org/repo`)
name of the affected repository. Messages for the following events are
changed:
- `public`
- `star`
- `watch`
- `repository`
- `team_add`
Background: we're using the GitHub integration for org-wide notifications
for the [Bytecode Alliance Zulip](bytecodealliance.zulipchat.com/), and
having all messages just say "the repository" isn't ideal. Even now one
can hover over the link to see the repo's url, but it'd be much nicer if
the message just contained the full name.
I also changed the message for `star` to include a link to the repository,
same as the `watch` notification.
I also fix the code formatting so it's more
considerate of folks that have smaller monitors
or do side-by-side editing. And it's more
diff friendly as well.
I want to avoid creating the same scheme checkers
for multiple tests, and it's easiest to create the
schema checkers at module scope (and we possibly
even move them to another file eventually).
This will allow us to more easily instrument our
code to find duplicate schemas.
The goal here is to make test_events.py be
primarily focused on testing specific actions
and then validating:
- schemas
- apply_events logic
Then the new module here is a bit of a
kitchen sink of old tests, although it's
primarily focused on the actual mechanics
of the event system:
- logging
- register
- queue_ids
- fetching initial state
- client descriptors
The classes toward the bottom arguably
should go into more feature-specific
test modules, but the main goal now is
to purify test_events.py. (We may eventually
want to rename test_events.py to something
more like test_action_events.py, but the
current name has some doc references and
tribal knowledge around it.)
Escape all the possible special characters.
We replaced \b with (?:^|\s) since it matches word boundries including
special characters.
Pasted relevant stackoverflow links which expain them properly.
This commit removes unnecessary code for adding and removing
"can_edit" class from emoji-settings element. This class is
unnecessary and is not used for styling or for any other purpose.
This commit removes the ".not(can-edit)" selector filter from
insert_tip_box. We do not require this selector filter because
this class is only used for emoji-settings and we already do not
prepend tip in emoji-settings page from this function and we handle
it separately.
As this class is only used for this purpose we can also remove this
class from emoji_settings_admin.hbs.
The organization_settings_tip is not visible if organization settings
overlay is opened with any section other than organization profile,
settings and permissions. This is because insert_tip_box is called from
settings_org.build_page, which is called only when we open any of the
above three sections after opening the overlay and not others.
We should call insert_tip_box function from admin.build_page instead
of settings_org.build_page because we need to insert the admin tips
each time the organization settings overlay is opened, irrespective
of the section which opens first.
The function insert_tip_box is moved to admin.js from settings_org.js,
because settings_org.js file handles the organization profile,
settings and permissions page only, while we display the tips in many
other sections including bots, custom emoji, etc.
Thus, it makes sense to move insert_tip_box function to admin.js, which
renders the complete organization settings overlay using render_admin_tab.
The current description of object and array parameters in
zulip.yaml is wrong and renders incorrect requests on using OpenAPI
tools such as SwaggerIO, etc. Fix it by encoding it correctly and
changing tests accordingly.
Fixes#14380.
There is still some miscellaneous cleanup that
has to happen for things like analytics queries
and dead code in node tests, but this should
remove the main use of pointers in the backend.
(We will also still need to drop the DB field.)
This test is poorly written, in that it doesn't actually do any
verification of the results, but this at least is the correct
conversion of the data setup for it such that if we did verify its
results, the data we populated was relevant.
When we are just building lots of UserActivity
records to simulate user activity, it's misleading
to use some specific endpoint that people reading
the test may think is pertinent to the actual
test.
The pointer endpoints that I replaced in this
commit are a good example of this principle--
they will no longer exist in an upcoming commit,
but these tests would have kept running and be
even more confusing.
The isort author accidentally fixed the performance regression in his
latest commit. Quickly upgrade isort before he notices his mistake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Rename rest of function names, classes and comments containing bugdoown
to markdown in test_markdown.py. Also change the refactored classes and
functions occurences in other files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
rename ZEPHYR_MIRROR_BUGDOWN_KEY to ZEPHYR_MIRROR_MARKDOWN_KEY and
DEFAULT_BUGDOWN_KEY tp DEFAULT_MARKDOWN_KEY.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
Preparatory commit before removing bugdown alias for markdown. This
will prevent same variable name errors when name markdown is used
instead of bugdown.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
This commits changes class name of MarkdownListPreprocessor to
MarkdownListPreprocessor. It also changes corresponding references
in tests.
This is part of series of commits which aims for renaming bugdown to
markdown.
Rename the file and all the refrences to file and module test_bugdown.py
to test_markdown.py.
This commit is part of series of commit that renames bugdown to markdown.
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
This file was generated by GitHub's code analysis tutorial; we were
just approved from their waitlist.
I deleted the part to run compilers as it is not relevant for us.
We send user_id of the referrer instead of email in the invites dict.
Sending user_ids is more robust, as those are an immutable reference
to a user, rather than something that can change with time.
Updates to the webapp UI to display the inviters for more convenient
inspection will come in a future commit.
As alluded to in the previous commit, only 3.0 can use the new tooling
-- indeed, it requires it, as the zulip.conf entry must be changed.
Clarify that in the upgrade steps for earlier distributions.
Update the upgrade documentation for the tool added in the previous
step. Only the Bionic -> Focal upgrade step need be updated, because
none of the other upgrade steps can be run starting from a Zulip 3.0
installation.
Fixes#15415.
This is based on the existing steps in the documentation, with
additional changes now that the PostgreSQL version is stored in
`/etc/zulip/zulip.conf`.
In zulip.yaml, add `deprecated` tags to all parameters/keys with
`Deprecated` in the description. Then add tests to ensure that deprecated
parameters/keys will always have the `deprecated` key. Also, in
the API docs, sort the parameters according to presence of `deprecated`
key, presenting the `deprecated` keys at the end and add a `deprecated`
tag next to them.
This commit removes "realm_message_retention_days === null" check from
the conditionals, as we had updated the backend to replace NULL value
with -1 in 7a03e2a.
This commit fixes the UI for stream message retention days to allow
only owners to set or update the setting.
We hide the setting for non-owners in the stream creation form and
disable it in the stream_privacy_modal.
Fixes a part of #15558.
Change variable `name` to `date_sent` as `name` actually stores
the date sent. Also change the data types of `name` and `create_time`
to integer. As they actually have empty decimal value.
Due to authentication restrictions, a deployment may need to direct
traffic for mobile applications to an alternate uri to take advantage
alternate authentication mechansism. By default the standard realm URI
will be usedm but if overridden in the settings file, an alternate uri
can be substituted.
We send a remove mobile push notification to the users who were
no longer mentioned after the content of the message was edited.
This also corrects the notification count for the mobile apps
where a user was prior mentioned in a muted stream / topic and the
message was edited and the user is no longer mentioned now.
Hence, fixing the case where user has read all his unreads
but the notification badge on the app is still positive.
Fixes#15428.
do_clear_mobile_push_notifications_for_ids can now be used to
clear push_notification for multiple users at once. This method
loops over users, so no performance optimization is gained.
We've been seeing an exception in server_event_dispatch.js in
production where in large organizations, sometimes when a new user
joined, every other browser in the organization would throw an
exception processing some sort of realm_user/update event.
It turns out the cause was that when a user copies their profile from
an existing user account with a user-uploaded avatar, the code path we
reused to set the avatar properly send a realm_user/update event about
the avatar change -- for a user that hadn't been fully created and
certainly hadn't have the realm_user/add event sent for.
We fix this and add tests and comments to prevent it recurring.
(Removed an incorrect docstring while working on this).
The restart event was always handled pretty similarly
to pointer, so I use restart events now for this
test (in preparation of eliminating pointer events).
This eval function performs the inverse of the implicit
stringification that’s implied by this type-incorrect assignment in
do_update_user_custom_profile_data_if_changed:
field_value.value = field['value']
We believe there’s sufficient validation for the data being passed to
this eval that it could only have been exploited by a PostgreSQL
administrator editing the database manually.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Tim thought commit 51acca2672 was needed
because he had an extra social_django directory in his checkout.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes an issues that causes HTML entities inside of inline code
blocks to be converted rather than being displayed literally.
The upstream python-markdown now handles this correctly, so we just use
their implementation with our changes for removing .strip(). As a result
of this migration, we switch backtick pattern to an inline processor
too.
Fixes#12056.
For the codeblock counterpart of this issue, we should follow the
upstream PR https://github.com/Python-Markdown/markdown/pull/990.
Co-authored-by: Rohitt Vashishtha <aero31aero@gmail.com>
* Reordered the settings relevant without stream creation to the top.
* Removed useless/misleading defaults for optional parameters.
* Clarified description of the announce and authorization_errors_fatal settings.
* Clarified that `invite_only` only applies for stream creation.
(It's annoying to do so for its friends because they are including
common description content and OpenAPI doesn't have a way to have
extra content in a place you included something)
Fixes#14705.
Now we are consistent about validating color/description.
Ideally we wouldn't need to validate the
`streams_raw` parameters multiple times per
request, but the outer function here changes
the error messages to explicitly reference
the "delete" and "add" request variables.
And for the situation where the user-supplied
parameters are correct, the performance penalty
for checking them twice is extremely negligible.
So it's probably fine for now to just make sure
we use the same validators in all the relevant
places.
There's probably some deeper refactor that we
can do to eliminate the whole `compose_views`
scheme. And it's also not entirely clear to
me that we really need to support the update
endpoint. But that's all out of the scope of
this commit.
Note that I don't actually convert the
checker from check_dict to check_dict_only,
because that would be a user-facing change,
but I think we can sweep a lot of things
like this after the next release.
This avoids some code duplication as well
as adding some missing fields.
We also use check_dict_only to prevent
folks from adding new fields to the
relevant events without updating these
tests. (A bigger sweep comes later.)
As the code comment indicates, we just
use a strict check here rather than
pretending that the test exercises a
more complicated schema for the config
data, which is dynamic in nature.
Cleaning up config_data is outside the
scope of this PR; my main goal is to
eliminate check_dict calls (usually in favor
of check_dict_only).
Because of other validation on these values, I don't believe any of
these does anything different, but these changes improve readability
and likely make GitHub's code scanners happy.
The helper should be used instead of constructing the dict manually.
Change get_account_data_dict, on GitHubAuthBackendTest
class, so it has a third argument, user_avatar_url.
This is a preparation for support using GitHub avatar
upon user resgistration (when the user logs using
GitHub).
If your browser width was between 701px and 750px you got the mobile
view without the mobile header preventing you from changing sections in
the settings menu.
This was caused by a media-query mismatch:
subscriptions.scss used @media (max-width: 750px)
settings.scss however used @media (max-width: 700px)
Comments added by tabbott to help avoid future bugs like this.
* Don't annoyingly open the first section when switching
between the Settings and Organization tabs.
* Don't highlight currently active section in the settings list
(we don't display the currently active section in the mobile settings
list so it isn't actually active).
* Remove nearly invisible and buggy no-border logic.
Update the REQ check for profile_data in
update_user_backend by tweaking `check_profile_data`
to use `check_dict_only`.
Here is the relevant URL:
path('users/<int:user_id>', rest_dispatch,
{'GET': 'zerver.views.users.get_members_backend',
It would be nice to unify the validator
for these two views, but they are different:
update_user_backend
update_user_custom_profile_data
It's not completely clear to me why update_user_backend
seems to support a superset of the functionality
of `update_user_custom_profile_data`, but it has
this code to allow you to remove custom profile fields:
clean_profile_data = []
for entry in profile_data:
assert isinstance(entry["id"], int)
if entry["value"] is None or not entry["value"]:
field_id = entry["id"]
check_remove_custom_profile_field_value(target, field_id)
else:
clean_profile_data.append({
"id": entry["id"],
"value": entry["value"],
})
Whereas the other view is much simpler:
def update_user_custom_profile_data(
<snip>
) -> HttpResponse:
validate_user_custom_profile_data(user_profile.realm.id, data)
do_update_user_custom_profile_data_if_changed(user_profile, data)
# We need to call this explicitly otherwise constraints are not check
return json_success()
This tightens our checking of user-supplied data
for this endpoint:
path('users/me/profile_data', rest_dispatch,
{'PATCH': 'zerver.views.custom_profile_fields.update_user_custom_profile_data',
...
We now explicitly require the `value` field
to be present in the dicts being passed in
here, as part of `REQ`. There is no reason
that our current clients would be sending
extra fields here, and we would just ignore
them anyway, so we also move to using
check_dict_only.
Here is some relevant webapp code (see settings_account.js):
fields.push({id: field.id, value: user_ids});
update_user_custom_profile_fields(fields, channel.patch);
settings_ui.do_settings_change(method, "/json/users/me/profile_data",
{data: JSON.stringify([field])}, spinner_element);
The webapp code sends fields one at a time
as one-element arrays, which is strange, but
that is out of the scope of this change.
The flake was caused due to the fact that current_msg_list was not
populated with any messages in rare cases. We were missing a check
to guard against current message list having no messages. The
failure screenshot show no messages to prove this.
Relevant error:
Evaluation failed: TypeError: Cannot read property 'raw_content' of undefined
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release. This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
Ubuntu 20.04 "focal" comes up to runlevel 5 several seconds before it
is able to successfully resolve hosts, causing `prepare-base` to fail
while fetching from the apt repositories.
Add an additional check to verify that outbound networking is running
before returning from `lxc-wait`.
`/api/v1/fetch_api_key`'s response had a key `email` with the user's
delivery email. But its JSON counterpart `/json/fetch_api_key`, which
has a completely different implementation, did not return `email` in
its success response.
So to avoid confusion, the non-API endpoint, `/json/fetch_api_key`
response has been made identical with it's `/api` counterpart by
adding the `email` key. Also it is safe to send as the calling user
will only see their own email.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.
We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here. Now we also want to deprecate
check_tuple, but it's at least isolated now.
We will use this for data structures that are tuples,
but which are sent as lists over the wire. Fortunately,
we don't have too many of those.
The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
This commit changes do_get_user_invites function to not return
multiuse invites to non-admin users. We should only return multiuse
invites to admins, as we only allow admins to create them.
As in the previous commit, we can no longer pre-install the wrong
version of postgres. Unfortunately, this leaves it out of the base
image and thus makes testing installs longer.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository. On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.
During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found. This is safer than attempting to automatically uninstall or
reconfigure existing databases.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed. Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL. This leads to failures in first-time installs:
```
Error: Evaluation Error: Error while evaluating a Function Call,
Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```
The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.
To do this, we must inspect `PUPPET_CLASSES`. Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`). We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.
Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.
Remove all code predicated on Xenial or Stretch. debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.
Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.
Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.
Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
The previous architecture did not work properly with the automatically
detected night theme, resulting in a weird mix of the night and day
themes on code blocks.
I'm not thrilled with the requirement this imposes that all of our
night theme CSS needs to be in one file, but we do need to get a quick
fix out here.
Fixes#15554.
This commit removes invited_as_values map in settings_invites.js.
This object has been removed to avoid duplication as we already
have role values in settings_config.js.
A similar map is created from settings_config.user_role_values
in settings_config.js and is used to populate invited_as_text
for invites.
This commit changes the PreregistrationUser.invite_as dict to have
same set of values as we have for UserProfile.role.
This also adds a data migration to update the already exisiting
PreregistrationUser and MultiuseInvite objects.
We leverage the composebox typeaheads to show flatpickr to pick dates
and times for the !time syntax.
We use moment.js to try and parse the time from current token. If we
are successful, we initialize flatpickr with the parsed time, else we
default to using the current time.
Streams can have lots of subscribers, meaning that the archiving process
will be moving tons of UserMessages per message. For that reason, using
a smaller batch size for stream messages is justified.
Some personal messages need to be added in test_scrub_realm to have
coverage of do_delete_messages_by_sender after these changes.
Currently we display -1 in input box of id_realm_message_retention_days
when realm_message_retention_days is -1, which isn't user friendly.
Displaying the input box as empty is more intuitive.
And if the user tries to submit an empty input box we throw invalid JSON
error that isn't user friendly either, so fixed that too. In the ideal
case, we shouldn't send the request at first place to the backend when we
don't have any input.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means
> disable retention policy for this stream unconditionally
that can be confusing from an API standpoint.
So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
`get_input_element_value()` function is more reliable to detect the input
element type and extract it's value. But the current way of setting the
value of input elements relies on first checking the `property_value` type.
Which is fine, but for the cases when the property value is null, and we
want to set element value as empty, this method will throw an error as it's
unable to detect the appropriate element type. This new function
`set_input_element_value` first rely on property value and then use
`setting-widget-type` as a fallback.
This fixes the change detection of org settings input elements. Luckily,
this regression didn't break our populate_data_for_request and that's also
why this didn't come to notice.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.
This commit enables this feature and adds support for it to the web
client.
In particular the Services ID and Bundle ID each have one of Apple's
random-looking 10-character identifiers, in addition to the Java-style
names the admin chooses. Best to be clear about what names are
supposed to be the chosen names and which are supposed to be the
random-looking assigned names.
(I don't know of any docs elsewhere making this clear -- but I guessed
it'd be this way, and empirically it works.)
Also mention you need to enable the backend. :-)
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID. In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
subdomain=None didn't make much sense as a value, and wasn't actually in
use anywhere, except one test where it was accidental. All tests specify
the subdomain explicitly, so we should change the type to str, and make
it an obligatory kwarg.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.
Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Only some branches define the `ZULIP_BOT_KEY`; others, including PRs,
thus currently fail to POST:
{"result":"error","msg":"Malformed API key","code":"INVALID_API_KEY"}
Simply abort early, without attempting to report the result, if the
key is not present.
If the `load_messages_for_narrow` function tried to fetch messages with
a floating point anchor (locally echoed message id) we get a 400 error.
The logic to remove the rounding of the anchor parameter was done in
commit bf2f36e6b4, as it would always be
an integer in all incoming code paths.
However, since then we have updated the GET /messages API, to also allow
strings as the anchor parameter and this value's parsing is done in the
backend.
Fixes#15497.
We update the logic for displaying the "message is outside current
narrow" notice, by comparing whether the message the client sent can
be appended to the current narrow's filter or not (this results in us
correctly falling through to reporting the user needs to scroll down).
This migrations use of url() to path() or re_path(). In this commit,
we only migration regular expressions to path where the translation is trivial:
* URLs with no parameters in them
* URLs with only integer parameters in them
* Strings where there regular expression just checked for `/`s
path; strings, which can have variable validation in the URLs that
need by-hand auditing, we leave for future commits that are easier to
review and think about the individual changes.
Modified by tabbott to convert back to `re_path` various URLs with
strings that had been converted to use `path()` with string
validation to simplify review.
Fixes#14770.
Also, `send_message` example is altered to send a message to the
stream 'social' to avoid getting a "first_message_id: null"
in the response for `get_subscriptions` example, that caused
`validate_against_openapi_schema` to throw an error.
Previously in desktop view, the realm day/night logo element is arranged
one by one which is not looking good since we have a lot of space on the
the right side of the logo elements so we can move day/night logo elements
to allow them being alongside.
In mobile view, we don't have any space on the right side of the logo
elements so we don't have to change anything.
Since `--postgres-missing-dictionaries` edits `/etc/zulip/zulip.conf`,
it interferes with the intent of `--no-overwrite-settings`.
Make the two settings conflict, to prevent this unclear state.
The `--no-init-db` option previously only controlled if
`initialize-database` was run, which sets up the tables inside the
database. If PostgreSQL was installed locally, it still attempted to
create the user and empty database.
This fails on hosts which are remote PostgreSQL hosts, and not
application hosts, as:
- They may already have a local database, and while
`initialize-datbase` will detect and offer to abort if one is
found,`--no-init-db` seems like it should be the option to not
overwrite it
- `flush-memcached` requires that a local venv be installed, which it
often is not on non-frontend machines.
Skip the database configuration when run with `--no-init-db`.
The `EXCLUDE_PROPERTIES` is a dictionary in `zerver/openapi/openapi.py`
which holds the undocumented properties of our API. Document all
properties other than:
*`delivery_email` which is in another PR.
*'events' and 'register'.
*'/setting/notification' since its response is about to undergo heavy
changes.
A generator that yields values without receiving or returning them is
an Iterator. Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In a decorator annotated with generic type (ViewFuncT) -> ViewFuncT,
the type variable ViewFuncT = TypeVar(…) must be instantiated to
the *same* type in both places. This amounts to a claim that the
decorator preserves the signature of the view function, which is not
the case for decorators that add a user_profile parameter.
The corrected annotations enforce no particular relationship between
the input and output signatures, which is not the ideal type we might
get if mypy supported variadic generics, but is better than enforcing
a relationship that is guaranteed to be wrong.
This removes a bunch of ‘# type: ignore[call-arg] # mypy doesn't seem
to apply the decorator’ annotations. Mypy does apply the decorator,
but the decorator’s incorrect annotation as signature-preserving made
it appear as if it didn’t.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Also enable warn_unused_ignores. I think the fact that there are so
few of these is good evidence that it’s not a significant burden for
people fixing type errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes these compilation errors from webpack and PostCSS, exposed by
commit b10f156250 (#14997) which tries
to @extend these directives:
Unexpected '}' at app.d5da4b9d46e79634b8fb.css:9103:4.
Unexpected '}' at app.d5da4b9d46e79634b8fb.css:9104:0.
Invalid property name '@-moz-document url-prefix() {
@nest & #settings_page select {
background-color' at night_mode.scss:788:0. Ignoring.
Invalid selector '}
}
.user_status_overlay .overlay-content' at night_mode.scss:797:4. Ignoring.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Migrations that add items to a table in the same migration htat
craetes the table require atomic=False.
This fixes this exception:
Running migrations:
Applying zerver.0287_clear_duplicate_reactions... OK
Applying zerver.0288_reaction_unique_on_emoji_code... OK
Applying zerver.0289_tighten_attachment_size... OK
Applying zerver.0290_remove_night_mode_add_color_scheme...Traceback (most recent call last):
File "/home/zulip/deployments/2020-06-22-23-20-36/zulip-py3-venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 33, in execute
return wrapper_execute(self, super().execute, query, vars)
File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 20, in wrapper_execute
return action(sql, params)
psycopg2.errors.ObjectInUse: cannot ALTER TABLE "zerver_userprofile" because it has pending trig
This adds a convenient way to review the upgrade notes for all Zulip
releases that one is upgrading across.
I thought about moving all the upgrade notes to a common section, but
in some cases the language is clearly explaining changes in the
release that are not duplicated elsewhere, and I think it reads better
having them inline alongisde related changes.
This is important because the Hashicorp boxes are better designed, and
in particular, https://www.vagrantup.com/docs/boxes.html recommends
using the hashicorp boxes over the "ubuntu" namespace.
This commit changes stream_data.create_sub_from_server_data to use
stream id, instead of stream name, for checking whether subscription
already exists or not. We are using stream ids so that we can avoid
bugs related to live update after stream rename.
This commit changes would_receive_message and want_normal_display
functions in compose_fade.js to use stream id, instead of stream
name, to get sub object. We are using stream ids so that we can
avoid bugs related to live update after stream rename.
This commit changes populate_group_for_message_container function to
use stream id to get sub object, instead of stream name.
We already have stream id from message object, so we can easily
use it to fetch sub. We are using stream ids so that we can
avoid bugs related to live update after stream rename.
This commit changes stream_data.remove_subscriber to use stream id
instead of stream name. We are using stream ids so that we can
avoid bugs related to live update after stream rename.
Thsi commit changes stream_data.add_subscriber to use stream_id
instead of stream name. We are using stream ids so that we can
avoid bugs related to live update after stream rename.
This commit removes stream_edit.rerender function. We directly
call subs.rerender_subscriptions_settings directly from
server_events_dispatch.js, which was the only caller of rerender
function, as we already have sub object.
We are using stream ids so that we can avoid bugs related to live
update after stream rename.
We can use get_sub_by_id instead of get_sub to get the stream info,
as we already have stream id from the message object. We are using
stream ids so that we can avoid bugs related to live update after
stream rename.
This commit changes receives_notifications function to use
stream_ids instead of stream names. We are using stream ids so
that we can avoid bugs related to live update after stream rename.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.
(And I expect we'll add more with time).
Prior to this commit has:link, has:attachment, has:image
filter couldn't be applied locally and deferred filtering to
web server. This commits make sure client filters all messages
it can instead of completely deferring to the server and hence
improve speed.
A tradeoff is also made to turn off local echo for has: narrows
as messages with link sent to has:link narrow were locally echoing
to another narrow and not appearing in the active has:link narrow.
Fixes: #6186.
Previously, the message controls had a bug where they would trigger on
mobile with a single tap over the area they occupy when visible. This
is wrong because a user would expect to first see the controls and
only trigger them once they are visible (with a second tap).
The above bug is caused by the fact that we were using "opacity: 0" on
".message_controls > div" to hide the controls and "opacity: 1" on
".messagebox &:hover .message_controls > div" to show the controls on
hover, however, this would not effect the click action because
"opacity". So we used "pointer-events: none;" and "pointer-events:
all;" with the hopes that it would prevented the above bug, but in
practice, it didn't.
(the most probable explanation being that tapping the message_control
area would cause the "&:hover" rule to trigger and change the
"pointer-event" to "all" before it could prevent the click trigger,
But that explanation is just conjecture.)
This commit replaces both "pointer-events" attributes with
"visibility: hidden" and "visibility: visible" respectively. The
result being that the message_controls behave identically to before,
except without the above bug.
The addition to the ".has_actions_popover .info" selector is important
because without it, we would regress on issue #3172.
Trivia:
An alternate approach to using "opacity" is to set the
"display" attribute to "none", however, using "display" prevents the
transition from animating (which is probably why we were using opacity
here in the first place). "visibility" does not prevent the transition
from animating.
History: The "pointer-events" attribute was introduced in
4d5aa3ddc9 and it replaced prior code
which relied on the "visibility" attribute... But it seems PR #3792
was mostly focused on improving the positioning through removal of
`display: none`, but introduced opacity to make the animations work
rather than visibility as the replacement solution, which requires the
pointer-events hack and resulted in the bug described here.
Fixes the second bug described in #13642.
This helps user understand "Include muted" filter better and
saves keypress to focus on more used filters like "participated"
and "unread".
See https://github.com/zulip/zulip/issues/15482.
Fixes#14960.
The default of 6 thread may not be appropriate in certain
configurations. Taking half of the numer of CPUs available to the
process will be more flexible.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.
This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.
Fixes: #14451.
Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
With the previous color it was hard to read the text and
also that color does not matches with the zulip style.
This commit changes the color of `name` field for
user-profile modal for better visability both in day
and night mode.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
Add arrow key navigation support for recent topics.
Simple jquery is used to allow navigation for filter buttons,
a grid system is used for navigation inside table.
Old: a validator returns None on success and returns an error string
on error.
New: a validator returns the validated value on success and raises
ValidationError on error.
This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit corrects the message shown when we click the add button
for subscribing users to stream with empty input.
We show 'No user to subscribe.' as the message when trying to add
subscribers with empty input.
Fixes#15450.
We have migrated ellipsis-v from chevron for sidebars,
it would be better to define it in glossary as it is gonna replace
"chevron" during conversations.
This improves the logic and fixes the bug where the href was calculated
based on the current URL and not the filter of the current message list.
We now add the '/streams/public/' operator at the start of the operators,
similar to how it is represented in all other cases.
Fixes#15405
The `.search_icon` lies inside the `.pill_container` so we had to
remove it's display attribute which should have no visual changes.
We add `flex-wrap: nowrap` to prevent the cursor from moving onto
a new line, below the pills.
Fixes#15480.
This commit removes the click handler used for
'empty_feed_sub_unsub' class.
This class was used only in home.html and was replaced
by 'stream_sub_unsub_button' in 576be51.
We had a bug in `validate_against_openapi_schema` that prevented it
from correctly inspecting nested arrays.
Fix the bug and address all the exceptions, either via
EXCLUDE_PROPERTIES or fixing them when simple. Also add a test case
for nested verification.
Attachment objects in production are only created in one place, which
passses a size. Additionally, I verified in multiple production
environments with old data that this never actually happens (or has
happened).
So we should make the data model correctly reflect the possibilities here.
Rename the validator to check_union, to conform
more to Python typing nomenclature.
And we rename one of the test helpers to the
simpler `check_types`. (The test helper
was using "variable" in the "var" sense.)
We assert that the post was successful, to give
more immediate feedback for tests that don't
bother to check the return value and may be
implicitly assuming this method just works in
all cases.
And we also make it more convenient for tests
that are happy-path tests--they don't have to
do the assertion themselves. (And they're still
free to do deeper checks on the json.)
We opt out with allow_fail=True. We probably want
a more direct API eventually for tests that are
clearly trying to test the failure path for
subscribing to streams.
It's possible that a couple tests here that I added
allow_fail=True to just have flawed data setup--
I don't have time to investigate all cases, but
hopefully they will at least stand out more.
This prevents memcached from automatically appending the hostname to
the username, which was a source of problems on servers where the
hostname was changed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reverts part of b0d632577f.
The problem was that multiple queries were combined as a single
search pill. And since we create the pills then narrow / search,
we added a comma seperator between them for the typeahead lookups
as required by the logic in `input_pill.js`.
This however introduced a new bug where the search suggestions
were incorrect as the typeahead lookup table wasn't updated, so
every time an item from the type ahead was selected it updated
the input string with an invalid operator.
Thus to resolve the first problem, we follow a simpler approach
by extracting all operators from the search string using our
`Filter.parse` logic and next add the pills, one by one.
Whenever a search pill is selected or deleted by a click the navbar
gets rendered as the searchbox loses focus. This allows the user to
be able to continue editing the search query without having to refocus
the searchbox.
A main change is that we now display the navbar if the search box
is not focused. This was already present in the search pills version
but adding it to the legacy version is an improvement.
We sufficiently increase the timeout so that the pills are actually
deleted. This was required when `filter.is_common_narrow()` is true,
as then only we render the narrow description and close the search bar.
This commit also matches another behaviour of the legacy search.
i.e. We narrow every time a search suggestion is clicked.
The now redundant "focusin" and "focusout" event handler tests are
also removed.
This change was only required for the search pills enabled
search. As there is a valid use-case where the user might
want to remove just the latest pill(s) and then narrow.
This wasn't possible previously because, the typeahead was
looked up every time the backspace key was pressed. And since
the only way to narrow in search is through the enter key,
if the user entered it then the searchbox would get updated
with the first suggestion in the typeahead.
The alternative for the user would be to first lose focus on
the searchbox (by clicking outside) the focus again which
doesn't generate the typeahead. Then only the enter key would
be available for narrowing.
We do not display the typeahead after the search pills are
created. This is done just to match the behaviour for the
deletion of pills case too.
We use this approach and we don't just change the line
in `search.js` from `helpOnEmptyStrings: true` to
`helpOnEmptyStrings: !page_params.searchpills enabled`
because we still need to provide suggestions for '',
on initial lookup or every time the empty input searchbox
with search pills present gains focus.
When we move the avatar upload widget to `image_upload_widget.hbs` file
the CSS `position:absolute` for `avatar-source` is preserved.
by removing CSS `position:absolute` we can fix the bug.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.
We can produce much more readable code by just passing an empty list
of locations in this case.
Two things were broken here:
* we were using name(s) instead of id(s)
* we were always sending lists that only
had one element
Now we just send "stream_id" instead of "subscriptions".
If anything, we should start sending a list of users
instead of a list of streams. For example, see
the code below:
if peer_user_ids:
for new_user_id in new_user_ids:
event = dict(type="subscription", op="peer_add",
stream_id=stream.id,
user_id=new_user_id)
send_event(realm, event, peer_user_ids)
Note that this only affects the webapp, as mobile/ZT
don't use this.
Currently the API docs do not specify whether a given API parameter
is to be specified in `query` or in `path`. Edit the docs so as
to show the type of argument right beside argument name.
We recently removed an API call for fetching user list for our
Settings>Users/Deactivated panels, which introduced a bug where
we rendered the users table before last active information is
processed by the frontend.
This commit makes us process presence before rendering our settings
panels. We move the presence init above because we need to initialize
settings_sections before hashchange.
Fixes#15453.
When the user clicks a button that opens a modal, and if we don't break off
the corresponding click event. This condition in the global click handler
will become true and enables all mouse events outside modal.
```js
$(document).on('click', function (e) {
...
// If user clicks outside an active modal
if ($('.modal.in').has(e.target).length === 0) {
// Enable mouse events for the background as the modal closes
$('.overlay.show').attr("style", null);
}
```
Related to #12369.
Currently, the OpenAPI extension for rendering description in docs
cannot parse {!api-admin-only.md!}. Edit order of markdown extensions
in app_filters.py so that rendering of OpenAPI elements takes place
before substitution of files using `include`.
The loop I added here in 5b49839b08 was
ill-conceived. The critical issue was that despite its name,
do_clear_mobile_push_notifications_for_ids does not immediately clear
push notifications (Except in our test suite, where `send_event`
immediately calls into the queue worker code!).
Instead, it queues work to clear those push notifications. Which
means that the first user to declare bankruptcy with a large number of
unreads will fill the queue, and then this will just be an infinite
loop adding more work to the queue.
This fixes a missing unique constraint on the Reactions data model
state when using multiple aliases for an emoji code. As with any
missing unique constraints, we first need to apply a migration that
eliminates violations of the rule; in this case, deleting the
duplicates is correct.
Added unique constraint for "user_profile", "message",
"reaction_type", "emoji_code".
Fixes#15347.
Mostly, this is a change in ordering to make more sense, but we also
fix several names that were clearly confusing.
We restore the convention that each endpoint has the same title at the
top of the page as what we have in the sidebar menu, which appears to
have been violated in many recent updates to API documentation.
api docs filenames are basically the operationId of their endpoint
in zulip.yaml with `_` replaced by `-`. But some operationIds have
changed, so change the affected filenames. Make changes in other
files accordingly.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.
Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.
This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.
Fixes#15287.
We need this field to avoid O(N) database operations
while fetching realm user data for clients with
`user_avatar_url_field_optional` flag enabled.
Part of #15287.
This extends get_accounts_for_email test by adding a deactivated
user and assert that get_accounts_for_email doesn't return any accounts
for that deactivated user.
Fixes#14807.
The zerver.models hack does not appear to be necessary now.
Meanwhile, get_wsgi_application has its own django.setup call, which
would overwrite the parts of our logging configuration pulled in by
zerver.models.
This fixes part of #15391; specifically, fixes it in production, but
not in development, where ‘manage.py runserver’ calls its own
django.setup and then imports various bits of our code before finding
zproject.wsgi.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
With #14378, we regressed back to the state of that
prior to 7e0ea61b00.
We fix this by getting our avatar bucket on
object initialization, and use the appropriate means
of gathering the network location for the urls.
Fixes#14484.
_setup_export_files modifies the zulip realm. We used to
call realm.refresh_from_db in tests after _setup_export_files was
called to make sure that the change is reflected. But sometimes
calling refresh_from_db was missed out here and there.
This commit makes calling refresh_from_db after _setup_export_files
unnecessary.
This commit adds message retention policy details in the subscription_type
text below the stream description.
We do not show any text when realm-level settings is set to forever and
stream-level is set to either forever or realm_default.
This commit adds frontend support for setting and updating message
retention days of a stream from stream settings.
Message retention days can be changed from stream privacy modal of the
stream and can be set from stream_creation_form while creating streams.
Only admins can create streams with message_retention_days value other
than realm_default.
This commit also contains relevant changes to docs.
This commits adds the code for live update of stream_post_policy in
subscription_type text in stream settings.
This is done by passing stream_data.stream_post_policy_values to the
template data, which were not passed previously and the if conditions
were not evaluated correctly.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.
'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
Because of a security release that required a migration, there
are two migrations numbered 0261. To avoid breaking existing
installs renumbering the migrations, we skipped migration 0261
when running tools/renumber-migrations.
Previously, we had implemented:
<span class="timestamp" data-timestamp="unix time">Original text</span>
The new syntax is:
<time timestamp="ISO 8601 string">Original text</time>
<span class="timestamp-error">Invalid time format: Original text</span>
Since python and JS interpretations of the ISO format are very
slightly different, we force both of them to drop milliseconds
and use 'Z' instead of '+00:00' to represent that the string is
in UTC. The resultant strings look like: 2011-04-11T10:20:30Z.
Fixes#15431.
There is apparently some way to have two instances
of `.emoji-popover-emoji-map`, although I can't
reproduce it.
This causes an `expectOne` check to fail fairly
deep in the stack.
Now we report it more directly.
This commit and a few previous ones mostly
address #15348 by trying to either a) not
depending on having a single instance of
the popover or b) making it more explicit
in cases where do expect that invariant.
Fixes#15348
This is just a pure refactor for now, but
we may want to modify this to more precisely
determine the active map (in case multiple
pickers are open for some reason).
This is clearly a better home for it, since message_scroll.js is the
only place that reads it, and also lets us provide a clearer name for
the functionality.
Since we are no longer using the "pointer" value sent in
page_params.pointer for anything, there's no value in continuing to
send it from the server to the client.
The remaining code in pointer.js is logic managing state for the
currently selected message.
Since the pointer is no longer used to set the browser's position, we
no longer need this complex code to send updates to the server during
the bankruptcy flow.
It's crazy that we need to do this; one would think that Electron apps
whose sole purpose is to be used with multiple team chat tools would
at least implement the standard desktop notification API correctly.
But it seems worth making this tactical change to prevent every
desktop notification throwing a traceback on those platforms, which if
nothing else results in a lot of error spam.
Fixes#15103.
This was broken when moving the code being called to another file.
This exception caused a pretty weird/nasty bug by interrupting the
message_fetch response handler before it finished updating the
fetch_status data strutures. The end result was that in views where
the "history limited notice" was displayed, local echo would be broken
a confusing notice would be displayed.
The term `parameter` is a better word than `argument` for data passed
to an API endpoint; this is why OpenAPI uses in their terminology.
Replace `argument` with `parameter` in the API docs to improve their
readability.
Fixes#15435.
In rare situations we would get tracebacks from
list_cursor on the line that I changed here. We
went the entire month of May without a traceback
here, and I can't reproduce the problem.
This is a pretty clear fix, though, and it will
hopefully lead to a more enlightening symptom.
The likely scenario here is that you use `q` to
navigate the stream list and then unsubscribe.
I tested that and couldn't get a traceback,
but I do think the traceback indicates some
possible issues.
The behavior I saw when I did this
appeared to be mostly harmless.
When I deleted a row (by unsubscribing), the code
seemed to effectively disable the cursor. It's
possible we should go to the next row or fully disable
the search.
I opened #15439 to follow up on this and other
cursor-related issues.
The stream_events tests were kinda messy, but
I mostly just consolidated a few sections of
code so that we didn't have to keep
re-stubbing the same functions.
For the actual code, I extracted add_sidebar_row
and then removed the unnecessarily complicated
jQuery trigger mechanisms.
This merges the `exports.get_search_result_legacy` and
`exports.get_search_result` function.
The key differences between the two code paths are as follows:
* We only want to generate suggestions for the queries which
the user is typing or can edit.
For the legacy version, suggestions are displayed for the
entire search string in the searchbox. (`all_operators`)
For the pills enabled version, suggestions are displayed
only for the input which hasn't been converted to pills.
(`query_operators`)
`all_operators` = `base_query_operators` + " " + `query_operators`.
trim is added at the end just to handle the legacy case
where we pass the `base_query` as ''.
* It is not possible to detect whether the user wants to
continue typing in the legacy version. However if the
the searchbox is still focused even after pill creation
we can assume the user still wants to continue typing.
To handle this we push an empty term as the `last` operator.
This is possible since the previous queries have been
completely entered as evident from it's generated pill.
* When using the legacy version, `search_operators` are
the same as `all_operators`, as mentioned in point 1.
In the pills enabled version we perform most of the
computations from the `query_operators`, but we do
require all `all_operators`, only for filtering the last
query's suggestion.
* And there is just one block unique to the legacy search
system. More details are mentioned in the comments of that
block.
We also refactor both the search suggestions node tests,
mainly to make them similar and easier to detect differences
when we switch over to the new version.
Previously we narrowed every time a search pill was created or deleted.
This commit allows the user to be able to continue typing without the
lag of narrowing.
This behaviour matches with the legacy version, whose code path remains
unchanged.
Under the search pills paradigm it is more natural for the
user to add pills and still continue typing.
Previously everytime a pills gets added the narrow activates
(this is still the case) and then the user had to refocus the
searchbox the continue typing the remaining search query.
The 2 function calls of `open_search_bar_and_close_narrow_description`
was removed from the 2 event handlers since it was called again, from
the `search.initiate_search`.
The "focusin" event was redundant since there are multiple other event
handlers (like the `tab_bar` or `hotkeys`) for this purpose, and all
of them call the `search.initiate_search` function.
The only change made here is the renaming of `operators` variable
to `search_operators`.
That is mostly evident from the fact that we do not need to
make any changes to `node_tests/search_suggestion_legacy.js`.
As mentioned in the previous commit, we make this change
to get a minimal diff between the legacy and search pills
enabled version.
The only changes made here is the renaming of `query_operators`
variable to `search_operators`.
That is mostly evident from the fact that we do not need to
make any changes to `node_tests/search_suggestion.js`.
This will be helpful when we combine this function with it's
legacy function. As most of the logical decisions to generate
the result is based on the `query_operators` variable for the
search pills enabled version and the `operators` variable for
the legacy search version.
This commits moves the css of upgrade-tip class from settings.scss
to app_components.scss as this class will also be used in stream
settings page for message-retention-days setting in further commits.
tip class in settings.scss is also moved as it has the same styles as
upgrade-tip class.
Fixes#14498.
When a topic is moved to a different stream, the message may no
longer be reachable to guest user, if the user is not subscribed
to the new stream.
We used to send message update event to the client in these cases,
which seems to be confusing both to the client updating the message
and the server sending push_notifications for it.
Now, we delete the UserMessage entry for these messages for the
user and send a delete message event to the client; which makes
both push_notification and the event handling client think that
the message was deleted and hence no confusion in the code is
raised.
Replace word 'chevron' with 'ellipsis'(because of the
recent migration to ellipsis for sidebars menu icons)
and mention where to click more specifically for sidebar
popover testing.
This makes the system store and track PushDeviceToken objects on
the local Zulip server when using the push notifications bouncer
and includes tests for this.
This is something we need to implement end-to-end encryption for
push notifications. We'll add the encryption key as an additional
property on the local PushDeviceToken object.
It also likely adds some value in the case that a server were to
switch between using the bouncer service and sending notifications
directly, though in practice that's unlikely to happen.
This fixes one of our oldest important user experience issues, namely
that if you never visit the home view, the Zulip webapp would often
load "deep in the past" because the pointer had not advanced.
Fixes#1529.
When fetching older/new messages, we used to resort to the pointer
to act as anchor when message list was empty.
This appears to be an impossible case, as
`fetch_status.can_load_newer_messages`
should be false in this case and user cannot be scrolling an
empty message_list in the first case.
Hence, we raise a fatal error to inform user of the same.
Since our translation functions don't support passing a variable into
them and still being found by manage.py makemessages, we need to use
translation function before passing as variable into
image_upload_widget.hbs file.
Since we use common HTML template 'image_upload_widget.hbs' for
user avatar, realm icon and realm day/night logo `realm-logo-widget.hbs`
file is replaced by 'image_upload_widget.hbs' therefore
we can delete `realm-logo-widget.hbs` file.
Now we can use common HTML image upload widget template
`image_upload_widget.hbs` for realm day/night logo and
we should access those day/night logo elements using
e.g., "#realm-day/night-logo-upload-widget .realm-logo-elements".
since we use image_upload_widget.hbs for realm day/night logo upload
widget we need to extract CSS for realm day/night logo and
place them separately under `#realm-day-logo-upload-widget`
and `#realm-day-logo-upload-widget` css id.
Now we can use common HTML image upload widget template
`image_upload_widget.hbs` for realm icon. we can access icon
element using "#realm-icon-upload-widget .realm-icon-elements".
also we need to extract CSS for realm icon and place them
separately under `#realm-icon-upload-widget` css id.
Previously, --interactive used to run tests from the start on a
repeated run triggered when tests failed and we decide to trigger
a re-run by pressing Enter key. Rerunning passed tests is of no interest.
It also used to run all tests in a loop even if all pass.
This commit fixes those both issues i.e it runs again from the
test that failed on pressing Enter and exits if all tests pass.
Messages are automatically marked read when all the messages in
the current narrow are visible. While this is handy, this is
should not happen when any of the overlays are open.
We fixed the main issue of this form in CVE-2020-9444, but the audit
done at that time only included links found in rendered_markdown; this
change completes our audit for links with target=_blank anywhere in
the codebase.
This migration fixes any PreregistrationUser objects that might have
been already corrupted to have the administrator role by the buggy
original version of migration 0198_preregistrationuser_invited_as.
Since invitations that create new users as administrators are rare, it
is cleaner to just remove the role from all PreregistrationUser
objects than to filter for just those older invitation objects that
could have been corrupted by the original migration.
This migration incorrectly swapped the role associated with invitation
objects between members and organization administrators, resulting in
most invitation objects that existed before the upgrade to Zulip
2.0.0-rc1 or later to be incorrectly administrator invitations.
Fixing the migration is safe and will help those installations
upgrading directly from 1.9.x to 2.1.5 or later.
A migration to fix the corrupted records will appear in an upcoming
commit.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.
Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident. But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.
The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
This fixes a bundle of issues where we were missing "" around
attributes coming from variables. In most cases, the variables were
integers or fixed constants from the Zulip codebase (E.g. the name of
an installed integration), but in at least one case it was
user-provided data that could potentially have security impact.
Previously, in `make_tab_data()` we were using the stream name,
which we got from the filter, to call `stream_data.get_sub_by_name()`.
This commit switches to just using `filter._sub`, which is simpler and
better.
Previously, this function relied on the return value of
`filter.get_icon()` which made it brittle.
Directly using the properties of the filter and sub object makes this
more explicit about the intentions and robust.
In commit 4f6377d493 we added
`_stream_params` as a way of storing attributes such as stream name
and stream privacy, this involved adding a few calls within functions
that updated these values (in order to maintain consistency).
This commit replaces `_stream_params` with an always consistent `_sub`
object and removes unnecessary `_stream_params` related code. Once the
`_sub` object is available, calls to `stream_data` may be considered
suspicious as they can often be avoided by just picking the desired
attribute off of the `_sub` object.
Previously, this bit of code was looking for specific icons on the
navbar, but it's more semantic to just look for the `.fa` which is a
direct child of `.stream`. It also makes the code cleaner, to have a
single call here.
This commit removes a redundant line of code which was converting from
hex to RGB rounding off and then converting from RGB to hex again.
This line was (mistakenly) introduced in
eb4a2b9d4e while removing a hover effect
that had become irrelevant.
Previously, there was a small dead spot in the click area between the
sub_count and narrow_description, such that the mouse cursor would
switch from pointer to the default.
This commit corrects the dead spot by adjusting the margins and styles
on navbar elements.
This should be workable, but there is scope for improvement especially
given that the current margins and paddings are messy and not very
semantic.
The end result is that the entire navbar becomes a smooth, clickable
region.
Previously the click area to open the settings modal was limited to
just the stream name (just the text). This, inconveniently, created a
lot of empty, unclickable space around the stream name.
This commit resolves the problem by:
* Extracting the title and icon into a separate template as
`navbar_title_and_icon.hbs` and calls this partial in
`tab_bar.hbs`.
* Calling the partial within an <a> tag for stream based narrows
and in a <span> tag for non-stream narrows.
* Making some CSS changes so that everything still renders correctly
(visually).
This commit also:
* Leads us to "piggy back" all stream based narrow elements on the
`stream_settings_link` conditional. (Previously the only "piggy
backing" was by `narrow_description` on `sub_count`, which was
necessary for the rendering of the `(no description)` string.)
The end goal here is that the entire navbar is clickable. This is a
step towards that goal, but some of the margins on the sub count and
its ::before and ::after pseudo-elements still need to be fixed.
Previously the click area to open the settings modal was limited to
just the stream name (just the text).
A nice goal to strive for here is to make the entire navbar a
continuous clickable region.
This adds the same click action as `stream_name` to the `sub_count`.
There's still scope for improvement after this change because of the
margins on `sub_count::before` and `sub_count::after` as well as
because only the text in `stream_name` is clickable.
Currently the styles for the navbar are in a confusing and ugly state.
One of the problems is that we have several styles within the `span`
including some nested pseudotag selectors within the `span`.
This is bad because it gives semantic meaning to the `span` element
which we do not intend. We should remove as many styles which intend
to target "direct children" instead of "direct children that are
spans" and (iff there are styles for the later) then substitute the
"span" for a semantically meaningful class name.
Another problem here is that these pseudotag based selectors aren't
very clear and readable, which is something we can look into
correcting now that they are separate from the `span` tag.
This is a prep commit that aims to set us on the path for further
improvements. It also enables us to switch some tags around and allows
us to use the styles in the `span` block with other selectors via `,`.
This should make no visual or behavioral changes.
Since we now support Postgres versions from 10 to 12, we might as well
have new installations start on Postgres 12 to avoid unnecessary
migration/upgrade work.
We would prefer to use the postgres packages from Postgres themselves,
if available. However, this requires ensures that, for existing
installs, we preserve the same version of postgres as their base
distribution installed.
Move the version-determination logic from being computed at puppet
interpolation time, to being computed at install time and pinned into
zulip.conf.
Google has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.
With changes by tabbott to improve the overall video call documentation.
Fixes: #15298.
Fixes#14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.
We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.
Fixes#5802.
Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
Now we can remove `user_avatar_file_input_error` id and added new class
`image_file_input_error`.we can access this class using
`#user-avatar-upload-widget .image_file_input` so that we can
have only one id at top-level and 'image_upload_widget.hbs`
can be more dynamic so we can use for other similar widgets also.
Now we can remove `user-avatar-block` id and added new class
'image_file_input'.we can access this class using
`#user-avatar-upload-widget .image_file_input` so that we can have
only one id at top-level and 'image_upload_widget.hbs`
can be more dynamic so we can use for other similar widgets also.
Now we can remove `user-avatar-block` id and add common class `image_block`.
we can access this class using `#user-avatar-upload-widget .image_block`
so that we can have only one id at top-level and 'image_upload_widget.hbs`
can be more dynamic so we can use for other similar widgets also.
Now we can remove the id `avatar-spinner-background` and access spinner
element from `#user-avatar-upload-widget .image_upload_spinner` so
that we can have only one id at top-level and 'image_upload_widget.hbs` can
be more dynamic so we can use for other similar widgets also.
Now we can remove the id `avatar-spinner-background` and access spinner
element from `#user-avatar-upload-widget .settings-page-upload-text` so
that we can have only one id at top-level and 'image_upload_widget.hbs` can
be more dynamic so we can use for other similar widgets also.
The upload text element is wrongly named as id=user_avatar_upload_button.
now we can remove that id and access upload text element from
`#user-avatar-upload-widget .settings-page-upload-text` so that we
can have only one id at top-level and 'image_upload_widget.hbs` can
be more dynamic so we can use for other similar widgets also.
We can remove id="user_avatar_delete" and access delete-text from
`#user-avatar-upload-widget .settings-page-delete-text` so that
we can have only one id at top-level and 'image_upload_widget.hbs`
can be more dynamic so we can use for other similar widgets also.
we can remove `user_avatar_delete_button` id and access delete button
from `#user-avatar-upload-widget .settings-page-delete-button` so that
we can have only one id at top level and 'image_upload_widget.hbs`
can be more dynamic so we can use for other similar widgets also.
Renaming "user-settings-avatar" to "image_upload_button" since the
`user-settings-avatar` name is irrelevant/confusing for the upload
button, and converting the id into a class so that we could just have
only one outer id.
We can check for the `is_editable_by_current_user` condition once in
the upper level instead of checking twice in middle for the same
conditions and to match the implementation of style realm icon and
realm logo since similar implementation between avatar, logo, the icon
will help us to use `image_upload_widget.hbs` for logo and icon
widgets also.
This likely fixes a bug with the delete text being shown incorrectly
for non-administrator users.
We extract image_upload_widget.hbs from user avatar upload widget.
The plan is to the same HTML template for all 4 widgets (user avatar,
icon, day logo, night logo) across the two settings UIs and different
image upload widgets as possible in future.
This breaks i18n; we'll fix it in follow-up work.
This changes the user avatar image display implementation to more
closely match how the realm icon and realm logo image features are
structured. This is early preparatory work towards sharing this code
between the various widgets.
This adds a new function `get_apns_badge_count()` to
fetch count value for a user push notification and
then sends that value with the APNs payload.
Once a message is read from the web app, the count is
decremented accordingly and a push notification with
`event: remove` is sent to the iOS clients.
Fixes#10271.
Mocking `get_base_payload()` verifies the wrong output
when the code is actually correct. So, its better that
we call the real function here, especially when we are
adding the Apple case.
The previous commit introduced a bug where it was not intuitive
for the user to scroll again.
For the current narrow, new messages were fetched again only when
scrolled to the bottom as usually there are many messages displayed.
However when the edge case mentioned in the previous commit
occured, it was not very obvious that a scroll should be done
or we could already be at the bottom and could not scroll again
to trigger a fetch.
`message_viewport.at_bottom` has a relevant comment explaining
this behaviour.
The previous commit handled the rare race condition. However,
there is a possibility that the rare race condition might occur
again while we are handling the previous condition.
This commit resolves these 2 problems by performing a re-fetch
while also resetting the `expected_max_message_id` and this
approach has two benefits:
1. The reset prevents an infinite loop, if somehow the expected
max message's id gets corrupted resulting in a situation
where the server can never send an id greater than that even
after fetching.
2. Even though we stop after just one re-fetch the race condition
might recursively occur while we handle the previous race
condition. And even though the reset prevents multiple re-fetches,
we don't have the missing message problem.
This is because we treat the next race condition as a new race
condition instead of it being a continuation of the previous.
The `expected_max_message_id` gets updated again, on receiving
a new message. Thus it can again enter the `fetch_status` block
as the reset value is updated again.
If a user sends a message while the latest batch of
messages are being fetched, the new message recieved
from `server_events` gets displayed temporarily out of
order (just after the the current batch of messages)
for the current narrow.
We could just discard the new message events if we havent
recieved the last message i.e. when `found_newest` = False,
since we would recieve them on furthur fetching of that
narrow.
But this would create another bug where the new messages
sent while fetching the last batch of messages would not
get rendered. Because, `found_newest` = True and we would
no longer fetch messages for that narrow, thus the new
messages would not get fetched and are also discarded from
the events codepath.
Thus to resolve both these bugs we use the following approach:
* We do not add the new batch of messages for the current narrow
while `has_found_newest` = False.
* We store the latest message id which should be displayed at the
bottom of the narrow in `fetch_status`.
* Ideally `expected_max_message_id`'s value should be equal to the
last item's id in `MessageListData`.
* So the messages received while `has_found_newest` = False,
will be fetched later and also the `expected_max_message_id`
value gets updated.
* And after fetching the last batch where `has_found_newest` = True,
we would again fetch messages if the `expected_max_message_id` is
greater than the last message's id found on fetching by refusing to
update the server provided `has_found_newest` = True in `fetch_status`.
Another benefit of not discarding the events is that the
message gets processed not rendered i.e. we still get desktop
notifications and unread count updates.
Fixes#14017
This line was effectively hardcoding a specific stream_post_policy,
overriding the value already present in the event, to no purpose.
(I believe it got here via cargo-culting induced by #13787.)
This commit removes is_old_stream property from the stream objects
returned by the API. This property was unnecessary and is essentially
equivalent to 'stream_weekly_traffic != null'.
We compute sub.is_old_stream in stream_data.update_calculated_fields
in frontend code and it is used to check whether we have a non-null
stream_weekly_traffic or not.
Fixes#15181.
The automated tests running in CircleCI don't actually use the `zulip`
db, so we can skip running migrations on it in some CircleCI shards to
save time.
NOTE: This only effects build jobs that run provision, except the
`production-build` job where we skip building the dbs altogether.
Migrations still run on `focal-backend` build job to ensure
we are testing all our development setup code.
We refactor these 2 notices to match with the loading indicators,
thus they have been moved to `message_scroll.js`.
After a successful message fetch, we have logic to decide whether
we want to display the notices and also whether we want to hide
the loading indicators (which are already displayed).
We also conservatively hide the notices similar to the indicators
every time we narrow.
The only exception is that we show the history limit notice on
deactivating the narrow (visiting `home_msg_list`).
Since on narrowing we call `load_messages_for_narrow`,
which fetches both top and bottom messages, two loading
indicators were temporarily displayed.
This was also the case for the `home_msg_list` when we
call `mesage_fetch.initialize` on startup.
To resolve this we do not display the bottom loading
indicator (for new messages), if the older messages
are being fetched too. This is only for the initial
narrow change, and the bottom loading indicator will
be displayed correctly when the user is at the bottom.
This fixes a regression introduced when we added bottom loading
indicators at all, which was temporarily reverted in
67053ff479 before being restored in the
last couple commits.
This commit makes the `loading_older_messages_indicator` similar
to the `loading_newer_messages_indicator`.
Now all the decisions about whether to show a loading indicator
will be made from the `fetch_status` API. We still hide the
indicators everytime the view is changed, as explained in the
previous commit.
As explained in 67053ff479,
multiple message fetches may be taking place at the same time.
So some other narrows / the home message list's indicator might
get shown for the current narrow.
This commit moves the updation of the indicators display logic
to the `fetch_status` API.
Now the `loading_newer_messages_indicator` gets displayed along
with the `loading_newer` = true updation for that narrow's message
list, i.e. just before we send the API request. But only if the
message list we are fetching matches with our current message list.
The same indicator is hidden similarly, along with the
`loading_older` = false updation for that narrow's message list,
i.e. just after the success response is recieved. But only if
the message list whose data we recieved matches with our current
message list.
Also the indicators are hidden everytime we activate narrow
or deactivate narrow (`home_msg_list`). And on entering
`narrow.activate` we fetch for it's messages so they get
displayed again, if need be.
This is the reason `message_scroll.hide_indicators();` was
moved to a location above `fetch_messages`.
Fixes#15374.
This is a prep commit. Running terminate-psql-sessions command on
docker-zulip results in the script exiting with non-zero exit status
2. This is because the current session also gets terminated while
running terminate-psql-sessions command. To prevent that from happening
we don't terminate the session created by terminate-psql-sessions.
This commit actually just deletes the `get_default_suggestion`
function while the `get_default_suggestion_legacy` function's
logic remains the exact same, it is just renamed.
Since the operator can never be undefined as mentioned in the
previous commit, we do not require the check with undefined
and as a result the, `if (suggestion)` condition can be removed.
The 3 changes made here are as follows:
* The if block for both the functions is simplified as
`Filter.parse` will always return an array and also
[].slice(0, -1) === [] is true.
* The code where `base_operators` is declared is moved
to just before where it is actually used.
* The `base` variable declaration is changed to match
the pattern of that present in the non-legacy function.
Its value remains the same.
This is a prep commit for when we want to merge the
`get_search_result_legacy` and `get_search_result` functions.
This is the exact same bug as observed in
02ab48a61e.
The bug is in the way we invoke `Filter.parse`.
`Filter.parse` returns a list of operators which
can contain only one 'search' term at max.
All strings with the 'search' operator present
in the query are combined to form this 'search'
term.
However on concatenating two filters we may get
two terms containing the 'search' operator. This
will lead to the search suggestions getting
generated based on only the last 'search' operator
term instead of all the terms having the 'search'
operator.
This is evident from the test change as suggestions
should be based on "s stream:of" but instead they
were based on just the latest query.
Since 9e8f1aacb3, zulip_ops machines
might have two Package declarations for `certbot`, which doesn't work
in puppet.
The fix is, as usual, to use our `zulip::safepackage` wrapper instead.
This likely fix a bug that can leak thousands of messages into the
invalid state where:
* user_message.flags.active_mobile_push_notification is True
* user_message.flags.read is True
which is intended to be impossible except during the transient process
between marking messages as read sending the "remove push
notifications" event.
The bug is that if a user who is declaring bankruptcy with 10,000s of
unreads ends up having the database query to mark all of those as read
take 60s, the Django/uwsgi request will time out and kill the process.
If the postgres transaction still completes, we'll end up with the
second half of this function never being run.
A safer ordering is to do the smaller queries first.
We do this in a loop for correctness in the unlikely event there are
more than 10,000 of these.
In commit 35c8dcb599 we introduced the
`_stream_params` object within filter.js but we didn't correctly
handle cases where `_stream param`s is undefined within `get_title()`,
`generate_url()` and `get_icon()`, which cause the navbar to if eg a
guest user tries to access a stream they weren't subscribed to.
This commit fixes this by:
* Adding the relevant checks
* Adding node tests that include non-existent streams.
* Adds the 'question-circle-o' icon for non-existent stream narrows.
A side note here is that "non-existent streams" fall under
"common narrows" as per our current definitions, which doesn't really
make sense but shouldn't bother us.
Fixes: #15387.
This commit adds translation tags to a few user facing strings which
weren't translated prior:
- "Unknown streams" text and description.
- "All messages" heading.
- Tooltip text for precise count of subscribed users.
The numeric count itself is not translated, because we do not do
similar anywhere else in the UI.
The style guide for Zulip is to always use "primary" and "replica"
when describing database replication. Adjust a few comments under
`puppet/` that do not adhere to this.
Unfortunately, some references still remain to the insensitive and
inaccurate "master" / "slave" terminology. However, these are only in
files which we are attempting to preserve as close to the upstream
versions they are derived from (e.g. postgresql.conf,
postfix/master.cf).
65774e1c4f switched from using the bundled check_postgres.pl to using
the version from packages; the file itself remained, however.
Remove it, and clean up references to it.
Fixes#15389.
Instead of SSH'ing around to them, run directly on the database hosts.
This means that the replicas do not know how many bytes behind they
are in _receiving_ the wall logs; thus, the monitoring also extends to
the primary database, which knows that information for each replica.
This also allows for detecting when there are too few active replicas.
We simply pass the visible message ids to remove_and_rerender
which supports bulk delete operation.
This helps us avoid deleting messages in a loop which freezes the
UI for the duration of the loop.
Fixes#15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.
Generated by pyupgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
After merging PR #15355 we found that CircleCI may send
super minimal payloads. This does not seem to depend on
.circleci/config.yml since we received two different
types of payloads off of the same configuration.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.
Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).
Backend implementation of #15285.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.
Dropdown for changing retention policy is disabled in UI for admins also.
This commit adds the code to disable deactivate organization button
for admins. We now allow only owners to deactivate the organization.
The backend implementation for allowing only owners to deactivate
is already added in 81c28c1.
This commit adds the restriction of deactivating owners for admins
by disabling the deactivating button in the UI. Only owners are
allowed to deactivate other owners. The backend part of this is
already implemented in 86b52ef.
This commit adds the option of owner role in user role dropdown
and also takes care of the restrictions while adding/removing
owner status of the user.
This commit also handles the places where we dispaly role of
the user in UI.
Whenever we use API queries to mark messages as read we now increment
two new LoggingCount stats, messages_read::hour and
messages_read_interactions::hour.
We add an early return in do_increment_logging_stat function if there
are no changes (increment is 0), as an optimization to avoid
unnecessary database queries.
We also log messages_read_interactions::hour Logging stat
as the number of API queries to mark messages as read.
We don't include tests for the case where do_update_pointer is called
because do_update_pointer will most likely be removed from the
codebase in the near future.
The chevron sometime can be confused as an icon for expanding the
stream topics especially for the new users.
This commit replaces the confusing chevron icon from the stream-sidebar,
topic-list, user-presence-row, all-messages and starred-messages with
ellipsis-v icon(vertical three dots).
Fixes: #7115
This uses `where_active_push_notification()` instead of
the flag condition to keep it consistent with the app
code as we use the same function to filter active push
notifications everywhere else.
• Specify disabled rather than enabled protocols, so as not to disable
TLS 1.3.
• Provide an explicit cipher suite list (Mozilla intermediate config
version 5.4).
• Respect the browser’s preferred cipher suite ordering over the
server’s.
• Use FFDHE2048 Diffie-Hellman parameters.
• Disable SSL session tickets.
(SSL stapling is also recommended but SSLStaplingCache cannot be
configured inside a <VirtualHost> block.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Overrides some of internal functions of python-social-auth
to handle native flow.
Credits to Mateusz Mandera for the overridden functions.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
The left simplebar sometime interferes with the
chevrons on the stream filters.
This commit reduces the width of active stream filter
by adding margin-right and hence fixes the overlap of
the simplebar over the active filters
This adds a powerful end-to-end test for Zulip's API documentation:
For every documented API endpoint (with a few declared exceptions that
we hope to remove), we verify that every API response received by our
extensive backend test suite matches the declared schema.
This is a critical step towards being able to have complete, high
quality API documentation.
Fixes#15340.
The get_rendered_messages now returns a Map, that is formatted as:
{
'Verona > topic 1': ['message 1', 'message 2'],
'You and Cordelia Lear': ['private message 1']
}
We also add check_messages_sent (which was expected_messages in
casper). It now takes in a object, formatted similar to
get_rendered_messages above, instead of an array with topic and
messages.
We include all method needed from casper except
get_rendered_messages, and expected_messages. We will want to tweak
their API when we migrate them.
We also rename bunch of method here while migrating them.
The changes made here are as follows:
* We rename `show_history_limit_message` and `hide_history_limit_message`
to `show_history_limit_notice` and `hide_history_limit_notice`
respectively.
* We rename `hide_or_show_history_limit_message` to
`update_top_of_narrow_notices` as now this function is responsible
for updating the history limit notice as well the end of results
notice.
* We extract 2 functions responsible for hiding and showing the end
of results notice, similar to that of the history limit notice.
All instances of `$(".all-messages-search-caution").hide();` are
replaced with the call of `hide_end_of_results_notice` function.
The streams:all advertisement notice in search should only appear
after all results have been fetched to indicate we've gotten to the
beginning of the target feed.
The notice gets hidden at the start of `narrow.activate` and is
shown just after we've fetched an older batch of messages if the
"oldest" message has been found.
Previously it would get displayed after the first fetch which
takes place from `narrow.activate`. Thus we move this logic to
`notifications.hide_or_show_history_limit_message` which gets
called after a successful message fetch.
Since the home message view contains all the messages we are not
required to display this notice. However if it is already shown
we hide it as a part of `handle_post_narrow_deactivate_processes`.
To accomplish this we need to add `has_found_oldest` key to the
`fetch_status` API.
We also removed the `pre_scroll_cont` parameter as this was it's
only use case and is now redundant.
On update_message events, we were changing narrow before we
locally updated the data, this resulted in a weird mismatch
between locally available data and that fetched from the server.
Ideally, we should not be requesting any data from the server
in most scenarios since the messages for new narrow is
locally available.
As a result, the new narrow didn't have any messages other than
a breadcrumb message. To fix this, we change narrow post
locally updating the data.
The original bug was not exactly reproduced, but a similar version
of it was simulated and was found to be fixed.
Tweaked by tabbott to preserve an optimization.
This ensures that we do this fetch, which is intended to get data on
the pre-event state, before we start perturbing the message list data
structures via rerendering.
Apparently iamcal/emoji-data has a dedicated category for flag emojis.
And get_all_emoji_categories() in emoji_picker.js doesn't return the
Flags category, because we haven't declared that category in our emoji
data logic.
Note that the category looks quite sparse because it lacks country
flags, since we don't yet support emojis combined with a Zero Width
Joiner (ZWJ) (see #992 & #11767).
Fixes#15303.
In the very common event that one ends up looking at not the home view
while the browser is catching the home view up, this ended up
resulting in loading indicators being displayed at the bottom of
whatever narrowed view one was looking at incorrectly.
A proper fix for this will involve making these loading indicators
conditional on what view one is looking at. Since one can change
views rapidly from a narrowed message list to the home view (and in
the future, between narrows), probably the best approach would be to
move the state in `message_scroll.js` the state for whether a loading
indicator is expected to be shown into the `fetch_status` data
structures, and then make all decisions about whether to show/hide a
loading indicator be calls to a function with a name like:
current_msg_list.data.fetch_status.update_newer_loading_indicator()
At least, that's probably what we should call in places like
`narrow.deactivate()`.
Before 77a26d41ae, there was only one
loading indicator (at the top of the page), so the if/else logic for
hiding loading indicators was correct, if confusing. Since we've now
added a new bottom-of-page loading indicator, it's important to have
the logic correctly reset the state to hide all existing loading
indicators on narrowing, and then just render the ones needed/desired
by the current view.
Combined with similar code in `narrow.deactivate`, this achieves the
goal that we correctly update loading indicator state when switcing
views.
The psycopg2.SQL API unfortunately doesn’t work with
django.db.migrations.RunSQL, so we need to take a detour into
PL/pgSQL for EXECUTE and format.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When doing query for same topic names in a stream, we should do a
case-insensitive exact match for the topic, since that's the data
model for topics in Zulip.
This is a test case verifying the current codebase produces incorrect
results. All the messages should have been edited in this case
regardless of the topic's case unless they belong to a different
stream.
This deduplication helps with readability.
Pass get_topic_key in recent_topic_row instead of
computing it in DOM.
Fix broken test_update_unread_count
after this change. This was a regression
which went unnoticed.
Generated by pyupgrade --py36-plus --keep-percent-format.
Now including %d, %i, %u, and multi-line strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The Python 3.6 style does support non-total and even partially-total
TypedDict, but total gives us better guarantees.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We now use the real implementation of
`stream_data` in our tests (as well as `people`),
while still just checking stubs for "heavier"
functions that we dispatch to.
Also, we use our regular blueslip helpers.
This is mostly a code move. There is a bit of
boilerplate at the top, and I just use
`assert.deepEqual` instead of `assert_same`.
I also use a little wrapper to provide
output like this:
Starting node tests...
running tests for dispatch_subs
test: add
test: peer add/remove
test: remove
test: update
test: add error handling
test: peer event error handling
One little piece of code that was obsolete
simply got deleted, not moved.
One of the goals here is to un-stub the
stream_data layer.
We extract stream_edit.rerender to make
the live-update code easier to follow.
The function should eventually be inlined,
but I want to clean up some other stuff first.
These are basically shims for some deeper refactorings.
I basically just try to make the code express the
problems more clearly:
- use stream_name instead of sub
- make early-exit more explicit
- make it clear that add_subscriber needlessly
requires a name
- make it clear we have an unnecessary loop
I also fixed some phony data in the test.
We are trying to phase out the trigger-event way
of telling modules to do something.
In this case we not only remove the indirection
of the event handler, but we also get to remove
`compose_fade` from the `ui_init` startup sequence.
This also has us update `compose_fade` outside
the loop, although that's only a theoretical
improvement, since I don't think `peer_add` events
every actually include multiple streams.
To make the dispatch tests a little flatter, I
added a one-line change to zjsunit to add
`make_stub` to `global`.
To manually test:
* have Aaron reply to Denmark (keep compose box open)
* have Iago add Hamlet to Denmark
* have Hamlet unsubscribe
The narrow parameter was incorrectly documented as a one-level array
rather than an array of arrays, and the tests incorrectly expected
this due to a combination of design and implementation bugs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
There seems to have been a confusion between two different uses of the
word “optional”:
• An optional parameter may be omitted and replaced with a default
value.
• An Optional type has None as a possible value.
Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type. But in
other cases, optional parameters should not have Optional type. Fix
them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
To take screenshots, we log in as Desdemonia who doesn't seem
to have permissions to the streams where we capture screenshots.
This commit fixes it by logging in as Iago.
Also added a waitForNavigation until DOMContentLoaded to
avoid flakes that could occur because of us going to another URL
before Iago gets logged in.
The discard button stub in createSaveButtons is set same
as that of save button. This isn't desired because it hides
the save button on running `change_save_button_sate` instead
of hiding discard button. It was previously working as we weren't
testing visibility of save button anywhere.
All differences between the primary and replica roles having been
merged, fold the `postgres_common`, `postgres_master`, and
`postgres_slave` roles into just `postgres_appdb`.
These values differed between the primary and secondary database
hosts, for unclear reasons. The differences date back to their
introduction in 387f63deaa. As the comment in the replica
confguration notes, settings of `vm.dirty_ratio = 10` and
`vm.dirty_background_ratio = 5` matched the kernel defaults for
"newer" kernels; however, kernel 2.6.30 bumped those to 20 and 10,
respectively[1], as a fix for underlying logic now being more correct.
Remove these overrides; they should at very least be consistent across
roles, and the previous values look to be an attempt to tune for a
very much older version of the Linux kernel, which was using an
different, buggier, algorithm under the hood.
[1] 1b5e62b42b
This file controls streaming replication, and recovery using wal-g on
the secondary. The `primary_conninfo` data needs to change on short
notice when database failover happens, in a way that is not suitable
for being controlled by puppet.
PostgreSQL 12, in fact, removes the use of the `recovery.conf` file[1];
the `primary_conninfo` and `restore_command` information goes into the
main `postgresql.conf` file, and the standby status is controlled by
the presence of absence of an empty `standby.signal` file.
Remove the puppet control of the `recovery.conf` file.
[1] https://pgstef.github.io/2018/11/26/postgresql12_preview_recovery_conf_disappears.html
We triage on the event's "op" field instead
of using the type check.
And then we still get the implicit assertion
that we any "add" event produces a dict
for "subscriptions", since we dereference it
in the assertion for "subscribers".
Since the nagios authentication is stored _in the database_, it is
unnecessary to run if the database is simply a replica of the
production database. The only case in which this statement would have
an effect is if the postgres node contains a _different_ (or empty)
database, which `setup_disks` now effectively prevents.
Remove the unnecessary step.
481613a344 updated the `setup_disks` script to no longer reference
`mdadm`, since we no longer set up RAID on servers.
Update the puppet that would call it to remove the `mdadm` dependency,
and run only if the state is not what it produces -- namely, a symlink
for `/var/lib/postgresql`, which must point to an existent
`/srv/postgresql` directory.
The end state it produces is _either_:
- `/srv/postgresql` already existed, which was symlinked into
`/var/lib/postgresql`; postgres is left untouched. This is the
situation if `setup_disks` is run on the database primary, or a
replica which was correctly configured.
- An empty `/srv/postgresql` now exists, symlinked into
`/var/lib/postgresql`, and postgres is stopped. This is the
situation if `puppet` was just run on a new host, or a
previously-configured host was rebooted (clearing the temporary
disk in `/dev/nvme0`)
In the latter case, where `/srv/postgresql` is now empty, any previous
contents of `/var/lib/postgresql` are placed under `/root`,
timestamped for uniqueness.
In either case, the tool should now be idempotent.
This fixes errors when provisioning a new system (or version of
postgres) when the configuration file cannot be written because its
parent directories do not exist.
Files inherently depend on their containing directories, so no
explicit dependencies are necessary.
The `pg_datadir` variable was only used, and accurate, for CentOS.
Pull it out into `postgres_app_base`, broaden it to being accurate on
Debian-based systems as well, and use it consistently in the
templates.
As the previous commit, this is currently only used in tuning, but is
a property of the whole postgres configuration; move it there, as just
the directory, not the file.
Use this directory consistently in the erb templates. Since we
produce a `pg_hba.conf`, it makes sense that we point to the path that we
know that we explicitly wrote to, for instance.
While it is only currently used in the tuning configuration, it is a
property of the base configuration, and fits more clearly into the
case block there.
This is a prep commit for replacement of chevron
from sidebars.
This commit will add ellipsis-v icon in svg format downloaded
from font-awesome 5. This has to be done because font-awesome 4.7
(the version we are using) does not have this icon with
circular dots.
And font-awesome 5 as a whole doesn't make sense to upgrade to because
it's intentionally semi-crippled as part of their business plan.
Also include entry in THIRDYPARTY and Licence details.
This will adjust the height of user presence list elements to
match with the left-sidebar list elements i.e. 23px.
Extra margin is removed to avoid increase in too much spacing
between elements.
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Previously, the navbar failed at managing the searchbox text state in
cases where, eg, the user performs navigation by browser history.
This commit resolves the issue by ensuring that the searchbox text is
only (and always) set when the searchbox is made visible, and as such
there is no "state" to manage and we will always display the correct
text.
It also adds a test in `search_legacy.js` to make sure that the search
text is placed as intended.
Fixes: #14771.
The navbar currently fails at managing the searchbox text state in
cases where, eg, the user performs navigation by browser history.
This is a prep commit that will help resolve the bug.
1f565a9f41 removed the `package` lines which install
`python-dateutil`, but not the line in `puppet_ops` that reference it;
as such, Puppet manifests in puppet_ops fail to compile.
Remove the stale reference to `python-dateutil`, which is unnecessary
since the code is python3, not python2.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
The zulip user needs to be able to read the file, when running the
backup tool.
We put root:root as owner on other nginx config files, so it's probably
correct to keep the ownership as it is, and set the mode to 0644.
I tried to make the most minimal change here
possible, since I don't really know this module
well. Possible alternatives were:
* $('#tab_bar') everywhere
* elem
* tab_bar_elem
I don't feel strongly.
Long term I believe we intend to change the name
of this module to something more like `navbar.js`???
When we call functions inside our own modules that use
the `window.foo = exports` pattern, we have always had
a pretty strong preference to call `exports.internal_function`
instead of `foo.internal_functions`.
The stragglers here weren't violating this convention
for any intentional reason. Some of the places here
probably were part of code moves where somebody
(probably me) moved functions into the modules to avoid
unnecessary indirection, and I missed a spot where I
could change from `presence` to `exports` (or whatever).
And other places are probably just kinda arbitrary
decisions by the original developer, and we just haven't
bothered to clean it up until now.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.
Fixes#14966.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.
There may be a very tiny number of mobile
clients that still use the old emails API.
This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.
Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.
The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:
validator=check_list(check_int)
This also allows us to remove a test hack
related to the API documentation. (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
This combines `upload_realm_logo` and `upload_realm_icon` into single
function called `upload_realm_logo_or_icon`. The functions wer near
duplicates.
Additional refactoring should be able to deduplicate the logic further.
Part of #14730.
We can remove a {{theme_mode}}-settings class on the upload button
and access day/night from `.closest("realm-logo-section")`
so that only the outer ID differs between the two widgets.
Part of #14730.
Currently there are no checks in validate_against_openapi_schema
to check whether the `content` it received actually matched with
the `response code`. For example during testing if a certain endpoint
was returning 400 but it was expected to return 200, then it would
pass schema validation as it would only have `msg` and `result` keys.
Add this validation and fix the wrong response returning points.
Add test to validate example responses in zulip.yaml. Also change
zulip.yaml for some wrong examples or for cases which were not
covered by `test-api`. Also enhance `validate_against_openapi_schema`.
Prevent `JsonableError(_("Missing content"))` from
ever being triggered.
That error wasn't handle by anything, and thus just threw a 500, as
it's not a response to an HTTP request.
The right fix is to adjust the caller to ban the empty string in
content (or content that strips to the empty string).
Closes#15145.
ACCESS_TOKEN_URL works a different for apple authentication, so,
we removed and remocked the ACCESS_TOKEN_URL mock in
`register_extra_endpoints` override of apple auth test class.
It is cleaner to have it as generic feature of `social_auth_test`.
So, this commit adds a function that returns token_data_dict that
we had earlier and is called in the ACCESS_TOKEN_URL mock.
This function is overriden in apple auth test class to generate
payload of the format that apple auth expects.
Thanks to Mateusz Mandera for the simple idea.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Previously, renaming the stream would cause `colorize_tab_bar` to fail
because the search filter would provide it the old stream name and the
look up for the stream color would fail.
A quirk of how this system currently works makes it so that even
though the filter is set to the old stream name (and so becomes
inconsistent with the stream_data state) the `_stream_params` object
is maintained to be consistent with stream_data and as such can be
relied upon to find the correct color of the icon.
Previously the navbar did not live update the stream name correctly.
The correct behaviour was blocked on the `_stream_name` prop on the
filter object. The original purpose for maintaining this prop was
convenience, to reduce calls to `subs`, however, it would become
inconsistent with the value from `subs` on stream rename.
In this commit we add a call to `filter.fix_stream_params` in
`subs.update_stream_name`. This fixes live rerendering in the navbar,
despite the fact that searchbox in the nav (and the filter object via
`filter.operands("stream")[0]`) will still have the old name.
This is a slightly hacky way of masking some of the problems in the
Filter object. However, it should make do until we migrate to a stream
ID based state there.
Fixes: #14728.
Prior to this commit the icon in the navbar didn't live update to
reflect changes in stream privacy.
Here, we add a call to `tab_bar.render_title_area` in
`subs.update_stream_privacy()`, to enable live updates on the icon.
Fixes#14728.
The navbar currently does not live update the stream icon correctly
for changes in stream privacy.
One place where the correct behaviour gets blocked is on the
`_is_stream_private` prop in the filter object. We keep props such as
this for convenience, to reduce calls to `subs`, however, this prop
becomes inconsistent with the value we get from `subs` when the stream
privacy is updated.
In this commit we add a call to `filter.fix_stream_params` in
`subs.update_stream_privacy`. This change does not fix the live
rerendering in the navbar because we don't call redraw yet, but
it's a prep commit to towards that goal
The navbar currently does not live update the stream name or the icon
for stream privacy correctly.
One place where the correct behaviour gets blocked is on the
`_stream_name` and `_is_stream_private` props on the filter object.
We keep these props for convenience, to reduce calls to `subs`,
however, these props become inconsistent with the values from `subs`
when the stream is renamed or stream privacy is changed.
This refactor extracts out `get_stream_params` and `fix_stream_params`
methods as a prep commit towards correcting the problem, but does not
make any behavioural changes.
This is a prep commit for solving #14728.
A comment about the difficulties relating to live updating stream
names in the navbar was incorrectly placed within the function for
live updating stream descriptions in
023187b3f1.
This moves the comment to the right place.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.
This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.
We need to apply that restriction both in the role change code path
and the deactivate code path.
This commit sets the role of the user creating the realm as
realm owner after the realm is created.
Previously, the role of user creating the realm was set as admin.
But now we want it to be owner because owners have the highest
privilege level.
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this error in the dev environment:
$ ./manage.py checkconfig
Error: You must set ZULIP_ADMINISTRATOR in /etc/zulip/settings.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We store the relevant data to hide/show a topic in the row itself,
and use jquery to hide/show it on filter change.
This also fixes search breaking the set filters.
This fixes the bug that message was undefined since we used to store
locally echoed message id and were not updating it after new message
id for the same message was received from the server.
We remove all trace of the old topic and reprocess all the new
messages in old and new topics.
process_topic_edit function was moved since it needs to be below
get_topic_row function.
We reuse the existing logic for displaying and updating stream color
from the stream left sidebar.
Tests fixtures were extracted and updated for this commit.
The approach that supposed to use this data was not implemented
and hence this data will no longer be used.
If this feature is implemented in future,
this data will still not be used since we would depend upon
starred_messages.js library to provide us the required information.
* Add action to mute topics.
* We don't need to store muted data per topic as previously planned.
* Moved launch topic test to the top so that they run on non-modified
data.
* Show an empty overlay of recent topics.
* Register click event to open recent topics.
* Launch recent topics on "t" keypress.
This is based on the draft overlay.
This is part of a refactor that aimed to remove /json/users calls,
as we can get all the information needed on people API.
Now, the list render for $users_table and $deactivated_users_table
uses user_ids instead of user objects, as the people API give us
a filtered list of active_user_ids and non_active_user_ids.
The populate_users function doesn't need to sort the list of
active and non-active users, because the list_render is called
specifying to sort users by their full_name.
Author: Clara Moraes Dantas <clara.moraesd@gmail.com>
As part of a refactoring, we are now able to remove the
/json/users calls and get all the information needed on people.js.
To do this, now the populate_users uses the people api to get
all the active and non active human users.
This is part of a refactoring aimed to eliminate /json/users calls,
as we can have all the information needed on people.js.
Now, human_info() will call is_person_active() because the person
object it will receive won't have is_active field anymore, as
we'll use the people api to get a set of filtered active/non active
users.
Author: Clara Moraes Dantas <clara.moraesd@gmail.com>
For narrows where the `.all-messages-search-caution` or
`.history-limited-box` are visible the indicator is not
aligned with `top-messages-logo`.
Now the indicator is rendered in it's appropriate position
instead of overlapping with the text.
The logger defines a full list of handlers, meaning propagate=False is
needed, to avoid the log line propagating further up the logging tree
and getting logged multiple times by the duplicated handlers.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.
Signed-off-by: David Wood <david@davidtw.co>
This commit aligns the search icon in the navbar (with the search bar
closed) to be in the same position as the "search_exit" or "x" icon
(which appears when the search bar is open).
Commit c4e59309e4 introduced a
regression that caused a small part of the navbar in night mode to not
have the correct background color.
The relevant changes in that commit intended to fix the margin for the
search box for when the search pills feature was set to active.
This commit slightly increases the padding for the search box (when
pills are active), to improve pill alignment, and adds styles for
"#searchbox_legacy" to correct the background when search pills are
disabled.
This also reverts the change from commit
29b8e11e20 which tried to improve the
alignment of pills by adding a margin left but didn't address the
background color issue.
This was previously hardcoded with agreement between the Zulip backend
and frontend as 86400 seconds (1 day). Now, it's still hardcoded in
the backend, but arranged in a way where we could add a setting
without any changes to the mobile and terminal apps to update logic.
Fixes#15278.
I don't believe it's actually been possible for this to be shown in
Zulip in several years; and we just made it more obviously so
(resulting in a linter error).
We now trigger realm day/night logo upload by clicking on realm
day/night logo element itself rather than having a big upload button
and to match our user avatar UI. Added new spinner over the logo
element itself to show while uploading realm logo for both day and
night logos.
Display logo at full width regardless of the size of the image to
reduce the dependency on the logo image in determining the logo
container size. This also fixes a problem owhere the night/day logos
would lose their default-dark/white background color when we upload an
image in jpg format rather than png.
Change user avatar spinner implementation to match
realm icon spinner implementation and have common css class
since similar implementation between similar widgets may help
in future deduplication.
The orig_initial_pointer variable was part of the implementation for
ensuring server-initiated reloads preserve the user's selected message
and scroll position (so that they are not disruptive). Previously,
the logic did some unnecessary contortions to ensure the two goals:
* The `pointer.js` logic knows what the server thinks the pointer is.
* The `message_fetch.js` logic knows what anchor to use to center it's
home view fetch.
It's a lot cleaner to do this by not mutating page_params.pointer.
In the past, the anchor message has always been the same as the
pointer, but we're about to change that as part of removing the
pointer entirely.
Using the anchor is logically what we meant, anyway, since we always
want to select a message that's actually within the range we just
fetched.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history. However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
Now that we have production support for Ubuntu Focal, we update the
documentation to state our support for it.
(We also drop deprecated Xenial and Stretch from supported platforms).
Run production suites on Ubuntu Focal.
Added separate success-http-headers files for Focal and Bionic.
Also excluded them from whitespace rules in lint.
memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.
We already use a workaround for this while applying puppet configurations in
99e71f3786 but for docker builds we used
do memcached hack since we can not use systemd in docker containers.
certbot-auto doesn’t work on Ubuntu 20.04, and won’t be updated; we
migrate to instead using the certbot package shipped with the OS
instead. Also made sure that sure certbot gets installed when running
zulip-puppet-apply, to handle existing systems.
We already override the umask in upgrade-zulip-stage-2, but that’s too
late since we’ve already written a bunch of files in stage 1. I would
have removed the stage 2 override, but the OS upgrade documentation
references running stage 2 directly.
Fixes#15164. Note that an affected installation will need to upgrade
twice, because the first upgrade uses the old stage 1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
This commit adds `name` attribute for the backends that do not
have them.
This is just a kind of prep commit in case if we want to use
`self.logger.xxxx()` in the future which is dependent on the
`name` attribute. But right now these logging calls aren't used
anywhere in those backends.
`HTTPError` has empty string for `str(HTTPError())`. Logging it
as it is would not be much helpful. So, this commits adds code
to log the name of error also.
Adds a top-level logger in `settings.LOGGING` `zulip.auth`
with the default handlers `DEFAULT_ZULIP_HANDLERS` and
an extra hanlder that writes to `/var/log/zulip/auth.log`.
Each auth backend uses it's own logger, `self.logger` which
is in form 'zulip.auth.<backend name>'.
This way it's clear which auth backend generated the log
and is easier to look for all authentication logs in one file.
Besides the above mentioned changes, `name` attribute is added to
`ZulipAuthMixin` so that these logging kind of calls wouldn't raise
any issues when logging is tried in a class without `name` attribute.
Also in the tests we use a new way to check if logger calls are made
i.e. we use `assertLogs` to test if something is logged.
Thanks to Mateusz Mandera for the idea of having a seperate logger
for auth backends and suggestion of using `assertLogs`.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.
* short_name hasn't been present anywhere else in the Zulip API for
several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
returned by the register_event_queue process.
The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.
To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.
This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
Slack owners and primary owners will be mapped to zulip
realm owners on import.
Previously, we mapped the owner and primary owner roles of slack
to realm admins in zulip. As we have added ROLE_REALM_OWNER in
8bbc074, we now map slack owners and primary owners to owners in
zulip.
Tests are modified for checking all the 3 cases-
- Slack workspace primary owner
- Slack workspace owner
- Slack workspace admin
This commit also has docs changes in 'import-from-slack.md'.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning. This is for protecting against
symmetric/asymmetric key confusion attacks.
This is a backwards-incompatible configuration change.
Fixes#15207.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
At the time of creating streams in test_counts.py we earlier did not saved
recipient in the stream object.
stream.recipient is used in many functions so they would throw error.
The right long-term fix here is probably to just use the standard
stream creation functions rather than having a hacky duplicate
here.
Previously, it was unnecessarily difficult to parse the sentence to
determine that "HTTP response" and "internal state of the server
following the request" are the coherent ideas. Even if length wasn't
an issue, e.g. "... and checking both the A and B are correct", the
sentence still feels a bit fragile without a "that".
Since the second phrase is indeed relatively long, and "internal state
of the server" is a reasonable guess for the second coherent idea, the
"the" helps to reset the reader's expectation about where the next
coherent idea starts, and ends.
Lastly, having "both" in front of the two phrases encourages an
assumption that they're shorter (which is especially problematic for
the second phrase), while having it at the end of the sentence helps
to anchor the end of the second phrase; this is especially true since
the absence of "both" before that point encourages an assumption that
you haven't finished reading yet, given that two things have been
mentioned.
This declaration already exists in the default CSS.
This declaration was present when the edit history modal was first
given a night mode (then called "dark mode") style in November 2017 in
4f81bdd0a6. It also existed in the
default CSS at that time.
Previously, topic edit diffs in the edit history modal were not
highlighted in the same way as content diffs because the highlighting
CSS rules were inside a .rendered_markdown block. So they affected the
content diffs, which are classed as such, but not the topic diffs.
This commit moves the highlight rules to a
.message_edit_history_content block inside the already existing
#message-edit-history block. .message_edit_history_content had
already existed in the edit history template message_edit_history.hbs,
and is assigned to both the content and topic diffs.
The ability to see topic edits in the edit history was added in
March 2019 in 38be5ea74394d2fd8586038de6ac447b4bbfbf67; the
highlighting worked at that time. It broke four mounths later in July
2019 in 38ffde37e5 when the highlight
rules were moved into a .rendered_markdown block after having been
global.
(As a further aside, .rendered_markdown was only added to the content
diffs in April 2019 in 5c36918c17.
.message_edit_history_content had been first added, to the content
diffs, in February 2019 in 7d42d7b4dbe6eb144a148135db50ad35efc01295.)
Aside from fixing topic edit diffs, this change is just more correct;
the highlight rules don't belong under .rendered_markdown, and they
don't need to be applied globally.
Previously, the edit history modal did not respect the time format
setting (whether to show times in 12-hour or 24-hour format) when
displaying message edit times (#15171).
This commit fixes that by passing the edit times to
timerender.stringify_time(), which takes that setting into account,
instead of just doing a static string formatting operation.
This bug has existed since February 2017, when the edit history UI
was first added in 1a697b6e02.
Fixes#15171.
Currently, the edit history modal does not respect the time format
setting (whether to show times in 12-hour or 24-hour format) when
displaying message edit times (#15171).
This commit refactors how fetch_and_render_message_history() handles
times in order to make fixing that issue in a reasonable way easier.
It will be fixed in a following commit.
Previously, the show_date_row flag for the first entry in the edit
history modal was directly set to `true`, while in all other entries
it was calculated with identical code. Though show_date_row for the
first entry should indeed always be true, there's no need for it to be
a special case.
In preparation for factoring out the calculation of show_date_row,
this commit nominally calculates the first entry's show_date_row with
the same code that is used to calculate show_date_row for all other
entries. Nominally, because it will still always end up being true.
Previously, the logic for when to add a date row to an edit history
entry was checking against the date of the original message (which is
always the first entry in the message history), not the date of the
previous edit. This caused every edit not made on the date of the
original message to show a date row, even if it wasn't the first edit
on that date.
This commit fixes that bug by updating prev_timestamp after processing
each message history entry, whereas before it was only updated after
processing the first one — the original message.
This bug has existed since June 2017, when
84e5fe733c changed how date rows worked;
from only showing one at the top labeled "Earliest" to each entry
having a possibilty of showing one.
Previously it was impossible for a topic-only edit to show a date row
in any circumstance; the code that handles topic-only edits didn't
even attempt to set show_date_row, the flag that determines whether a
date row should be rendered. Now a topic-only edit will show a date row
in the same circumstances as any other edit[1].
This bug has existed since March 2019, when rendering of topic-only
edits was first added in 38be5ea743.
[1] Currently, "the same circumstances as any other edit" means
there'll be a date row on the original message, and then on every edit
not made on the same date as the original message, even if it was't
the first edit on the date it was made. This is a bug that will be
fixed in a following commit. This commit is being made first since
it's fixing a lack-of-information bug, whereas the other bug is a
somewhat less important repeating-information bug.
The activity.process_loaded_messages code path was called when these
tests were originally written in f8e0137. We stopped calling that
code path in 43e5b2d (#15118). This assert test code is no longer
relevant; tested by adding console.log in the function. I came
across this when working on removing activity from the window.
Waiting till DOMContentLoaded event is triggered helps avoid flakes since
puppeteer is very fast and starts doing another task before
everything on the page is loaded. Adding this to log_in function
as almost all tests depend on this which leads to flaky tests if
the other parts of tests just start without even the page being
loaded.
One example this commit helps is for the test `02-site.js`
which is dependent on a function that runs some jquery on the site.
But because of the page not being loaded, we miss jquery and thus
the test fails. `02-site.js` and related code is added in the next
commit.
Co-authored-by: Priyank Patel <priyankp390@gmail.com>
When no credentials are sent to `log_in` function we want to
use the default generated test credentials. This saves us the
work of importing test_credentials everytime we run this function
in a different test which doesn't focus on what credentials are sent
to login.
The `wildcard_mentions_notify` key was missing from the initial
sub data when a new stream was created. Thus `wildcard_mentions_notify`
was undefined and `wildcard_mentions_notify_display` was false.
(This key is used to render the data in the templates)
This caused a bug where the wildcard notifications was unchecked
in the stream personal settings and the newly created stream was
displayed in the stream specific notifications table.
Old topic of the msg edit event can be used to help the client
calculate useful information such as if a change
in current narrow is required.
This fixes our re narrow logic after a stream edit of a topic, with
no change in topic name itself, since the original topic was not
present in the event received and hence the `orig_topic` was
undefined in this case.
This adds 'target_users' parameter to 'attempt_unsubscribe_of_principal`
function in test_subs.py, which accepts list of UserProfile objects to be
unsubscribed, instead of defining users in the function itself.
This change makes the code cleaner and more readable.
Also, 'other_user_subbed' parameter is changed to 'target_users_subbed'
to clearly depict the use of this parameter.
Prior to commit 8b7e70ac27 this system
would simply just .hide() forms when they were closed and
.empty().append() every time it needed to "show_edit". This was not a
very clean way of handling the action of canceling an edit, so
8b7e70ac27 introduced a change that had
the "show_edit" function append the form and the "hide" function
.empty() the form.
However, we overlooked the fact that the user could use browser
history to navigate away and back to the form and use "e" or
"left arrow key" to successfully append another form and get to an
ugly, broken state.
This commit does not revert 8b7e70ac27
as it is still accurate to .empty() when hiding the form, but we add
an early exit if a form already exists, to avoid bugs like the above.
Using an early exit instead of eg .empty().append() ensures that the
user doesn't accidentally lose the entire content of an edit, if they
deselect the input box and press `e`.
Fixes: #15045.
Fix a bug where the color picker reverted the custom color to the
previous one after clicking outside the popover. The bug occurred
because although there is a confirm button, the 'clickoutFiresChanges'
option was set as true, which made the frontend always send two POST
request to edit the stream and, for a reason I wasn't able to discover,
if you hit the confirm button and click outside fast enough, the change
fired by the clickout event sent the old color to the server. It was
fixed by setting the 'clickoutFiresChanges' option to false.
Fixes#15101
If typeahead is used, this adds comma separated search queries
so that multiple search pills don't get combined as one and the
search behaviour remains same as search_pills_enabled = False case.
If typeahead is not used, this prevent the typing of a single comma
after the pill gets created.
Now node_tests/search.js and node_tests/search_legacy.js have
almost identical test besides the search pills related data
processing work.
This is possible since we no longer depend on the values stored
in the input pills and can narrow or search for term based on
the `#search_query` box value.
This was done in commit 02ab48a61e.
This commit removes the 'get_active_user_for_email' function
from people.js. We have removed the use of this function
in the previous commits, which changed the functions using
'get_active_user_for_email' to use user_ids instead of emails.
This commit changes the would_receive_message to use user_id
instead of emails.
This change is done because user_ids are immutable and using
user_ids is the correct way of uniquely identifying user.
The change in 'would_receive_message' also leads to change
in util.is_pm_recipient to use a string of user_ids instead
of emails.
We also know that user_ids passed to 'would_receive_message'
are active user_ids, since we get them from buddy_list.
So we don't need to check whether the user is active, which
was previously being checked by get_active_user_for_email.
This commit changes the needs_subscribe_warning function to
use user_id instead of emails.
This change is done because user_ids are immutable and using
user_ids is the correct way to uniquely identify a user.
We already know that user_ids being passed in this function are
active user_ids, since they come from typeaheads.
So, we only need to call 'people.get_by_user_id', to get the user
object from user_id and do not need to check the active status of
user, which was done previously using 'get_active_user_for_email'.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.
This is a follow up to 2187c84.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.
User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
We pipe realm_id through functions where it is available,
this helps us avoid doing query for realm_id in loop when
multiple messages are being processed.
It's safer and cleaner to simply just rerender the entire navbar for
small updates like these since they're rare events. Given that we're
not doing anything unique in such updates, it's best if we just call
".render_tab_bar" wherever required instead of having several
functions in tab_bar.js which do exactly that. This also sets a good
precedence of what to do for stream privacy and subscriber count live
update.
However, we can't easily fix the "subs.update_stream_name" code path
because of how the Filter objects represent the stream by name, not be
ID. Given that the above would be out of scope for this change, it's
left as a TODO.
It's best to separate these in order to simplify the "build_tab_bar"
function. We also correct a comment about the "search_exit" click
handler being for the searchbar.
OneLogin has removed the app that these instructions used to rely on.
This app choice should be more stable, as there are other providers
that rely on it in their instructions for setting them up with OneLogin.
Ideally, in the future, we'll get our own app added to OneLogin's app
catalogue, which will simplify the setup process for administrators.
Fixes warnings like this:
/srv/zulip-py3-venv/lib/python3.8/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField MutedTopic.date_muted received a naive datetime (2020-01-01 00:00:00) while time zone support is active.
warnings.warn("DateTimeField %s received a naive datetime (%s)"
Signed-off-by: Anders Kaseorg <anders@zulip.com>
datetime.timezone is available in Python ≥ 3.2. This also lets us
remove a pytz dependency from the PostgreSQL scripts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The commit fixes the spacing between the search icon and the input
field by adding `margin-left` to the search input field. The following
issue is only visible in dark mode as the nav and search input have
different background color while normal theme uses same background color
for nav and search input.
Since we use this option in our docker-zulip project also
so rather than using it as a test suite option we made it
more specific i.e. --build-release-tarball-only.
This commit changes the compose_invite_users template to use
data-user-id as property intead of data-useremail.
This is changed to maintain consistency with other parts of the
code where user_ids are used for referring to users.
This also helps in removing some of the checks for the case of
undefined emails.
We now send user_ids to the backend API for subscribing/unsubscribing
users to a stream instead of emails.
This change is done now because we have just migrated the backend API to
support sending user_ids in 2187c84, so it wasn't possible before.
This change is helpful because sending user_ids is more robust, as those
are an immutable reference to a user, rather than something that can
change with time.
Unable to upload a realm logo once we encounter file input error bug
was fixed by clearing `get_file_input()` after file input error
with `get_file_input().val('')`.
The previous .clone() logic was preserved over many years but
apparently was also just wrong.
Fixes#15198
Fixes warnings like these with python -Wd:
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:327: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
for child in currElementPair.value.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:328: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
if child.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:282: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
for child in currElement.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:283: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
if child.getchildren():
https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this warning:
/srv/zulip/docs/conf.py:337: RemovedInSphinx40Warning: The app.add_stylesheet() is deprecated. Please use app.add_css_file() instead.
app.add_stylesheet('theme_overrides.css') # path relative to _static
https://www.sphinx-doc.org/en/3.x/extdev/deprecated.html
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This isn't a complete fix, but we move the widget's popup to be
on/below the button to open the widget. We also move the bot owner
field to be on the top of the page so that we can see most of the
widget before it is clipped by the parent overlay.
We have discussed some approaches for a permanent fix on:
https://chat.zulip.org/#narrow/stream/321-s/topic/DropdownListWidget/near/894674
The get_active_humans and get_non_active_humans functions used
to return a list of user objects. The get_active_humans is used
on settings_users.js and settings_bots.js, and in both places the
only attributes needed of the person object are the user_id and
full_name.
To make the function return smaller, instead of a list of active
humans, we are returning a list of active human ids, saving memory.
With the ids we can call the people API to get the full_name attribute.
This is done to avoid flakes where our code for focusing first input
interferes with puppeteer's typing. More details in comments.
Thanks Dinesh for testing this out earlier as part finding a solution
to this flake.
Co-authored-by: Dinesh <chdinesh1089@gmail.com>
This has been a deprecated alias for django.db.backends.postgresql
since Django 1.9, removed in Django 3.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
url_to_a returns Union[Element, str], but str cannot be appended to
Element; that would raise TypeError at runtime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
zerver/lib/i18n.py:34:28: E741 ambiguous variable name 'l'
zerver/lib/webhooks/common.py:103:34: E225 missing whitespace around operator
zerver/tests/test_queue_worker.py:563:9: E306 expected 1 blank line before a nested definition, found 0
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous code for this event was using a key that's not actually
a part of the payload. So here we simple remove the usage of that key
and add a (previously missing) test for this event.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
When migrating to dropdown list widget, we incorrectly used the same label
for both realm_notifications_stream and realm_signup_notifications_stream.
This was introduced in b580baf682.
Objects whose properties are not described were validated by the
current validator. Edit it so that objects with no `properties`
or `additionalProperties` attribute i.e. opaque objects get
invalidated.
Also make changes in zulip.yaml to fix any opaque objects (tweaked by
tabbott to edit the documentation for better clarity).
Currently, `validate_against_openapi_schema` checks only the top
level of the response dictionary. Improve it so that it can
validate objects and arrays at all levels. Also edit zulip.yaml
accordingly. And for new response keys which were not defined
before add VERY basic documentation.
This commit fixes the bug for subscribing the user from mention
warning which was introduced in e52b544.
This is fixed by changing email to be passed as list to
'invite_user_to_stream'.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
The reason for this change is that, this is where `Filter` and
actual tracking of what messages are contiguous lives. This
will be beneficial when we will to move to a model where we
cache `MessageListData` objects for a large number of views.
This commit changes the stream settings UI for adding subscribers to
use our standard user pills in the input box, rather than just
plain-text email addresses. This is important progress towards
removing display email addresses from the Zulip UI.
It also allows subscribing multiple users at the same time, which is a
nice improvement.
request_retry and notify_bot_owner don't use request_data so might
as well not send it to them at all.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Using the Python Standard Library's abc library and NotImplementedError
we can better define interfaces (this is mainly to improve readability
and consistency).
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Integrations can be supplied a logo parameter which is used to contruct
their `logo_url`. It would be useful to store this parameter, instead of
computing the path from the URL.
Some UTF-8 characters (★ for example) are not displayed correctly, with
fonts-liberation. Puppeteer recommends[1] installing fonts-freefont-ttf in
their docs on running Puppeteer in docker.
Provisioning forward is sufficient. There's no need to remove the
new font and replace it with the old font, I think.
[1]: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-in-docker
This commit fixes the tests to use role instead of is_admin in
update user endpoint. These changes got missed in the original
commit 9fa6067 which included the change of using role in update user
endpoint and were also not caught in tests.
This commit removes redundant lines from the test for changing
full name in test_users.py. Those lines were passing is_admin=False
for already non admin user and were added in 41fbb16, but these lines
are of no use now.
This commit modifies the backend to accept user ids when subscribing
users to streams.
It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
Previously, the unsubscribe logic just called
exports.show_subs_pane.nothing_selected() if one had been viewing the
edit UI for a stream that the user just unsubscribed from, which
clears the selection, but didn't update the hash or do other cleanup
logic.
We should instead be calling stream_edit.open_edit_panel_empty(),
which is the appropriate function for this purpose (and has
exports.show_subs_pane.nothing_selected as a subroutine).
There's no reason to send data beyond the user `id` of the uploader,
and reason not to, as the previous model was both awkward when
`author=None` and resulted in unecessary parsing complexity for
clients.
Modified by tabbott to add the frontend changes and API documentation.
Fixes#15115.
This commit changes the person dict in event sent by do_change_user_role
to send role instead of is_admin or is_guest.
This makes things much more straightforward for our upcoming primary
owners feature.
This is vestigial.
It requires manually altering the `htdigest` file (not stored in this
repo) to change the digest realm from `wiki` to `monitoring`, and will
re-prompt users for their passwords if the browsers currently store
them.
Drop the change to move `/tmp` onto the local disk. Doing this move
confuses `resolved` until there is a restart, and has no clear
benefits. The change came in during bf82fadc95, but does not describe
the reasoning; it is particularly puzzling, since postgresql stores
its temporary files under `$PGDATA/base/pgsql_tmp`.
Do not RAID the disks together. This was previously done when they
were spinning media, for reliability; running them on an SSD obviates
this sufficiently. This means that updating the initramfs is also not
necessary.
Earlier this `standard_relay_params` was used only for SAML auth,
now "Sign in with Apple" also requires this to store those params
in session for reuse. So, this acts as a prep commit for "Sign in
with Apple" auth support.
This saves the completely unnecessary work of mapping the Client name
to its ID. Because we had in-process caching of the immutable Client
objects, this isn't a material performance win, but it will eventually
let us delete that caching logic and have a simpler system.
Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml
There are several places where the return values description is better
in the api docs than in the zulip.yaml file. Import such changes to
zulip.yaml before we replace the current hacky API docs with the
zulip.yaml content.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.
User role dropdown in user info modal is also modified to use
"dropdown_options_widget".
Modified by tabbott to document the API change.
This reverts commit 9f5725d265.
I was trying to fix how we size the buddy list in
narrow mode, which was off by 10px, but my fix worsened
things for regular mode.
Also, somebody reported a traceback related to my fix.
I didn't fully research the traceback,
but I suspect it was related to some media-query settings
for small screens or due to a put-buddy-list-in-left-pane
setting. (Basically, `$('#right-sidebar').position()` may
be undefined in some cases, and I wasn't handling that.)
After reverting this, we still have the original
off-by-10px bug that I was trying to fix, but I will
attempt to do that more cleanly in a separate commit.
This should make it so that in normal situations where
the buddy list is in the right sidebar, we will be
able to see the "Invite more users" link again.
I am still a little puzzled how I didn't catch this in
testing, but it was toward the end of a long PR, so
it could easily just be simple human error.
Fortunally, this regression was only on master for a
couple days, and users could still invite users via
the gear menu.
Restored old behavior accidentally removed in
1ae07b93d8 (diff-e353fab8bea58b8746ec68c83aa39b36L48)
The server only remembers the most recent presence status update per
device. Meaning that, for instance, if the user only uses one client and
that client's last status update was IDLE, then the server only knows
that, doesn't know anything about the user's last ACTIVE time. Thus the
"active_timestamp" the server will serve about this user to the webapp
will be "undefined".
The old behavior was that for the sake of the "Last active: x ago"
status in buddy list popover, the latest status timestamp was used,
whether IDLE or ACTIVE.
The change linked about changed that to only pay attention to
ACTIVE. Thus, if the server doesn't remember any ACTIVE statuses, webapp
would show "Last active: More than 2 weeks ago", which was incorrect.
We restore the old behavior and further improvements can be made on top
of this.
This no longer has any rules specific to it. We leave the `postgres`
munin group (which now only contains `postgres_appdb`) as
future-proofing, and so that `postgres_appdb` matches to the puppet
manifest of the same name.
This method is called on a `bitfield.types.BitHandler` object, not a dict; as
such, it is not a runtime failure.
This was missed in 035ed93111 and adjacent commits because it was not
`six.iteritems()` but just a bare `iteritems()` -- having been added
after 9e3c3e14f5 moved all other calls from `iteritems()` to
`six.iteritems()`.
Comment on why this is not a leftover python2-ism.
Grep finds no other remaining callsites to `iteritems` or friends:
```
$ rg '\.iter\w+\(' -t py
zerver/lib/url_preview/preview.py
44: content = next(response.iter_content(1000))
zerver/lib/email_notifications.py
49: for link_info in fragment.iterlinks():
zerver/lib/push_notifications.py
508: nested_levels = len(list(ol.iterancestors('ol')))
zerver/lib/bugdown/__init__.py
1364: parsed = etree.iterparse(StringIO(rendered))
zerver/lib/bugdown/nested_code_blocks.py
47: if parent.text is None and len(list(parent)) == 1 and len(list(parent.itertext())) == 1:
```
Make sure that we use the max password length defined in
RegistrationForm.MAX_PASSWORD_LENGTH when validating the
password for the password reset form.
Fixes#15087.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Previously, we had to fiddle with the generated HTML to update
individual values. Now, we can simply ask the widget to rerender
the row that we updated.
This is done by passing an html_selector function that returns
a selector for the rendered item.
If:
- we do not provide html_selector function
- item is not currently rendered
- new html is not a string.
then the render_item() call is a noop.
We do not shift much of the validation logic here just
yet. This function has been declared at the top of the
file to act as usage docs for the widget as well, in
terms of what combinations of opts are valid and what
are not.
We remove the "GROUP PMs" section that used
to be in the lower right sidebar.
Most of this is straightforward code removal.
A couple quick notes:
- The message fetching code now just
calls `huddle_data.process_loaded_messages`,
which we still need for search suggestions.
We removed `activity.process_loaded_messages`.
- The `huddle_data.process_loaded_messages`
function no longer needs to return `need_resize`.
- In `resize.js` we now just calculate
`res.buddy_list_wrapper_max_height` directly
from `usable_height`.
This fixes the calculation for how far from the
top of the viewport we think #right_sidebar's
top is. To fully explain this commit requires
some background info.
Normally `#right-sidebar` has 50px of top margin
and 0px of top padding. And our `resize.js`
calculations have been accurate for the normal
case.
But when you are in the so-called `.expanded` mode
(i.e. when you're in a narrow window) we split up the
50px as follows:
- 40px margin
- 10px padding
Why don't I make the CSS just be more consistent here?
- If you go to 50px in the "expanded" mode
you mostly cover up the right scrollbar,
except for the 10px gutter that is below
the 40px-tall `.header` section. To fully
cover it we apparently want the padding;
otherwise you see a small, unusable remnant
of the scrollbar which just looks funny.
- If we were to make the "regular" right sidebar
just always have the 40/10 split, then we
would start to diverge from the left sidebar,
which is currently 50/0 as well.
- If we went to make both the left and the right
sidebars 40/10 split, well, that's just an
even riskier change.
So instead I fix the resize calculation:
I just calculate the actual `top` position.
Is any of this actually user-facing?
Yes. Now if a user is a narrow window and
they open the buddy list, we will make
the buddy list 10px smaller to account for
the padding. This makes it less likely for
the invite link to get squeezed out.
We'll use this in the next commit.
Note that there's a minor change in the order
in which we apply new heights--we now
do sidebars before bottom whitespace.
We had a bunch of places where we
were calling `resize.resize_bottom_whitespace`
with no arguments, which has been a no-op
since the below commit that removed support
for our `autoscroll_forever` option:
fa44d2ea69
With the `autoscroll_forever` options things
like opening/closing the compose box could
alter how much bottom whitespace you'd want,
but we stopped supporting that feature in
2017.
Since then bottom_whitespace has just always
been 40% of the viewport size. So we only need
to change it on actual resize events.
It's worth noting that we still call
`resize_bottom_whitespace` indirectly in many
places, via `resize_page_components`, and
the latter actually causes
`resize_bottom_whitespace` to do real work,
but that work is redundant for most of those
codepaths, since they're not triggered by
changes to the viewport. So there are other
opportunities for cleanup.
The `buddy_list_wrapper` has zeros margins, so it's
just noise in the current calculations. You can
verify this pretty easily with console statements,
as well as looking at the code. I tried it with
various permutations of narrow windows and display
settings.
The header is 40px tall, with a 10px gutter
below it, which means the top of our sidebars
are 50px from the top of the viewport.
Now all the places that share these values
use `$header_right` and related values.
This is pretty easy to test out by just doubling
or tripling the two numbers at the top of the
file.
The section for `@media (max-width: 500px)` seems
to have its own smaller values for things like
the `height` of `.header`, so I left it alone.
Fixes#14962
* codecov needs `.coverage` file in the pwd to upload coverage
results.
* concurrency='multiprocessing' forces `.coverage` file to have a
data_suffix, we explicity set it to "" for the file to have no suffix.
Filed issue as https://github.com/nedbat/coveragepy/issues/989
Trigger realm icon upload by clicking on realm icon element itself
rather than having a big upload button and to match our user avatar UI.
Added new spinner over the icon element itself to show while
uploading realm icon.
The bug this was working around does not affect our current toolchain,
as confirmed by grepping through the minified output.
(Also, this linter rule only matched calc(x + y) with two arguments
and we were already using calc($far_left_gutter_size + $left_col_size
+ 4px).)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit changes do_change_user_role to support adding or removing
the realm owner status of user and sending an event.
We also extend the existing test for do_change_user_role to do a bit
more validation to confirm the audit log records all values of role.
The new realm_owner role is added as option for role field in
UserProfile model and is_realm_owner is added as property for the user
profile.
Aside from some basic tests validating the logic, this has no effect
as users cannot end up with set as realm owners.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.
We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.
Fixes: #12629
Co-authored-by: Arunika <arunikayadav42@gmail.com>
This tests if a user, that is already registered, is
redirected to the login page when they click on an
invitation.
Co-authored-by: Arunika <arunikayadav42@gmail.com>
Tests attached a UserProfile to confirmation objects,
which is not very valid as this is the only place
where this is done. Now we attach PreregUser to
the confirmation object, making the tests correct.
Co-authored-by: Arunika <arunikayadav42@gmail.com>
This commit has us piggy back the conditional for narrow_description
off of the conditional for sub_count, the reason for this approach is
that "narrow_description" needs to handle four unique cases:
- The stream exists and has a description.
- The stream exists and does not have a description.
- The stream does not exist and we must render appropriate text.
- We are not in a stream narrow and the span should not be rendered.
By piggy backing off of sub count we can get the first and last case,
with the inner conditional (on rendered_narrow_description) handling
the second case and the tab_bar.js passing appropriate values to the
template to handle the third case.
This unfortunately makes the template more brittle such that breakage
of the subcount can cause breakage (non rendering) of the description
as well.
Since we are doing a assert call checking a url without no page.waitForSelector
or similar calls, we add this to avoid flakes where the assert is called
when we are in process of navigation to the page. Also, we use Promise.all
here, and call page.waitForNavigation first, to avoid race conditions where
form is submitted quickly and navigation occurs and then we are stuck waiting
for a navigation. There were no flakes in the CI related to this issue, but
I expect someone with a slow hardware would probably stumble upto this in future.
We were missing the step for clicking the checkbox to show the subdomain
input field. There was a flake because of this issue in CI where the
"testsubdomain" input was getting typed into the name field. This fixes
that flake.
There were two reason there were flakes in login tests. One problem was
that the assert call for checking that the login page url was happening
too early in rare cases. To fix this issue we wait for email input field
element in the login page. The second issue was the sometimes the url was
/login/ instead of /accounts/login/. The time between the redirect were
super close here too (around ~10ms) so I am suprised this didn't occur flake
locally more often.
Server logs that show the redirect:
2020-05-26 18:55:27.065 INFO [zr] 127.0.0.1 GET 200 36ms (db: 5ms/2q) (+start: 14ms) /accounts/login/ (unauth@zulip via ?)
2020-05-26 18:55:27.074 INFO [zr] 127.0.0.1 GET 200 28ms (db: 3ms/2q) (+start: 23ms) /login/ (unauth@zulip via ?)
To take a screenshot on failure where we have our common error
handling code in common.run_test method we need to have access to
the Page instance. Currently, we create a new Page in the test method
we pass into run_test, so we pass in a new Page instance to that method
so the test and the run_test method have access to the same Page. And,
then we take a screenshot on failure. It will be saved as `failure-${num}`
in var/puppeteer directory.
This is mostly just moving code from the `activity.js`
tests, but I also now explicitly cover the "100%"
use case (i.e. all four folks in the huddle are present).
Computed indexes into these raw objects should be guarded with
Object.prototype.hasOwnProperty; make our accessors do this
automatically and use them consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous styling was brittle and ended up breaking in very small
phone-size views with the text overflowing the boundaries of the page.
The right fix is to move those heading outside the portico-header
class, since the CSS for that isn't generally appropriate here.
This is a prep commit which combines the previous `#searchbox`
block with the newly updated `#searchbox_legacy` block which
contains the modifications related to the new navbar display.
This only consists of changes to `#searchbox` and is still broken.
But it integrates the searchbox with the new tab_bar changes so that
only one searchbox is shown (instead of two, previously).
This is helpful because if the user pastes multiple queries in the
searchbox and there are invalid search operators, then it is visible
through the typeahead.
The main reasoning for this change is as follows:
* When the search bar contains multiple search queries
but no search results, the last search operand does
not get displayed.
This happens due to the fact that filter object
contained 2 terms having the operator key value as
"search" instead of a single term where operator is
"search" and operand is a single string containing
the space seperated search queries. This condition
occurs for search_pills_enabled case only because
we used to Filter.parse the query twice
(once for the `base_operators` and once for the
`suggestion_operator instead of doing both at once).
Thus the `search_query` value inside the
`narrow.show_search_query` function which only
selected the operands of the first term displayed
an incomplete result.
* Another benefit of this commit is to display the narrow
operators in the URL fragment the same way as when
search_pills_enabled = False.
For example, On entering the queries in the mentioned
order -> 'is: starred', 'abc', 'def', 'is: private',
'ghi'. This is the URL:
Previously:
/#narrow/is/starred/is/private/search/abc.20def/search/ghi
Now (same as pills disabled case):
/#narrow/is/starred/is/private/search/abc.20def.20ghi
* We are also able to de-duplicate the non-typeahead search
query code path.
As mentioned in the comment for `KEY.BACKSPACE` event
in `input_pills.js`, we do normal character deletion
if there is input present. However this wasn't the case
if spaces were present. Also the input wasn't cleared
after the last pill was removed.
Thus `trim()` is removed from the input length check and
the new pill is still created from the trimmed value.
We can remove the typeahead by clicking outside the search box
after we have entered the search string to be filtered and then
focus on the searchbox and press enter or just by pressing enter
on an empty string.
Previously, the narrow would just deactivate for the above condition
as the searchbox value which was passed as the raw_operators parameter
to the narrow.activate function was empty.
This happened because we called the activate function on pressing
enter for the keyup event, while the keydown event in the parent
container made a pill from the text and cleared the input. (as
mentioned in the comment for `KEY.ENTER` case in `input_pill.js`)
The /apps page webapp link now takes the user to /accounts/go to find
their organization's login page, rather than failing to do anything.
Fixes#14977.
Commit 9b78a73e36 (#15005) made some of
our poorly written Casper tests fail. Now they’re just as poorly
written but passing again.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.
Fixes: #12281
The people.js tests were using _add_user function to add
cross realm bots. The problem is that _add_user function
doesn't properly simulates the adding process as it doesn't
add the user in cross_realm_dict as well.
To solve this and eliminate the need of calling
people.initialize(), which means the params obj needs to be
defined, we extracted the whole logic of adding a cross realm
user into a separete function, add_cross_realm_user.
This fixes some issues with unclear terminology and visual styling in
the pages for the new free trial.
There's probably more we can and should usefully do in the future.
This makes it so that search_suggestion.js
does not depend on activity.js.
That dependency hasn't really been "elegant"
for quite some time, but it will become particularly
unnecessary when we go to remove the "Group PMs"
section from the right sidebar.
This commit introduces a temporary wart
where we have these two functions with the
same name in a sort of unnecessarily
complicated code stack:
activity.process_loaded_messages
huddle_data.process_loaded_messages
But we will eliminate the former function
very soon, and our message-related codepaths
will just call the `huddle_data` version
directly.
TESTING NOTES:
Now that `huddle_data` is a tiny leaf
module, it's super easy to just use the
real implementation of what was formerly
called `activity.get_huddles()` (and is
now in `huddle_data`).
When I first wrote this commit, introducing
the real implementation of `get_huddles` exposed
some bugs that I fixed in the immediately
prior commits to this.
When the tests were originally written,
I believe `activity.js` had some annoying
`jQuery` dependencies that made it hard
to unit test against. We've slimmed it over
time to be mostly just a "controller" module.
But even in its current state it would have
been a bit of a bloated dependency.
The other friction for using the actual
version of `get_huddles` was setting up
the message data, but that's pretty minor.
If you have a group PM where some users have
three-digit user_ids and some with four-digit
user_ids (or similar), a huddle could effectively
be ignored when determining the order of
search search suggestions.
Basically, we need a way to canonically sort
user_ids in "huddle" strings, and it's somewhat
arbitrary whether you sort lexically or sort
numerically, but you do need to be consistent
about it.
And JS is not exactly helpful here:
> [99, 101].sort()
[ 101, 99 ]
This is a pretty obscure bug with pretty low
user-facing consequences, and it was never
reported to us as far as I know, but the fix
here is pretty straightforward.
We have had similar bugs of slightly more consequence
in the past. The reason this bug has shown
up multiple times in our codebase is that every
component that deals with huddles has slightly
different forces that determine how it wants
to serialize the huddle. It's just one of those
annoying things. Plus, bugs with group PMs
do tend to escape detection, since most people
spend most of their time either on streams
or in 1:1 PMs.
This is a pure code extraction. The current
code is buggy with respect to user_ids with
different lengths of digits, i.e. it does
a naive lexical sort instead of a numerical
sort. We'll fix that in the next commit.
We already have a loading indicator for fetching older
messages. Thus it makes sense to implement the same
for displaying newer messages.
We set the display of `bottom-messages-logo` to none,
to prevent displaying two loading indicators during
the initial message load.
Fixes#15060.
`loading_more_messages_indicator` is renamed to
`loading_older_messages_indicator`.
This is a prep commit to introduce
`loading_newer_messages_indicator`.
I consolidate most of our users toward the top
of the file, so that we don't have to clutter
up individual tests. This also avoids some
confusion where charles/maria got repeated
in different tests with different ids.
I also introduce a couple four-digit ids to
try to expose more bugs related to sorting.
Note that it's still easy to keep tests
isolated here, as we have always been able
to cheaply re-initialize `people.js` and then
add individual users back.
There are still some tests where it makes
sense to just declare users locally, especially
if we are mutating their data.
There are a few minor incidental cleanups here,
mostly involving replacing hard coded ids
with things like `maria.user_id`.
mock is just a backport of the standard library’s unittest.mock now.
The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).
The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit fixes the alignment of emoji in the navbar by removing a
redundant style which was breaking the emoji alignment.
This block is probably just a remanent from WIP development of this
version of the navbar & its inclusion on master was as an oversight.
This creates a little bit of noise in some
tests where we don't care about users, but
it's worth avoiding confusion about which
users exist at which time. Also the noisy
aspects here may actually catch regressions.
Finally, if the noise gets annoying, we can
do things like rename "Ted" not to collide
with the "Test" stream.
Using "bob" as the current user was a bad
choice, as our convention is to use "me" or
"myself" or "alice" for the current user.
It also particularly complicated the tests
around Group PMs.
Now we have both "bob" and "myself", which
makes the intentions of the tests a little
more clear.
We handle fenced code blocks in a preprocessor, and > style blockquotes
are parsed in a blockprocessor. Pymarkdown doesn't run the preprocessors
again on any blocks that it is parsing, and is unlikely to accept our
solution upstream; they intend to convert fenced_code to a block parser.
We simply run all the preprocessors on the text again, with the exception
of NormalizeWhitespace which removed delimiters used by HtmlStash to mark
preprocessed html code. To counter this, we subclass NormalizeWhitespace
and use our customized version for when it is called from a blockparser.
Upstream issue: https://github.com/Python-Markdown/markdown/issues/53Fixes#12800.
Ubuntu Focal comes with ruby 2.7 and the latest puppet
has some issues with it so to suppress puppet
warnings with ruby 2.7 we added RUBYOPT = "-W0" in
the environment.
This commit replaces fa-file-text-o with fa-file-code-o which is a
better signal for the "view source" action. It also deletes a single
line comment that had suggested the change once we moved into font
awesome 4, which Aditya Bansal <adi.bansal241996@gmail.com> helped
out in doing, first via
91962aa6ab and most recently via
75ae94e459 with several commits in
between.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.
do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
This is important, because lack of this meant that the POST request in
our tests still had the old session, with various params stored in it.
This mechanism doesn't work in reality in SAML, so the backend uses
redis to store and recover the params from redis. Without flushing the
session, these tests would fail to catch some breakages in the
redis-based mechanism.
This commit adds code to live update the message edit history.
Message edit history is fetched and rendered again if the edit
history modal is open.
This also adds 'data-message-id' attribute to 'message-history'
when opening history modal element which is used for checking
whether the history modal opened is of the message which is
edited.
Fixes#15051.
This will protect us in case of some kinds of bugs that could allow
making requests such as password authentication attempts to tornado.
Without restricting the domains to which the in-memory backend can
be applied, such bugs would lead to attackers having multiple times
larger rate limits for these sensitive requests.
Helps to see if users are often trying to login with deactived
accounts.
A use case: Trackdown whether any deactivated bot users are still
trying to access the API.
This implementation adds a new key `inactive_user_id`
to `return_data` in the function `is_user_active` which
check if a `user_profile` is active. This reduces the effort
of getting `user_id` just before logging.
Modified tests for line coverage.
Instead of plumbing the idp to /complete/saml/ through redis, it's much
more natural to just figure it out from the SAMLResponse, because the
information is there.
This is also a preparatory step for adding IdP-initiated sign in, for
which it is important for /complete/saml/ to be able to figure out which
IdP the request is coming from.
Rather than showing the "Saving" widget beside stream name, it's more
intuitive to have it in personal settings section because it's the only
section which uses `settings_ui.do_settings_change` function and we follow
having a separate "Saving" widget for each section everywhere.
"Saving" widget was working for all personal stream settings but "Mute
notifications". This was because the change to the "Mute" property follows
a slightly different path.
After a message was reset in our caches cache via message editing or
adding/removing a reaction, we were sending corrupt data to the cache
because build_message_dict (and thus build_dict_from_raw_db_row) was
improperly being called before sewing in the reaction data.
As a result, we were sending raw database data in the reaction
dictionaries, rather than the reformatted version expected by the API.
Bug introduced in 2a4c62a326.
Fixing this correctly required moving the rendering_realm_id logic one
step higher in the call chain, which is a useful refactoring anyway
(since we're no longer passing a `Message` object down)
This should help ensure everyone uses the SSH key approach for Git
authentication; the HTTPS one is basically unusable as one has to
provide one's GitHub password after every command.
This code generates the timestamp string to be shown to the user
from the given timestamp in unix format using moment.js.
We also render the timestamp in a pill.
Previously, we handled this code only in message_list_view.js.
Now we support rendering stream descriptions and some dynamic
elements can be rendered in them, so we extract this new module
and use it in both the places.
We now parse tex and latex as regular languages, highlighting them
with pygments. We only allow 'math' to trigger latex rendering,
which is in line with the documentation.
This commit shifts our timestamp syntax to be of the form:
<span class="timestamp data-timestamp="123456"></span>
since value is not a valid attribute of span elements.
This adds support for syntax like: !time(Jun 7 2017, 6:30 PM) so that
everyone sees the time in their own local timezone. This can be used
when scheduling online meetings, etc.
This adds some hardcoded values for timezones, because of there
being no sureshot way of determining the timezone easily. However,
since the main way of using the feature should be a typeahead for
entering the time, this shouldn't be cause of much concern.
Fixes#5176.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.
Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
This extends `put_dict_in_redis` to take token as an argument
and return that with the as a `key` with following key format.
Also, edit regex for token to include uppercase letters as
a token sent during apple authentication contains uppercase
letters.
Useful for Adding "Sign in with Apple" support.
The swagger validator is a basic tool to check whether our
openapi specification file follows the basic syntax. But to ensure
that our zulip.yaml file is not only syntactically compatible but
also describes our API well, we need to add custom tests. This
commit currently checks whether each endpoint has an `operationId`
and a valid tag. It also makes it easier to check for custom rules
in the future.
* Encountered `No emoji specified for status 'Pending'` when using `on_start: always`:
```yaml
notifications:
webhooks:
...
on_start: always # default: never
```
`margin-bottom` property is ignored due to the display.With
'display: inline', the width, height, margin-top, margin-bottom,and
float properties have no effect.
The currrent css of codeblocks left too much empty space between
the lines as well as indented the first character of a codeblock
slightly towards the right.
This commit fixes both of these, thus making codeblocks look closer
to what someone would expect codeblocks to look like in their text
editor.
This refactors render_javascript_code_example to avoid shelling out to
node and parse the javascript file with python instead, to get example
code snippets.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
This refactors `extract_code_example` to return a nested list
of code snippets between '{code_example|start/end}' instead of
returing a list of all the lines between '{code_example|start/end}'
markers in the code examples.
Appropriate changes have been made to render_python_code_example.
This refactors `ExamplesHandler` to avoid running examples in a loop
and add result objects to `response_data` array one by one with
`generate_validation_data`.
This file will act as the container for all JS API examples to use
in our documentation, similar to our python and curl API testing
and examples generation code.
This module has two modes of operation:
- node javascript_examples.js generate-responses
This mode runs all the examples against a server and prints the JSON
output of all the examples we ran.
- node javascript_examples.js generate-example <endpoint>
This mode prints example code for endpoints like: /users:post. We then
want to render this full example code in our docs.
We wrap the [reset] anchor tag in a button so that we can set 'disabled'
attribute on it. We change the styles to hide the [reset] button and the
pencil icon when the widget is disabled.
We also need to call `e.preventDefault()` in the event handler since now
the anchor tag behaves as a button.
* This fixes the gov find error when dicovering files.
From v2.0.16, codecov-python uses "find" to discover files which
is buggy on some platforms.
Filed issue as https://github.com/codecov/codecov-python/issues/250
Add methods to extract recent topics from received messages.
Process new messages as they are received.
Use new messages received from the server to extract recent_topics.
Node tests added.
Otherwise, if someone tries to create a droplet with a mixed case username,
the existing DNS records (which are always lowercase) would not be deleted.
Also, existing droplets would not be destroyed(--recreate) if there is a mismatch
in case. Unlike DNS records, droplets are created with case sensitive
hostnames.
Plus it's neater to have lowercase hostnames.
If the IdP authentication API is flaky for some reason, it can return
bad http responses, which will raise HTTPError inside
python-social-auth. We don't want to generate a traceback
in those cases, but simply log the exception and fail gracefully.
'tags' attribute is helpful in differentiating and grouping the
endpoints on basis of their usage. For example tags like 'messages'
help in grouping all endpoints related to messages and thus make the
api specification more user-friendly. So give tags to the endpoints
on the basis of what heading they are under in the API docs.
'operationId' helps code generators in naming functions and other purposes.
So name operationId of endpoints as their function names in python-zulip-api
if it exists else use most appropriate function name.
Part of #14100 .
In common.js, we now have a single browser instance for the whole
test. When we update the test-js-with-puppetter to spawn a single
node process, like we do for node tests, we will save time on having
to open an new browser for every test + puppetter start up cost.
We will also add some more helpers here like a method for
filling out a form easily etc.
Before fixing `test-js-with-puppeteer` to to use the right webpack
setup, redirects weren't happening properly. The line this commit
removes did a redirect to the register page. This line is removed
as the redirect will happen automatically as expected with fixing
`test-js-with-puppeteer`.
As puppeteer tests are similar to casper i.e both being frontend
screen scraping tests, we need to use the same webpack setup as
that of casper's instead of `build_for_most_tests` which is used
mostly for backend related tests.
This sets env variable `CASPER_TESTS` to `1` which does all the
work to make it use the same setup of casper. We would want to
rename that to `PUPPETEER_TESTS` and make changes to do the
same as casper webpack setup when we completely replace casper
with puppeteer.
Previously, we tried to read the value from page_params, which was just
a hack to make the calling code look cleaner. We now remove that hack
and thus, our dependency on page_params existing. Now, if the caller
does not specify a default value, we'll use the null-value.
This also creates a new init() function to cleanly wrap the code that
makes changes to the opts passed to the widget.
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.
Add the link, rather than directing to #production-help.
The problem is not the list comprehension, as the previous wording
implied, but rather the fact that data is needed from the linked
table.
Be explicit about _what_ in the QuerySet API is helpful for addressing
this -- namely, use of `select_related`.
Change in stream color occurs very rarely, and the palette is taking a lot of space in the popover.
This commit will hide the palette in default view of stream popover.
During events such as stream / topic name edit for a topic, we were
running queries to db in loop for each message for reactions,
submessages and realm_id. This commit reduces the queries to be
done only for realm_id, which is yet to be fixed.
This is accomplished by building messages with empty reactions
and submessages and then updating them in the messages using bulk
queries.
This fixes a minor regression in a very recent
commit.
In 7ad5bea3e6 I was
a little too aggressive about deactivating users.
We do want a few users who are outside the realm,
just to prevent regressions where we fail to filter
on realm. The likelihood of such regressions are
fairly low, but it would certainly be an ugly bug.
Without this change, you could get obscure
failures when logging in as Cordelia if you
modified test data by doing something
fairly innocuous like adding a new test user.
Also the complicated query here to exclude
users was flaky, since it didn't explicitly
order by any field before doing the 'LIMIT 6'.
Part of the problem with debugging this flake
was that the failure would happen for the login,
but the data actually gets changed in `setUp,
which is easy to overlook, since it's not
explicitly invoked.
We continue to keep the seat_count set to
a constant, predictable value, since some
tests are very sensitive to having 6 users.
The navbar uses rendered markdown and rendered html within the narrow
description, this inserts eg katex--html and allows rendering of
inline math formulae. Unfortunately, in the previous SCSS file, this
fact was overlooked and a generic "span" selector was used with would
target all spans within the parent element, direct descendants or
otherwise, which caused the side effect of applying padding and margin
to inner katex elements which broke appearance.
This commit replaces the "span" selector with "& > span" so that only
spans which are the direct children to the parent element are selected
and katex--html is rendered correctly.
Fixes: #14947.
This commit allows non admins to set stream post policy while creating
streams.
Restriction was there to prevent user from creating a stream in which
the user cannot post himself but this will be taken care of with
stream admin feature.
JSON.parse behaves as we want for numbers but for strings, we would
throw an error like 'unexpected token at position 0'. This meant we
couldn't read back the value set by `$input.data('val', 'text')`.
We had removed this function from the codebase when we switched to
using dropdown_list_widget. This was accidentally left as it is when
making that change.
This significantly reduces the time required to handle events like
stream & topic name edit for topics.
Verified using the Chrome Profiler for a topic with 100 messages:
With this commit: 0.64s to move the topic to a different stream.
Without this commit: 5.5s.
For unknown reasons, deleting 10,000s of ArchiveTransaction objects
results in rapidly growing memory in the job making the request in the
Django process, eventually leading to an OOM kill.
I don't understand why Django behaves that way; I would have expected
the failure mode to instead be a serious load problem on the database
server, but perhaps the way Django's internal deletion logic handles
cascading the deletes to many millions of ArchiveMessages and other
ForeignKey objects requires tracking a lot of data in memory.
The solution is the same in any case, which is to batch the deletions
to execute a reasonable number of them at once. Doing a single
ArchiveTransaction at a time would likely result in huge numbers of
database queries in a loop, which performs very poorly. So we balance
by batching deletions in groups of 100 ArchiveTransactions; testing
this in production, I saw no spike of memory usage materially beyond
that of a normal Django process, and each bulk-deletion transaction
takes several seconds to process (meaning per-transaction overhead is
negligible).
Before we used a selector to check whether the "Subscribed" tab
is active or not. The problem is that if a new sibling of the tab
switcher is inserted in the DOM and uses the same classes as the
"Subscribed" and "All Streams" button do, the selector can get
the sibling element instead the tab button.
To avoid that, we are using a variable that tracks the current
active tab instead of using the selector.
I'm not sure exactly what series of history got us here, but we were
fetching the mobile_user_ids data for all users in the organization,
regardless of whether they were recently active (and thus relevant for
the main presence data set). And doing so in a sloppy fashion
(sending every user ID over the wire, rather than just having the
database join on Realm).
Fixing this saves a factor of 4-5 on the total runtime of a presence
request on organizations with 10Ks of users like chat.zulip.org; more
like 25% in an organization with 150. Since large organizations are
very heavily weighted in the overall cost of presence, this is a huge
win.
Fixes part of #13734.
* Stream bar color logic is borrwoed from compose stream bar.
* Use flex containers to align elements and automatically set their
height to be same, them automatically filling the stream color bar
height to be the height of the select box.
* Use flex-wrap to wrap the propagate selector when out of space.
* To make sure stream select box and stream color box are closest possible,
select box has been moved under stream color box.
A separate outer span with the title text needs to be added because:
(1) The default bootstrap popover behavior takes the title as the
popover title, if provided.
(2) We need to avoid having the title area be too big in the
me_message template.
Fixes#12769.
Co-Author-By: Vaibhav <vrongmeal@gmail.com>
The query to fetch the latest user activity was missing an
`.order_by('last_visit')`. This meant that the results were being
ordered by the `id`, which resulted in us getting `update_message_flags`
action performed on the client that the user installed last, instead of
being client agnostic and fetching the "global" last
`update_message_flags` action performed by the user.
"pyr_" events are like refund events some source called it a pseudo
refund event (https://stackoverflow.com/questions/46296374/how-can-
i-get-the-original-charge-and-refund-ids-of-an-automatic-payout).
Though due to the lack of any documentation on this event I'm not
confident if this is the right fix.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Fix a bug where the compose box didn't collapse when sending a message
from the preview area by hitting the send button. The bug ocurred because
the preview area wasn't being properly cleared when this flow was executed.
This was fixed by moving the clear_preview_area function call for a place
that will be reached by both the enter and button flow.
Fixes: #14889
Previously, we handled these updates in server_events_dispatch
and could accidentally call widget.render() before initializing
the widget.
Original report: https://chat.zulip.org/#narrow/near/875608.
The sync_realm_settings function ensures that if the settings are
not open, any updates are a noop.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.
Part of #14100.
Some examples mentioned in zulip.yaml did not match their schema.
Change either the schema or the example so that all examples are
valid with respect to their schemas.
The `email` field for identifying the user being modified in these
events was not used by either the webapp or other official Zulip
clients. Instead, it was legacy data from before we switched years
ago to sending user_id fields as the correct way to uniquely identify
a user.
Use get_release_event_message from webhooks/git.py to format release
events using the newly implemented release message template.
Tweaked by tabbott to handle name=None.
Builds on #14746. Proposed in #14934.
If the #random channel in Slack is deactivated, we should follow
Zulip's data model of not allowing deactivated, default streams.
This had apparently happened in zulipchat.com for a few organizations,
resulting in weird exceptions trying to invite new users.
If you were on "All Streams" tab and unsubscribed to a private
stream, that stream row would momentarily disappear. If you
click again on "All Streams" button, it would appear again.
The problem was that the selector was finding two elements
instead of just the tab element. To solve this we used a more
specific selector to make sure we are getting the Subscribed
tab only.
This commit fixes the bug of incorrectly showing/hiding the
realm logo delete button by using realm_night_logo_source for
checking the source of night mode logo instead of previously
used realm_logo_source for both day and night logos.
When a user changes its avatar image, the user's avatar in popovers
wasn't being correctly updated, because of browser caching of the
avatar image. We added a version on the request to get the image in
the same format we use elsewhere, so the browser knows when to use the
cached image or to make a new request to the server.
Edited by Tim to preserve/fix sort orders in some tests, and update
zulip_feature_level.
Fixes: #14290
We remove the `owner` field from `page_params/realm_bots`
and bot-related events.
In the recent commit 155f6da8ba
we added `owner_id`, which we now use everywhere we need
bot owners for.
We also bump the `API_FEATURE_LEVEL` to 5 here. We
had already documented this in the prior commit to
add `owner_id`.
Note that we don't have to worry about mobile/ZT clients
here--we only deal with bot data in the webapp.
GitHub supports opening a draft/WIP pull request and then marking it
as ready for review later on. This PR supports the ready_for_review
action for pull_request events.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
* Remove old topic and reprocess both old and new topic to ensure
that we are correctly storing the last_msg_id of users in the
topic. Also, Handle topic's stream (& topic) edit updates.
* Add function to get all messages in a topic in message_utils.js.
* Send topic edit event to recent_senders.
* Add func get sorted list of recent_senders to topic.
The function will be useful to handle topic edits in Recent Topic UI.
This commits improves how we handle <a> tags within the navbar
description. The code previously overlaid click regions on top of each
other, which was messy and probably somehow buggy.
It is cleaner if we just check if the click was on an <a> tag or not.
* This feature is currently only visible to admins.
* Locally echoed messages are also updated.
* Add UI for editing stream if user is admin.
* Show propagate mode selector if either stream or topic changed.
We use this new widget in bot settings panels
(personal and org). It lets you re-assign a
bot to a new human user.
Ideally we can improve this code to use
our existing list widgets to make it more
performant for realms with lots of users.
We no longer use `/json/users` in the codepath
for bot settings (admin side).
We also specifically don't load human users when
we load bots, so you no longer have to pay for
the server round trip as a side effect of loading
bots. Instead, there is a dedicated `set_up_bots`
entry point.
We also get the bot ids directly from `bot_data` now.
This commit, to some degree, builds on the prior commit
that had us hydrate data from `people.js` instead
of the payload from `/json/users`.
Our `list_render` list widget gives us the
option to use ids as our "list" and then
hydrate that list on-demand with an
`opts.get_item` function.
We now use that for the bots list, passing
in `bot_info` as that option.
And, importantly, we are now actually
hydrating the bot data from `bot_data.js`
data structures, and not `/json/bots`.
Using the `get_item` scheme has a couple
benefits:
- Our sort functions are based on the
actual items that we use to build the
template, so there's a bit less
code duplication. (Generally, the
data that we pass in to the template
is "finalized" in some sense, such
as the bot owner name.)
- We are less likely to display stale
data.
- We are less likely to wire up filters
to intermediate data elements that are
not actually displayed to users (think
of email vs. delivery_email).
We do rely on `get_item` (i.e. `bot_info`)
to be inexpensive, which it should be.
Note that we haven't completely decoupled
ourselves from `/json/bots`, which we still
use as our source for bot user_ids. We will
fix that in the next commit.
We want to move toward having list consumers
pass us in a list of ids that we hydrate later
in the process. This should help live-update
scenarios. The next commit will describe the
benefits in a bit more detail, using the
concrete example of our bot settings table
in the org settings.
A slightly longer-term goal here is to be
able to ask `list_render` to re-render a particular
id, and this moves us closer to that. But even
before that, this change should eliminate a class
of bugs dealing with stale data, such as when
you manually patch a list (with direct jQuery
hacks) but then later go to sort/filter the rows.
We will now re-hydrate the items in those scenarios.
We don't really need to know whether we've loaded
the user-related panels, since we only used `meta.loaded`
for a tiny optimization to avoid a jQuery lookup.
We rely mostly on the list widgets from `list_render`,
and they are smart enough to repopulate themselves
when they're called subsequent times.
For the below payloads we want `owner_id` instead
of `owner`, which we should deprecate. (The
`owner` field is actually an email, which is
not a stable key.)
page_params.realm_bots
realm_bot/add
realm_bot/update
IMPORTANT NOTE: Some of the data served in
these payloads is cached with the key
`bot_dicts_in_realm_cache_key`.
For page_params, we get the new field
via `get_owned_bot_dicts`.
For realm_bot/add, we modified
`created_bot_event`.
For realm_bot/update, we modified
`do_change_bot_owner`.
On the JS side, we no longer
look up the bot's owner directly in
`server_events_dispatch` when we get
a realm_bot/update event. Instead, we
delegate that job to `bot_data.js`.
I modified the tests accordingly.
This fixes the fact that we update the bot table
with the owner's email instead of a name, but as
the TODO indicates, this is not a full fix, since
I don't linkify the owner name.
To do the full fix properly, I want to make it
so that the `list_render` widgets can just be given
an id of a row to update, and that's coming soon,
hopefully. If I get sidetracked, the ugly ways to
do this are one of the following:
- just duplicate what the template does in
jQuery
- extract a partial to draw the bot owner link
The full solution here should fix ALL the live
update code in `update_user_data`, which is why
I'm hesitant to add any interim complexity.
This is just a lexical change. We are going
to use some shared code soon that we don't want
to export, and if `update_user_data()` is
declared too early in the file, then the function
we extract will either need to be exported (to
satisy the linter) or placed far away from its
most natural siblings.
We will use this for a patch to the live-update
code, and it also de-clutters `bot_info`.
This function could plausibly live in `people.js`,
but it's not worth the indirection at this time,
and, also, one of the upcoming callers to the
function will only temporarily need it.
There's a little bit of a chicken/egg problem
going on:
- It's hard to have nice system-wide
APIs related to bots while bot settings
are still in flux.
- It's hard to clean up the bot settings
code while the system-wide API is still
kinda messy.
But I'm making slow progress on that front.
We were still sorting them with the generic alphabetic
sort due to the markup, despite passing in a custom
sort to the `list_render` class.
The `sort_email` helper often behaves like a generic
alphabetic sort, so this fix is mostly just making the code
do what it claims to do (and it's consistent with how
we already sort active users).
The nuance with emails is whether we display real
emails or system-generated emails.
Apparently, the change and test I added before didn't quite cover the
corner case that was broken. This does, and exposes a second bug as
well, which we fix.
When editing a message where we mention a usergroup, we would remove
the 'mentioned' flag from messages, resulting in the message being
hidden from your mentions in the UI. This was reported by Greg Price in
https://chat.zulip.org/#narrow/stream/9-issues/topic/missing.20mention.
We add the same code that we use in do_send_messages to calculate the
updated mentions_user_ids. We add some tests alongside other user group
mention tests in test_bugdown.
Instead of taking a subsection option and calling the settings_org
function to update that subsection, we now take a callback function
as on_update. Also, we now store the value initial value of the
widget in opts.value instead of reading again from page_params.
These changes allow us to use this widget outside of settings_org
and for values other than settings that are in page_params.
Since notify is mostly depreciated in the latest versions of CircleCI.
Although we can use use notify in CircleCI 2.0 but currently
there is no documentation regarding it.
We could have use notify here rather than this hacky solution but
if we use notify it was not possible to trigger CircleCI webhook only
for the main Zulip repository.
Also corrected the circle ci webhook for the case where we don't receive
previous in post request
Slack has disabled creation of legacy tokens, which means we have to use other
tokens for importing the data. Thus, we shouldn't throw an error if the token
doesn't match the legacy token format.
Since we do not have any other validation for those tokens yet, we log a warning
but still try to continue with the import assuming that the token has the right
scopes.
See https://api.slack.com/changelog/2020-02-legacy-test-token-creation-to-retire.
Using the `host` virtual package confused Puppet into reporting it was
doing work every time one did a puppet run, resulting in unnecessarily
spammy output.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries. The right structure for this is just a log file.
We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers. Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
This adds a webhook that can be used to interpret standard Slack
payloads. Since there are a ton of existing Slack integrations out
there, having a webhook which can accept standard Slack payloads can
significantly ease transition pains. Obviously this can't do everything
that Slack payloads can (particularly WRT their widgets/interactions),
but we can ingest text and parse out multi-block payloads into a message
relatively reasonably.
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.
Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.
I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.
This commit adds the support for ".jpe" file extension.
Fixes#14863
The prior version of "me" confusingly had the same
user_id as one of our bots, so I fixed that.
I also avoid using a test email of 'owner@zulip.com',
which is confusing for earlier tests where I haven't
established "me" as the actual owner of any bots.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
We now no longer have to remember that
`is_guest` is on `user` but `is_current_user`
is in `..`.
And we no longer have to remember that
`full_name` is on `user` but `display_email`
is in `..`.
We now gather all the bot info in one place, rather
than grabbing some of it during the triage phase and
then some of it later.
We also explicitly copy over the fields that we
need for the template, in preparation for two
efforts:
- We want to get data from `people.js` and
avoid the round trip to `<server>/json/users`.
- We want to simplify the template by
flattening our data. (It's really somewhat
arbitrary whether `is_admin` is a calculated
value, for example, but we currently leak
that implementation detail to the template.)
We can't flatten this data quite yet, since we
share the same template for bot users as human users,
so we'll fix the human data in a bit.
We now close on status_field in our event handlers,
so that there's no chance of writing to the wrong
status field if somebody switches panels before
we have a status to report.
We can't eliminate `get_status_field` yet, but that
will go away in a future commit.
We now create the event handlers directly in
`set_up()`, and we explicitly tie them to
each of the three tables.
The goal here is to allow us to set up
the three tables individually, and this gets
us closer to that goal.
This is a purely lexical move (apart from changing
a closure variable to an argument), which is
simply designed to make less indentation for the
reader and to de-clutter `handle_human_form`.
When editing a bot, there are only two fields
that are similar to humans--full name and
email--which are trivial.
Before this commit we used a single codepath
to build the human form and the bot form.
Now we have two simple codepaths.
The tricky nature of the code had already led
to ugly things for the bot codepath that
fortunately weren't user facing, but which
were distracting:
- For bots we would needlessly set things
like is_admin, is_guest in the template
data.
- For bots we would needlessly try to update
custom profile fields.
The code that differs between bots and humans
is nontrivial, and the code was both hard to read
and hard to improve:
- Humans don't have bot owners.
- Bots don't have custom profile fields.
The bot-owner code is nontrivial for performance
reasons. In a big realm there are tens of thousands
of potential bot owners. We avoid the most egregious
performance problems (i.e we don't have multiple
copies of the dropdown), but we may still want
to refine that (at least adding a spinner).
The custom-profile-fields code is nontrivial due
to the dynamic nature of custom profile fields,
which can bring in specialized widgets like
pill fields.
Now each form corresponds to a single endpoint:
* human -> /json/users
* bot -> /json/bots
Before we had a lot of conditional logic in
the template, the code to build to views, and
the code to submit the data. Now everything is
much flatter.
The human code is still a bit messy (more work
coming on that), but the bot code is fairly
pristine. All three components of the bot code
fit on a page, and there are no conditionals:
- admin_bot_form.hbs
- open_bot_form
- handle_bot_form
We may want to grow out the bot code a bit
to allow admins to do more things, such as
adding services, and this will be easier now.
It would also be easier for us now to share
widgets with the per-user bot settings.
Note that the form for editing human data will
continue to be invoked from two panels:
- Users
- Deactivated users
There are some minor differences between
users and deactivated users, but the shape of
the data is the same for both, so that's still
all one codepath.
We eliminate `reset_edit_user` here, since
it was never used.
One nice thing about these forms was that they
had very little custom CSS attached to them
(at form-level specificity), and it turned out
all the custom CSS was for the human-specific
form.
This is purely refactoring.
The new call tree is:
on_load_success
populate_users
handle_deactivation
handle_reactivation
handle_user_form
handle_bot_owner_profile
handle_bot_deactivation
The actual sequence of operations should be
identical to before.
When reading the calling code, it's helpful to know
that we're really just passing in a selector. The
calls to open_modal/close_modal are nicer now to
reconcile with surrounding code, and you don't have
to guess whether the parameter is some kind of
"key" value--it really just refers directly to a DOM
element.
There is nothing user-visible about this change, but
the blueslip info messages now include the hash:
open modal: open #change_email_modal
The pointer doesn't get updated when a user is only reading messages in
narrowed views. But, we use the pointer position to determine the
furthest read time, which causes the bankruptcy banner to show up even
for users who have been actively reading and sending messages.
This commit switches to using the time of the last update_message_flags
activity by a user to determine the time of last activity.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.
Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.
Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.
This change made `to_not_negative_int_or_none` redundant so removed it as
well.
Fixes: #14854
It's a preliminary step to enable message_retention_setting in org settings
UI, which is a non-limited plan only feature. So we require a page_param
property that tells us the limited-plan state of the Zulip realm.
This change makes `.get_input_element_value()` return a `undefined` instead
of `null` when `input_type` is not defined. Which also make sense
logically, as
> null: absence of value for a variable;
> undefined: absence of variable itself;
Source: https://stackoverflow.com/q/5076944/7418550
Popular email clients like Gmail will automatically linkify link-like
content present in an HTML email they receive, even if it doesn't have
links in it. This made it possible to include what in Gmail will be a
user-controlled link in invitation emails that Zulip sends, which a
spammer/phisher could try to take advantage of to send really bad spam
(the limitation of having the rest of the invitation email HTML there
makes it hard to do something compelling here).
We close this opportunity by structuring our emails to always show the
user's name inside an existing link, so that Gmail won't do new
linkification, and add a test to help ensure we don't remove this
structure in a future design change.
Co-authored-by: Anders Kaseorg <andersk@mit.edu>
Previously, we had a restriction that we could only
edit and move the topics of 7 days old messages.
This buggy behaviour is now removed as in this
commit.
Fixes#14492.
Part of #13912.
We use cloud-config for setting up the SSH keys and executing
some commands. When cloud-config sets the SSH key it doesn't override
the existing keys. So we need to set the SSH keys manually using a command
instead. This means we no longer require cloud config. We can instead
pass a bash script as the user data instead of cloud-config.
I also included a command to set the SSH key of the root.
size_slug represents the plan the droplet should be created on.
Since the new base droplet is created on the cheaper but more
feature rich new plan we have to update the slug_size as well
to take advantage of the cheaper plan.
New path() function changed the way a regex pattern
is created from urls - it adds escape backslashes,
so for testing purposes we need to take care of them
and remove them, to check if urls were tested.
Additionaly, regex patterns from urls can have
[^/]+ instead of [^/]*, so we need to take care
of it too.
It appears that a recent pika release started logging spammy INFO
output on the pika.connection and pika.channel channels, in addition
to the existing pika.adapters channel.
It's probably best to just move to WARNING-level logging for all of these.
This significantly cleans up the output when run-dev.py restarts
services due to a code change in the development environment.
When the integration was originally rewritten, support for the
deprecated webhook payloads was removed. We later noticed that some
people using Zulip were still using versions of Sentry that required
the older integration code.
Thus this commit adds back the older integration code and whenever the
Sentry webhook payload does not have a "data" field (which must be
present in all modern payloads as per the documentation at
https://docs.sentry.io/workflow/integrations/integration-platform/webhooks)
we will use the older Sentry integration code.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
We no longer have intermediate constants of
`git_described` and `zulip_version_const`.
Instead, we make a `deployment_data` dictionary
that is grep-friendly, and we just let
`deployment_repr` do simple formatting
without translating string constants.
This is pretty easy to test:
- set DEBUG_ERROR_REPORTING = True
- modify some code to throw an exception
- see error output in #errors
- use "/emails" with text-only option to view
errors
This code was bitrotted--we no longer have a file
called `version`.
The info that was probably reported when that feature
was originally written probably lives now
in `zulip-git-version`, although I didn't research
all the history here. Here is the relevant
excerpt from `version.py`:
zulip_git_version_file = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
'zulip-git-version')
if os.path.exists(zulip_git_version_file):
with open(zulip_git_version_file) as f:
version = f.read().strip()
if version:
ZULIP_VERSION = version
The file gets written as follows:
$ cat tools/cache-zulip-git-version
#!/usr/bin/env bash
set -e
cd "$(dirname "$0")/.."
git describe --tags --match='[0-9]*' > zulip-git-version || true
Here is what that might look like:
2.2-dev-2102-gf256ea39eb
Here is an excerpt from one of our recent error reports,
which demonstrates that the code I eliminated here was not
functioning (the third field is missing):
Deployed code:
- git: 2.2-dev-2028-g99ce96d49b-dirty
- ZULIP_VERSION: 2.2-dev-2028-g99ce96d49b
This fixes the main problem reported on #7868. I think
we may just want to close the issue, since the other
`nocoverage` stuff seems harmless to me.
Previously a YAML syntax error resulted in an
UnhandledPromiseRejectionWarning and a successful exit code; now it
gives a clear message and a failing exit code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In our recent navbar changes, we made it so that the
Esc key auto-closed the navbar. Unfortunately,
that code would break other typeaheads with a traceback.
One user-facing symptom was that if you drafted a PM
and started a typeahead on a recipient, then hitting
the Esc key wouldn't close the typeahead.
Now we use an `on_escape` mechanism that is specific
to the navbar typeahead, so that it's both generic and
harder to break for widgets that don't opt in to it.
See bbdc66a214 for
more details on the commit that introduced this
regression.
Note that I only call `tab_bar.exit_search` now.
I don't check the class name of the input element,
since I know that the Esc key is happening in the
context of search. And I don't blur the input,
since it's going to be hidden.
We shouldn't be checking for #zfilt here, since we haven't done
anything that should cause #zfilt to load. Instead, we verify whether
messages have loaded into the DOM (the condition we actually want) by
checking whether at least one message row is in the DOM.
We were creating errors for task keys that were
from older versions of the widget. We don't migrate
data for the widgets yet (they're all still considered
to be somewhat beta); instead, we just drop bad data
on the floor.
Rohitt and I re-tested the widget on czo pretty
extensively to verify that these errors don't show
up for newly created widgets.
The click handler for closing stream settings in click_handlers.js
is removed as overlays.js contains common logic for closing all
overlays.
'exports.close' in subs.js is removed and 'hashchange.exit_overlay'
is used in 'overlays.open_overlay' call.
Previously api_description and api_code_examples were two independent
markdown extensions for displaying OpenAPI content used in the same
places. We combine them into a single markdown extension (with two
processors) and move them to the openapi folder to make the codebase
more readable and better group the openapi code in the same place.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "include_custom_profile_fields" in
components.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "history_public_to_subscribers" in
components.
Using SCSS nesting, refactor and merge various chevron's styling
elements(".all-messages-arrow", ".starred-messages-sidebar-arrow",
".stream-sidebar-arrow" and ".topic-sidebar-arrow") in
left-sidebar.scss.
Instread of using stream_name + Intergers as topics, we now
generate topics using pos in `config.generate_data.json`.
This helps us create and test more realistic topics.
This completes the implementation of support for moving a topic to
another stream by adding a basic UI for it.
Fixes#6427, which was previously the most-upvoted issue request in
Zulip.
There are likely to be a bunch of follow-up UI improvements on top of
this change to fully flesh out the feature.
This page isn't polished properly and I'm not sure it's the best
decision tree here, but it's definitely better to have this page than
not, and we can always adjust forward.
Fixes#10033.
We've completed the production performance testing that had been
blocking us from advertising this feature, so we're declaring the
feature implemented by documenting it.
As of this commit, you need to edit fields using `manage.py shell` to
configure it and manually configure the cron job, but those should be
addressed in the next few days.
Fixes#106.
For realms with no retention policy on themselves or any of their
streams, no archiving happens, but 3 lines of logs would be generated.
That's redundant and we make changes in this commit to avoid logging
those lines if nothing of interest is happening.
For privacy-minded folks who don't want to leak the
information of whether they're online, this adds an
option to disable sending presence updates to other
users.
The new settings lies in the "Other notification
settings" section of the "Notification settings"
page, under a "Presence" subheading.
Closes#14798.
Instead of having to filter `@noreply.github.com` emails in
`get_unverified_emails`, it's good to make `filter_usable_emails`
just filter `@noreply.github.com` and handle verified/unverified
part in their respective functions because of `@noreply.github.com`
exception being a fiddly special-case detail.
Also renamed `filter_usable_emails` to `get_usable_email_objects`
as a line that gets all associated github emails is removed in
`get_verified_emails` and `get_unverified_emails` and added to
`filter_usable_emails`. The name `filter_usable_emails` suggests
that it just filters given emails, whereas here it's getting all
associated email objects and returning usable emails.
This commit extends the template for "choose email" to mention for
users who have unverified emails that they need to verify them before
using them for Zulip authentication.
Also modified `social_auth_test_finish` to assert if all emails
are present in "choose email" screen as we need unverified emails
to be shown to user and verified emails to login/signup.
Fixes#12638 as this was the last task for that issue.
This separates the part of code that gets all the emails associated
to GitHub from `get_verified_emails` in `GitHubAuthBackend`.
Improves readability of code and acts as a preparatory commit for
extending the template for "choose email" in GitHub auth flow to also
list any unverified emails that have an associated Zulip account in
the organization.
As "choose email" screen is only used for GitHub auth, the part
that deals with it is separated from `social_auth_test` and
dealt in a new function `social_auth_finish`. This new
`social_auth_finish` contains only the code that deals with
authentication backends that do not have "choose email" screen.
But it is overidden in GitHub test class to handle the
"choose email" screen.
It was refactored because `expect_choose_email_screen` blocks
were confusing while figuring out how tests work on non GitHub
auths.
Sentry has client SDKs for many programming languages and frameworks.
Sentry has deprecated their old "Raven" series of client SDKs in favor
of a new series of client SDKs following their unified API format.
As it stood, our Sentry integration was already outdated being written
for the version 5 payloads (the Raven SDKs stopped at version 6 which
is already vastly different from version 5) when the current and
prominently used version is version 7.
This commit completely rewrites the existing Sentry integration.
Tested and supported events:
- Issue created, resolved, assigned, and ignored events.
- "Sentry events" for "capture exception" and "capture message" with
the Golang, Node.js, and Python SDKs (other SDKs should also work but
only these were used for testing).
For reference:
- Old (Raven) SDK for python:
https://github.com/getsentry/raven-python
- New (Unified API format) SDK for python:
https://github.com/getsentry/sentry-python
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This adds a way to keep track of max_message_id of a
stream and fetch it using the method get_max_message_id().
This will be useful for sorting streams by most recent
activity which will be implemented in the upcoming commit.
Essentially rewritten by tabbott to have a coherent tracking system,
and provide documentation.
Part of #10794.
**Features:**
Improving `./manage.py convert_gitter_data`
- If messages have been post-processed to add a 'room' field, we
create as many streams as existing rooms.
- Messages with a 'room' field go to the corresponding stream.
- This modification is backward compatible. I.e.
+ messages that have no 'room' field go to the default stream/topic
+ messages that do, go to a specific stream
**Implementation:**
- adding a map `stream_map` to map room names to stream ids
- create as many streams as room field messages + 1 default streamFeatures:
- If messages have been post-processed to add a 'room' field to messages,
we create as many streams as existing rooms.
- Up to renaming of the default stream/topic, this modification is
backwards compatible.
I.e. messages that have no 'room' field go to the default stream/topic
messages that do, go to a specific stream
Implementation:
- adding a map stream_map to map room names to stream ids
- create as many streams as room field messages + 1 default stream
Takes advantage of https://github.com/minrk/archive-gitter/pull/5.
The major PROVISION_VERSION bump would not be needed, but it was
missing in commit 5ab62a3514 (#14834),
so I’m doing it here.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Test fails at default timeout value indeterministically
because we have to wait for the server to start and the
above tests to pass, which takes more than the Default Timeout
of this test.
Hence, 4 x Default timeout value is kept.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.
Fixes#14007.
We now use the `--streamlined` options for `run-dev.py`
when we use `test_server_running` for `test-api` and
`test-js-with-casper` (and its experimental
replacement, `test-js-with-puppeteer`).
This means we don't slow anything down with
processes like thumbor, process_fts_updates, etc.,
which aren't meaningfully exercised by these tests.
We may eventually want some tests to meaningfully
exercise those processes, and when that day comes,
we will need to add an extra argument to
`test_server_running`, probably, but until then,
we just always set `--streamlined` in that codepath.
There is actually a tool called `./tools/test-run-dev`
that we run in CI, and it will use the full mode.
It just doesn't verify much stuff--it mostly polls
the server without testing specific features.
This seems to save about 1s of the startup time on a system I use
(~10.6s -> ~9.7s).
For basic testing (either manual or automated), we
generally only need the server and tornado running.
Obviously, it's nice to test the complete system,
but if you're on a slow PC, the overhead can be
annoying.
Note that we don't launch any of these processes
in `--streamlined` mode:
process_queue
process_fts_updates
deliver_scheduled_messages
thumbor
And then by not launching process_queue, we avoid
several child processes.
Basic functionality like sending messages will
still work here.
The streamlined mode may be helpful in debugging
our generally slow server startup time. Obviously,
some of the problem with startup is the auxiliary
processes here, but removing them as a variable
could help us focus on getting the core stuff fast.
Note that we still have the webpack watcher running
in streamlined mode.
For the particular case of thumbor, note that we
modify the proxy server to explicitly print and
return an error if we get a `/thumbor/*` request.
We clean up the code related to launching
processes here.
We extract:
server_processes
We also extract these helper for webpack
stuff:
do_one_time_webpack_compile
start_webpack_watcher
And then we move the code to actually launch
them lexically within the file (so as not to
be obscured by various function definitions).
Here is the new output for displaying ports:
Zulip services will listen on ports:
9991: web proxy
9992: Django
9993: Tornado
9994: webpack
9995: Thumbor
Note to Vagrant users: Only the proxy port (9991) is exposed.
I tone down the yellow for the Vagrant warning, and I show
the web proxy in cyan to emphasize it.
I also extracted the code into a function, and I don't call
that function until after `app.listen()`. (The users probably
won't notice much difference in the timing of this message, but
the message won't show if the `listen` step fails for some
reason, which I think is what we want here.)
Also, if the width of the texbox increased too much, the avatar icon
would move below. Fixed by setting max-width that matches the width of
the other textboxes.
Fixes: #14826
We remove the import-tools code that was plunked
right into the middle of our command line
arguments.
Then we add a local var called `DESCRIPTION` to
fix some ugly code formatting, and we stop with the
unnecessary `r` prefix to the multi-line string.
Previously, hanging_lists preprocessor didn't consider anything
indented at 4 or above spaces to be a list. This meant that when
we had a list like:
1. 1
2. 2
3. 3
2. 2a
1. 1a
We would insert a newline between 3. 3 and 2. 2a. This resulted
in the block processor breaeking down 1 list into 2 blocks, which
messed up the nesting and indentation for the second block.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).
It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.
This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.
Fixes#13613.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The alert-error class styling was not being applied to alert-notifications
because of incorrect styling. The alert-notifications and the
alert-error classes were applied to the same element, whereas the CSS
expected the alert-error class to be applied to children of the class
with alert-notifications. Fixed this by using the parent-selector
reference '&' to correctly select the styling.
Previously, a spinner was created and this spinner element passed to
do_settings_change function, which also created a spinner, making the
first spinner creation useless. This commit removes the spinner creation
logic and just passes the element where the spinner is to be rendered.
This fixes missing translation tags in our missed-message emails,
which is surely the most user-facing part of the production that
wasn't internationalized.
Fixes: #14398
We've had bugs in the past where users with a name in the format
"Alice|999" would confuse our markdown rendering or typeahead. While
that's a fully solvable problem, there's no real use case for that, so
it's probably simpler to just prevent users from setting their name
that way.
Fixes#13923.
Prior to this change, there were reports of 500s in
production due to `export.extra_data` being a
Nonetype. This was reproducible using the s3
backend in development when a row was created in
the `RealmAuditLog` table, but the export failed in
the `DeferredWorker`. This left an entry lying
about that was never updated with an `extra_data`
field.
To fix this, we catch any exceptions in the
`DeferredWorker`, and then update `extra_data` to
encode the failure. We also fix the fact that we
never updated the export UI table with pending exports.
These changes also negated the use for the somewhat
hacky `clear_success_banner` logic.
The trailing slash has no good reason to be there and is also
inconsistent with how we instruct to set up Audience Restriction in the
Okta SAML setup docs for the dev environment.
The emoji cache files are not available on production jobs. Also,
copying the files, creating and restoring caches doesn't add
much optimization over just rebuilding the caches here.
This will give help up write new digest only if the db rebuild
succeeds. We were relying on the caller to
be successful in building db, this was hacky and unreliable.
We write new db digest once the caller succeeds, this ensures
that we write new digest after every successful attempt.
This fixes the anomality we were facing that Databases were rebuild
on the 2nd provision attempt with no changes to files or migrations.
This was happening because we didn't write a new digest for db
after the first provision (The case of DB didn't exist).
During the 1st provision, we check the template_status() of
Database both Dev and Test, but database_exists() of Databases
obviously returned false, and we rebuild the database,
but forgot to write_new_digest and hence the anomaly in the
second provision explained above.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.
This saves about 130ms for a no-op provision.
We try to avoid importing Django settings unless
we really need them, since we want this program
to run very quickly during `provision` (when
secrets have already been generated earlier).
Our previous set of indexes for the Message table did not contain
anything to optimize queries for all the messages in a topic in an
organization where the same topic name might appear in 10,000s of
messages in many streams.
We add two indexes here to support common queries
* A `(recipient_id, upper(subject), id)` index to support
"Fetch all messages from a topic" queries.
* A `(recipient_id, subject, id)` index to support
"Fetch all messages by topic"
We use the `DESC NULLS last` on both indexes because we almost always
want to query from the "Latest N messages" on a topic, not the
"Earliest N messages".
These indexes dramatically improve the performance of fetching topic
history (which remains not good enough in my opinion; we'll likely
need caching to make it nice), and more importantly make it possible
to check quickly which users have sent messages to a topic for the
"Topics I follow" feature.
Fixes part of #13726.
This ensures that if one deletes `zproject/dev-secrets.conf`, we end
up rebuilding the databases from scratch (which, critically, will
ensure the password that gets setup matches what's in the current
version of the configuration file).
This should address a category of issue we've had where deleting
`zproject/dev-secrets.conf` would result in provision failing.
The logic in do_set_realm_property would previously "change" the email
addrssees of every user in the realm, even if they hadn't actually
changed.
We fix this by skipping the logic when it's unnecessary.
bulk_update is used to update the email of user_profile objects in
database when email_address_visibility is changed.
This helps resolve the problem of timeout errors in realms with large
number of users due to large number of database queries run in a
loop.
Since bulk_update doesn't flush caches, we need our own bit of code to
do that.
Fixes a part of #14600.
Increased opacity of the cross in night mode to make the cross easily
visible. Previously, low values of opacity caused the cross to be not
easily visible.
Fixes#14459.
We now just have two modes for setting up a dev/test
database. This makes it easy to see these things
side-by-side, when you're trying to understand how
the two different databases get built:
# dev:
USERNAME=zulip
DBNAME=zulip
STATUS_FILE_NAME=migration_status_dev
# test:
USERNAME=zulip_test
DBNAME=zulip_test
STATUS_FILE_NAME=migration_status_test
And then we make it more explicit the things that
are common between dev and test (which are
important things to understand when troubleshooting
provision-related glitches):
SEARCH_PATH=zulip,public
PASSWORD=$("$(dirname "$0")/../../scripts/get-django-setting" LOCAL_DATABASE_PASSWORD)
DBNAME_BASE=${DBNAME}_base
We lose some "generality" here, but passing in arbitrary
combinations of username/dbname/status_file to the script
would cause chaos for our digest checks, and all the different
template/base databases could cause confusion too.
This will make django automatically remove them when we run
squashmigrations. There are still some RunSQL statements which
we will have to take care of manually.
This removes the weird edit-bot sidebar, replacing it with a modal,
matching our edit-user widget (and various similar ones).
Fixes#13644 by removing the buggy code.
We add URLs to the `links_for_embed set`, only when
the `url_embed_preview_enabled` flag is turned on.
So, it is sufficient to check if `links_for_embed`
is not empty.
I imagine this can be improved in various ways, but I've initialized
this with all the **Changes** entries recorded in either zulip.yaml or
the rest of the API documentation, and I expect we'll be able to
iterate on this effectively.
It'll also be useful as a record of changes that we should remember to
document the API documentation as we document more endpoints that
currently don't discuss these issues.
While working on this, I fixed various issues where feature levels
could be mentioned or endpoints didn't properly document changes.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).
The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.
This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.
Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
This changes add_reaction in zerver.views.reactions to allow
calling POST ../messages/{message_id}/reactions api endpoint with
emoji_name only, even in the case of a custom emoji.
We now prevent these variations:
* <hr/>
* <hr />
* <br/>
* <br />
We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
Instead of figuring out the image path from the integration name in the
puppeteer script, we do it in the `generate-integration-docs-screenshot`
script and pass it as an argument to `message-screenshot.js`.
Firstly, change endpoint descriptions in zulip.yaml so that they
match their counterpart in the api docs. Then edit the api docs
so that they use api description markdown extension for displaying
endpoint description.
Add function in openapi.py to access endpoint descriptions written
in zulip.yaml. Use this function for creating a markdown extension
for rendering endpoint descriptions written in zulip.yaml.
We use this extension for a single endpoint to get test coverage.
When switching from Private Messages narrow to
All messages narrow, stream list max-height was not
correctly updated. Stream list max-height was calculated
before new height were updated by browser for
All message narrow.
Inshort:
Stream list max-height was being updated before the browser could
render height for `#global_filters`. Calling resize after narrow
completes removes this issue.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.
Also we used a hacky code in commit-lint-message for travis which is
now of no use.
Changing test_alert_words to use do_add_alert_words() and
do_remove_alert_words() from lib/actions.py instead of the
existing add_user_alert_words() and remove_user_alert_words()
as is the general policy of calling these functions when we
are updating the database.
This reverts commit 8f32db81a1.
This change unfortunately requires an index that we don't have, and
thus is incredibly expensive. We'll need to do a thoughtful reworking
before we can integrate it again.
The test-all tool is useful if you want to be really sure everything
passes, but it's almost always better to run a smaller subsuite when
doing active development, and just run test-all (or CI) at the end.
Our docs already covered this issue well, but this presents the idea
in a place one is definitely looking at when running test-all.
The post_init cache-flushing behavior in the original alert words
migration was subtly wrong; while it may have passed tests, it didn't
have the right ordering for unlikely races.
We use post_save rather than post_init hooks precisely because they
ensure that we flush the cache after we know the database has been
updated and any future reads from the database will have the latest
state.
Previously, alert words were case-insensitive in practice, by which I
mean the Markdown logic had always been case-insensitive; but the data
model was not, so you could create "duplicate" alert words with the
same words in different cases. We fix this inconsistency by making
the database model case-insensitive.
I'd prefer to be using the Postgres `citext` extension to have
postgres take care of case-insensitive logic for us, but that requires
installing a postgres extension as root on the postgres server, which
is a pain and perhaps not worth the effort to arrange given that we
can achieve our goals with transaction when adding alert words.
We take advantage of the migrate_alert_words migration we're already
doing for all users to effect this transition.
Fixes#12563.
Previously, alert words were a JSON list of strings stored in a
TextField on user_profile. That hacky model reflected the fact that
they were an early prototype feature.
This commit migrates from that to a separate table, 'AlertWord'. The
new AlertWord has user_profile, word, id and realm(denormalization so
we can provide a nice index for fetching all the alert words in a
realm).
This transition requires moving the logic for flushing the Alert Words
caches to their own independent feature.
Note that this commit should not be cherry-picked without the
following commit, which fixes case-sensitivity issues with Alert Words.
This is a precursor commit to change the name of
AlertWordNotificationProcessor to AlertWordsNotificationProcessor
to match the change from UserProfile.alert_words to Alertword.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.
For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
This will eventually let us delete a bit of annoying compatibility
code from the desktop app’s injected JavaScript.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Running the close handler won’t break anything; it’s safe to delete
from a Map while iterating through it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This exists in all versions of the desktop app that we still support,
and will eventually let us delete a bit of annoying compatibility code
from the desktop app’s injected JavaScript.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The condition was removed because in either case we
want to have the stream_row and not the sub/unsub
button, so we can always get the stream_row directly.
When a user is reading messages only in stream or topic narrows, the pointer
can be left far behind. Using this to compute the furthest_read_time causes
the banckruptcy banner to be shown even when a user has been actively
reading messages. This commit switches to using the sent time on the last
message that the user has read to compute the furthest read time.
This hack was important when only the mobile apps (and not the webapp)
were using the unread_msgs data structure and the first_unread
infrastructure. Now that the webapp is using those things, there
aren't leaked ancient unread messages that aren't accessible on the
webapp, so any few users still in this situation can get out of it by
just reading the problematic messages.
I don't think we've had a use for these tools since our unread systems
stabilized shortly after they were written, so it makes sense to just
remove them rather than updating them for the pointer migration.
In Django 2.1, the preferred way to express a nullable BooleanField
changed from NullBooleanField to passing null=True to BooleanField.
This updates our codebase to use the preferred API. Tweaked by
tabbott to update the linter rules.
The migration is a noop for Django accounting only.
Part of #11341.
We had a user have problems with the user
profile menus that you get when you click
on either sender avatars or mention pills.
If a deactivated user had a long enough email
that we would normally want to un-hide the clipboard
icon for them, we would crash inside of
`init_email_clipboard`, because the icon isn't
there for them. If the user didn't have the
console open to see the crash, the symptom
became that you would get multiple cards
visible and kind of "stuck".
I chose to fix this by just making the code
defensive with respect to the absence of the
icon, instead of short-circuiting it for
deactivated users.
It's a bit odd that we still have an element
matching `.user_email_popover` in the profile
card for deactivated users, since that element
doesn't actually include an email, but it instead
says "(This user has been deactivated)". I
considered removing the class, but the CSS
that we use for emails kind of applies here
too.
Testing this is a kind of a pain, as you want
either long emails or to just temporarily hack
this condition to true:
if (this.clientWidth < this.scrollWidth) {
// ...
}
And then test with a deactivated user, of course.
Fixes#14473
This cleans up any messages that might have been exchanged with
`NEW_USER_BOT` or `FEEDBACK_BOT` (cross-realm bots that were last
used, as far as we know, years ago) that have been completely removed
from the codebase.
Details on the algorithm are in the migration code itself.
Fixes#13583.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.
I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.
Fixes#12049.
Some sites don't render correctly unless you are one of the latest browsers.
YouTube Music, for instance, changes the page title to "Your browser is
deprecated, please upgrade.", which makes our URL previews look bad.
This allows to run scripts between extraction and install
process.
It will be used to restore npm caches for production install jobs.
We extract the tarball in the working directory so that yarn.lock and
package.json are available to restore cache.
(And also so the path is deterministic).
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.
Fixes#14644; resolves#14701.
This ancient migration imports boto, which interferes
with our upgrade to boto3.
> git name-rev f13d6a18ebf13d6a18eb tags/1.6.0~1082
We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
This ancient migration imports boto, which interferes
with our upgrade to boto3.
> git name-rev 8ae35211b58ae35211b5 tags/1.6.0~1924
We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
This ancient migration imports boto, which interferes
with our upgrade to boto3.
> git name-rev a32f666f5ca32f666f5c tags/1.6.0~2384
We can safely assume nobody is running servers on <1.6.0; there are no
supported platforms in common with 1.6.0 anyway.
We could anchor the regexes, but there’s no need for the power (and
responsibility) of regexes here.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In passing, we also get coverage for
list_render.get.
This test also implicitly demonstrates that
you can call `widget.sort` directly from some
external event handler; in other words, you
are not locked into the widget's conventions
for setting up <th> tags.
We now forbid tags of the form `<foo ... />` in most
places, and we also forbid it even for several void
tags.
We make exceptions for tags that are already formatted
in two different ways in our codebase. This is mostly
svg tags, plus these common cases:
- br
- hr
- img
- input
It would be nice to lock down a convention for these,
even though the HTML specification is unopinionated
on these. We'll probably want to stay flexible for
svg tags, since they are sometimes copy/pasted from
other sources (although it's probably rare enough for
them that we can tolerate just doing minor edits as
needed).
If folks put something like '<br/>' in the HTML,
we would think the tag's name was "br/" instead
of "br". I think we were assuming most folks
would write either "<br>" or <br />".
ASIDE:
We should probably have a consistent
preference among these styles:
* <br>
* <br/>
* <br />
I prefer the first.
The tarball will be persistent across jobs in /tmp directory.
We will use this tarball to install production server in a separate
job. This makes sure we are running the production installation
as a normal user would run it and stop any package leaks between
provision and production installation scripts.
Runs a test at the end of tools/ci/backend to check if any untracked
files have been created during this ci tests. Exits with error code 1
if untracked files are found, otherwise exits successfully.
Fixes#14691.
We recently changed our droplet setup such that their
host names no longer include zulipdev.org. This caused
a few things to break.
The particular symptom that this commit fixes is that
we were trying to server static assets from
showell:9991 instead of showell.zulipdev.org:9991,
which meant that you couldn't use the app locally.
(The server would start, but the site's pretty unusable
without static assets.)
Now we rely 100% on `dev_settings.py` to set
`EXTERNAL_HOST` for any droplet users who don't set
that var in their own environment. That allows us to
remove some essentially duplicate code in `run-dev.py`.
We also set `IS_DEV_DROPLET` explicitly, so that other
code doesn't have to make inferences or duplicate
logic to detemine whether we're a droplet or not.
And then in `settings.py` we use `IS_DEV_DROPLET` to
know that we can use a prod-like method of calculating
`STATIC_URL`, instead of hard coding `localhost`.
We may want to iterate on this further--this was
sort of a quick fix to get droplets functional again.
It's possible we can re-configure droplets to have
folks get reasonable `EXTERNAL_HOST` settings in their
bash profiles, or something like that, although that
may have its own tradeoffs.
The check here was too late, and it should
have given a blueslip error. We obviously
don't expect these errors at runtime; this
is a convenience for developers creating
new widgets.
This covers how we wire up the filter, and it
covers using `filterer` instead of `predicate`.
I also go away from the strange length-based
predicate that I had in the original test.
The intention behind the original test was
to show that filters could be more than simple
string-matching, but that was just a strange
way to demonstrate it.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`
We also update the tests to use a saner naming scheme for the variables.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
Includes this change:
* openapi/python_examples: Update get_single_user.
This updates get_single_user to pass keyword arguments to
get_user_by_id instead of passing a dictionary.
Which is required for CI to pass, as we indeed fixed the API of that
function (which had only been present with the wrong API for one release).
When clicking on the tick to subscribe to a stream,
an error occurred while trying to find the spinner
location because there are two DOM elements with the
same class, sub_unsub_button, and this made the
selector get the subscribe/unsubscribe button instead
the correct stream_row, where the tick is.
We now check whether the tick or the subscribe/unsubscribe
button was clicked, and if it was the last one we make
sure the stream_row and not the button is being passed
to the sub_or_unsub function.
We use this elsewhere in the frontend codebase with the parents
included in the i18n, so we should stick with that.
Fixes `check-capitalization` failing as a side effect.
1. Replaced the deactivate and reactivate buttons with icons.
2. Added (you) near the current user name to denote his/her account in
the entire user list.
Tweaked by tabbott to reuse the (you) formatting from the right
sidebar here for readability and consistency.
Fixes#6313.
Mobile Chrome includes the height of the address bar in its
calculation of 100vh, which was causing a corresponding part of our
content to be pushed off the bottom of the screen.
Fixes#11324.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This commit:
- Switches margin for padding on the search closed icon, to ensure we
cover the region to the right of icon as clickable area.
- Applies the click handler that initiates the search to the second
last element of the navbar:
- This will most commonly be the narrow_description element, but may
also be the entire navbar eg in the case of "ALL" or "starred".
Applying this change to user names in "group-pm-with: ..." based
narrows is a little questionable, but there are no other triggers
on these names so this change makes sense for now.
- The narrow_description may also contain links, which need to be
handled correctly so that the behave like links should. We work
around the onClick on the narrow_description, by applying a
handler to <a> tags and invoking stopPropagation.
- We also add CSS to change the cursor to a pointer to make the
search icon change color on hover over the clickable area to
indicate that the search box can be opened with a single click.
- However, since <a> tags are handled differently, we add a hover
listener which makes sure it behaves appropriately. We also increase
the vertical padding of the <a> tags so they cover the entire
vertical navbar region.
This is a finicky change; we need to adapt around bootstrap internals
to first steal focus from the list, and then if the user uses arrow
keys, send that key to the list letting bootstrap focus on the list
elements.
The reverse: stealing abck focus to the input from the list, is not
possible without changing third/bootstrap.js because it's concept of
currently selected item depends on the item being focused. We retain
the pre-commit behavior for this, where the user can SHIFT+TAB to get
back to the input and type.
Ideally, a user will now interact with this widget like this:
1. Click the button to open the widget. The input is in focus.
2. Type a query to filter the results.
3. Seemlessly start using arrow keys to select an option.
4. Press "enter" to select the option.
We shouldn't add redundant data to page_params. Since we already have
page_params.realm_notifications_stream_id, we can use that value instead
of creating page_params.notifications_stream.
We, however, still need the name of the notifications stream to render
it in templates. Thus we create stream_data.get_notifications_stream().
This commit removes most of the duplicate logic for the stream selection
dropdowns for the settings: `realm_signup_notifications_stream_id` and
`realm_notifications_stream_id`.
We also make minot changes to DropdownListWidget to accomodate the stream
rendering of the format: `#stream_name`.
We finally switch to using stream_ids instead of stream_name everywhere
which makes reading data from page_params simpler.
This commit removes can_create_streams and can_subscribe_other_users
to use has_permission as a generic function in UserProfile model for
these settings policies.
Relevant changes are made to events.py to avoid duplication at some
places.
Avatar images for bots used by the tool to generate integration
documentation screenshots are pre-generated and committed to the repository.
The `generate-integration-docs-screenshot` tool now uses these images,
instead of trying to create these avatar images on the fly.
Also, deleted the unused `create_png_from_svg` function.
'get_active_message_people` function is added which returns active
users who have sent the messages that are currently showing up in
the feed.
typeahead fetches the users from 'get_active_message_people` instead
of `get_message_people` and thus shows only active users in the
mention typeahead and excludes deactivated users.
Fixes#14310
It is more semantically accurate to remove these elements instead of
just hiding them. We were previously using a .empty().append() chain
during creation/display of these forms, hence, we clearly don't desire
to preserve the element anyway (neither are there any worthwhile
benefits of trying to).
Message_edit.js had a bug where if the inline topic_edit failed, it
would not show an error because it attempted to make a look up for
the message_id as though it were a message row edit, which would not
work. That was changed in a refactor, which made it apparent that
there was no error being rendered at all. This commit corrects it by
rendering the error, it also adds some styling to ensure the error
message is displayed inline and it makes a change to the template so
the error is rendered before the spinner.
This commit cleans up the dirty if/else structure of
handle_edit_keydown by switching to switch case statements, and also
separates the handler for inline_topic_edits and that for message
row edits.
This commit makes it so that inline (recipient bar) topic edits follow
a different path from full message row edits in `message_edit.js`.
This commit:
- deletes `.save()` endpoint and replaces all calls to it with
`.save_message_row_edit()` and `.save_inline_topic_edit()`
- deletes `.end()` endpoint and replaces all calls to it with calls to
either ".end_message_row_edit()" and ".end_inline_topic_edit()".
Some extraneous zrequires were added in
3bc818b9f7
This is not a huge deal, but it makes it
appear as if data modules are dependent
on things that they don't really care
about. The tests should provide a bit
of signal on how "deep" an object's
dependencies go.
As long as the current narrow isn't already a search narrow or empty,
we add a single space at the end of the current filter so that the
user can just press the right arrow key and begin typing their search
term, instead of having to add a space themselves.
The set_up_muted_topics_ui and templates have been
refactored to use list_render.
This is done to support filtering and sorting of
the muted stream topics.
This also includes the addition of a new Date muted header.
We have two different digest schemes to make
sure we keep the database up to date. There
is the migration digest, which is NOT in the
scope of this commit, and which already
used the mechanism we use for other tools.
Here we are talking about the digest for
important files like `populate_db.py`.
Now our scheme is more consistent with how we
check file changes for other tools (as
well as the aformentioned migration files).
And we only write one hash file, instead of
seven.
And we only write the file when things have
actually changed.
And we are explicit about side effects.
Finally, we include a couple new bot settings
in the digest:
INTERNAL_BOTS
DISABLED_REALM_INTERNAL_BOTS
NOTE: This will require a one-time transition,
where we rebuild both databases (dev/test).
It takes a little over two minutes for me,
so it's not super painful.
I bump the provision version here, even
though you don't technically need it (since
the relevant tools are actually using the
digest files to determine if they need to
rebuild the database). I figure it's just
good to explicitly make this commit trigger
a provision, and the user will then see
the one-time migration of the hash files
with a little bit less of a surprise.
And I do a major bump, not a minor bump,
because when we go in the reverse direction,
the old code will have to rebuild the
database due to the legacy hash files not
being around, so, again, I just prefer it
to be explicit.
Fixes#14595.
Invalid HTTP requests could end up in an unhandled exception in
skip_200_and_304 due the record not having the status_code attribute
set. With this change we'll avoid the exception
Example:
curl -X POST -H 'Transfer-Encoding : chunked' --data-binary 'a' 'http://zulipdev.com:9991/json/messages/57'
2020-04-21 10:56:22.007 WARN [django.server] "POST /json/messages/57 HTTP/1.1" 405 95
2020-04-21 10:56:22.007 INFO [django.server] code 400, message Bad request syntax ('a')
2020-04-21 10:56:22.008 WARN [django.server] "a" 400 -
This commit makes sure that we replace the text in the search box
every time a user calls `exit_search()` eg via the escape hotkey or by
clicking the `x` icon, so that the search box discards any input and
always starts at the current narrow.
We now have helpers for the two places where
we create databases.
There was already one helper in place, and
I gave it a more concrete name, to match
its actual database name in postgres.
So `generate-fixtures` only ever did 9 lines of code (really
3 lines of actual code) in its normal mode of operation.
But it was cluttered with lots of stuff that really only
happened when you called it with the `-force` option, which
was only invoked by `rebuild-test-database`.
Now we inline most of the code into `rebuild-test-database`.
And now `generate-fixtures` is simple (and doesn't support
a `-force` flag.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).
The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.
The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.
And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.
This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.
The other caller wasn't using `use_force`.
The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
Somewhat confusingly, we have two types of different
digests related to databases. The migration digests
are pragmatic, since changes to migrations are a bit
more frequent for certain use cases and don't
necessitate a complete rebuild of the database.
Anyway, these are just more specific names.
Versions should not be pinned in *.in unless specific circumstances
merit an exception to this rule. Every existing exception is
commented.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532. In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This refactors `extract_python_code_example` to accept an
`example_regex` parameter. It can now be used to extract code examples
from javascript_examples.py.
Upgrade libthumbor in main zulip venv. This version drops support
for python 2 and runs on py>=3.6.
As such, it is our first commit taking advantage of our having dropped support
for Debian Stretch and Ubuntu Xenial, our previous Python 3.5-based platforms.
The div containing options for filtering streams was placed in the
centre. Aligned it towards the right. Had to pass a special check
variable in subs.js:540 to add the specific class for this purpose.
This was a specific scenario where this sort of CSS was to be added,
hence had to make a specific case.
Also, fixed the bottom border color of the search streams bar for night
mode.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.
This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.
Fix this by ensuring all the files we generate are in this special
subdirectory.
This tool was part of a very ad hoc investigation
during 2017 into our JS dep dependencies.
It's very out of date, and it has a non-trivial
maintenance cost, as these type of tools seem
to come up in every code sweep.
Previously, we would always pick up the stream and topic name from
compose_state. This would work for message edits as well when the
composebox was open.
Now, if we are in a message edit, we get the stream and topic of the
message being edited before falling back on trying to populate using
the composebox state.
Fixes#14545.
Commit 35577a1f66 (#9406) moved the
OpenAPI definitions to zerver/openapi, and this script was not
updated, so it has been validating nothing for two years.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
While saving cache we used thumbor.txt in as key but while
restoring cache we used thumbor-dev.txt as the key so we could
never restore zulip-venv-cache in ci builds.
Restart postgres service if provision is called in production test suite.
This is required because terminate-psql-sessions script (used
in tools/ci/setup-production) throws error if postgres service is not running.
Restart rabbitmq service if provision is called in production test suite.
This is done to start the node as Circle CI don't start services on installation.
Removed memcached restart as flush-memcached script (which is furthur
used in tools/ci/production) throws UNKNOWN READ FAILURE if memcached is restarted
in development.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.
CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.
Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
Used postgres 10 inplace of postgres 9.5 as it is used in Bionic.
Upgraded nginx version in success-http-headers which is in CircleCI
Bionic enviroment.
This seems to be a complicated wrapper around `sed`.
It's not used anywhere or documented anywhere. I
assume it was used in some code sweep or something.
See cec45b0ae5
I built this in 2013 to help me quickly build
some sample data for our very early node tests,
I believe.
Ever since then it's just been changed for various
code sweeps. Also, if we wanted to resurrect this
idea for some reason, we now have the template
parser that was written since then.
This tool was last modified in 2016, has no
documentation, and requires `dot` to run, so
I think we can remove it to reduce some clutter
in the tools directory.
This tool is hackish and incomplete, but it's still
worth looking at if somebody wants to hunt down
obsolete CSS. There are probably better ways to
tackle this problem, so we should eventually just
remove this tool, but it's pretty low maintenance.
Current output looks like this:
$ ./tools/find-unused-css
actions_hovered
actions_link
bookend_tr
btn-skip
company-name
flatpickr-months
label_for_text
loading_more_messages_indicator_box
loading_more_messages_indicator_box_container
logoimage
messages-collapse
messages-expand
numInputWrapper
page_loading_indicator_box
page_loading_indicator_box_container
skinny-user-gravatar
sp-input
summary_colorblock
summary_row
summary_row_private_message
tutorial-done-button
Else for each retry the duplicate commits would be removed
again and again from the contributor's zulip/zulip commits.
This is a bug in the original commit
6bed6ccdcf that added the
functionality to remove duplicate commits.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that
* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;
* is reliable, unlike e.g. looking at the number of
commits since a release;
* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.
Tweaked by tabbott to extend the documentation.
Closes#14618.
Increase brightness on hover effect for "X" icon in upload
avatar area is added and change the text to "delete profile picture"
while hovering over "X" icon in the avatar upload area.
This commit changes the code to show user according to emails based
on email_address_visibilty_values and the type of user.
1. email_address_visibility = admins,members and guests
Typeaheads are shown according to original emails.
2. email_address_visibility = admins only
Typeaheads are shown according to original email to admins which
were previously shown according to system-generated email of
form "user10@zulipdev.com".
For non-admins, typeaheads are not shown according to emails as
they are not visible in the typeahead itself to non-admins.
3. email_address_visibility = nobody
Typeaheads are not shown according to emails for all type of users.
Previously, when email_address_visibility was set to admins only, the
non-admin users were not shown emails in typeaheads, while admins were
shown emails of the form 'user11@zulipdev.com'.
This commit corrects it to show the original emails in typeaheads and
input box of adding subscribers to admins and the behaviour is same
for non-admins.
This commit moves the get_visible_email function to people.js
as this function will be used in other places and people.js seems
relevant file for this.
Tests are added to get full coverage.
Due to added opacity of 0.5 through the readonly property, the button
for copy and close was not clickable. Increasing z-index of the button
solves it.
New colors are introduced in msg edit history modal for better readability
These color will have better constrat with their background colors
Fixes#13622
We now have two functions related to digests
for processes:
is_digest_obsolete
write_digest_file
In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.
In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.
Here is the typical sequence of events:
NEVER RUN -
is_digest_obsolete returns True
quickly (we don't compute a hash)
write_digest_file does a write (duh)
AFTER NO CHANGES -
is_digest_obsolete returns False
after reading one file for old
hash and multiple files to compute
hash
most callers skip write_digest_file
(no files are changed)
AFTER SOME CHANGES -
is_digest_obsolete returns False
after doing full checks
most callers call write_digest_file
*after* running a process
There's no real reason to do the lazy import any
more, as we use this unconditionally inside `main`
(indirectly), and `provision_inner` runs after we
have set up the venv.
I make these all functions for consistency,
and in particular I want to continue to avoid
`glob.glob` calls until we are actually
computing hashes.
This is mostly a prep to allow us to do
hashing in two separate places:
- check hashes
- update hashes
We would only update hashes **after** running
processes anew.
For `provision_inner` I considered using a
class to put the three path-related helpers
into a mini namespace, but it felt too heavy.
It wouldn't be completely implausible here
to extract something like a JSON config
file that has a list of globs for each
process that we do path-hashing for, but I
want to clean up other stuff first.
We now remove the `Type` and `_TYPE` suffixes,
as we will start treating this like a real
class with behavior, instead of a glorified
struct.
We pass in `platform_type`, so that we can
just derive some of our data from that,
where naming conventions apply.
And we use the name `migrations_status_path`,
instead of the name `migration_status`, which
had two different meanings before this change.
This is a pure refactor, and we just early-exit
in case the datbase doesn't exist (knowing that
that can be a bit of a lie now--see the comment
I added.)
Importing cairosvg in fails in production, because `libgtk-3-dev` is not
available. This commit moves the import for `cairosvg` into the function
where it is used. This code will soon be moved into a separate script that
will not be run on production.
We change the user facing interface to allow specifying expected
number of error messages (default=1). Now an average test can look
like:
```
// We expect 3 error messages;
blueslip.expect('error', 'an error message', 3);
throwError();
throwError();
throwError();
blueslip.reset();
```
This change adds a toggle widget to the "add streams" page that
lets the user change the sort order of the streams list. So far,
this supports sorting by stream name, by number of subscribers,
or by estimated weekly traffic.
Previously, in narrow viewports, the "filter"
option would disappear, which was very confusing.
This commit moves the filter streams input to the
next line, making it visible at all viewport widths.
@showell modified the commit message and got Casper
tests passing.
Fixes#12898.
This removes the messy click-or-drag detection algorithm originally
added in fe8f63c389, which fixed a messy
bug in an earlier algorithm from ~2013, whose sole purpose as to check
whether we're doing a selection and if so, not trigger the
click-on-message-body click handler.
The right fix is of course to do that check correctly.
We can use getSelection to distinguish between clicks and drags while
accessing the body of a saved draft. Previously, the draft would be
restored when trying to select draft text to copy/paste.
Fixes#14447.
Before this change, on clicking a checkbox to toggle subscription to a
stream no UI feedback was shown and users could toggle the checkbox
multiple times to send multiple requests causing bugs. This commit
initializes a spinner on clicking the checkbox, to provide a UI feedback
to the user. This commit also disables the checkbox once a request for
subscription has been sent and re-enables the checkbox only after a
response.
This change has been accomplished by introducing a div to display the
spinner in subscription.hbs. The corresponding styles for the spinner
have been added in subscriptions.scss. The ajaxSubscribe &
ajaxUnsubscribe functions in subs.js have been updated to show & hide
the spinners for the time the request is in process. An additional
parameter, the concerned stream object is passed to these functions(
through the sub_or_unsub function) to get the location where the spinner
is to be displayed. Finally, the checkbox click handler is updated to
support these changes.
The testing for this has been done by adding a wait of 2 secs in
actions.py for the response. This gives sufficient time to test the
working manually. Also, for error cases an error has been sent from
action.py and the behaviour has been manually observed.
Fixes#14481.
Some uploads can be rejected in the frontend, like when the file
size is too big, without sending the file to the server. Remove the
'Uploading file...' message from the compose box in such cases.
Refactored code in actions.py and streams.py to move stream related
functions into streams.py and remove the dependency on actions.py.
validate_sender_can_write_to_stream function in actions.py was renamed
to access_stream_for_send_message in streams.py.
This test was passing a string, not an Iterable[str], and effectively
a quirk in the remove_alert_words implementation happened to result in
processing each character in the string working.
Now, the system uses word='' and an editing=True for rendering an
form for addition of alert words. This is a very vulnerable
way to implement said feature and this commit fixes that.
The addition form has been moved to alert_word_settings.hbs
thereby rendering it only once but always. Now, we do not have
to manually add an empty word and editing for the form to be
rendered.
As part of refactoring, the editing parameter has also been
removed as there is no purpose left.
This change allows us to align the search icon to the right end of the
navbar, without having to rely on the `margin-left: auto` trick. This
is better because it's more convenient to let flex handle the
positioning and will, hopefully, be more resilient to breakages.
The motivation behind this change is that it is more sensible to have
search_icon styles appear first and then be overridden by
search_closed or search_open styles which appear afterwards. This is a
prep commit to correcting a bug that caused some browsers to
render the width (and hence margin and position) of search_closed
incorrectly.
This guarantees that we don’t accidentally upgrade one without the
other, which could happen for example due to different third-party
version constraints between the two.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Python 3.7 changed dict.update to avoid swallowing exceptions when
checking for the keys attribute (https://bugs.python.org/issue31572).
This broke {}.update(jinja2.Undefined()).
Fix it with an explicit default.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This updates the logged-in top navbar to display the stream/message
name, number of users, and description. It also replaces the search
bar with a search icon that expands into a full-width search bar.
Co-authored-by: Max Nussenbaum <max@maxnuss.com>
Fixes: #164.
Fixes: #5198.
In ee0d4541b4, we renamed the topic_date
-> stream_topic_history, and in the process renamed some local object
properties from .name => .topic_name, and accidentally change the
type for the data from the server as well.
The test fixtures were incorrectly migrated in the same way, so we fix
that as well.
This PR updates the puppet manifest to allow /etc/zulip to be a symlink. The current behaviour overwrites /etc/zulip if it is link to another directory, which is problematic with docker-zulip and
in particular the `LINK_SETTINGS_TO_DATA` setting.
Openapi had descriptive response codes for endpoints with multiple
responses for same response code. But this does not fall in line
with openapi specifications. So change descriptive response codes
like "400_auth" and "400_anauth" to "400_0" and "400_1" for all
such endpoints. Also make the necessary changes in openapi.py so
as to be able to read the schema in such cases and generate example
in such cases.
zulip.yaml is not in compliance with openapi specifications file.
Edit it so that it passes verification as an openapi specification
file.
Fixes#14582 .
While Hovering over some cell of the tables in settings Tab
increase the cell width since we are using the pseudo-Element `:: after`
the bug is fixed by adding `position: absolute;`.
fixes#14559
We do limit the length to 100 in the frontend, but no such check
exists in the backend.
Check that a new alert word has a maximum length of 100 in the
alert_words endpoint.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
So, `source_emoji_dump` is not the greatest variable
name, but at least we now define it relative to its
parent instead of the `.success-stamp` file. (And
then `success_stamp` is just another join.)
If the directory `templates/zerver/emails/compiled/`
is missing, then we need to run `inline_email_css`
again.
This can happen if somebody gets overzealous about
cleaning untracked files.
This is more encapsulated and more efficient.
In the cases where `is_force` is `True` or
`pygments_data.json` is missing, we now avoid
the unnecessary step of importing `pygments`, at
least up front.
(Of course, we probably import that once we generate
the artifacts.)
If somebody is having issues with provision, it's
plausible they'll do something like `git clean -fX`
to clean up old artifacts of earlier provision runs,
as part of debugging things.
We defend against this by detecting the most obvious
symptom as cheaply as possible.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.
If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.
Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.
For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.
We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
We change the message for skipping RabbitMQ
configuration to match nearby messages:
No need to run `tools/setup/build_pygments_data`.
No need to run `scripts/setup/inline_email_css.py`.
No need to run `scripts/setup/configure-rabbitmq.
No need to regenerate the dev DB.
No need to regenerate the test DB.
No need to run `manage.py compilemessages`.
To be able to get a screenshot of the personal message using the
`generate-integration-docs-screenshot` tool, this commit changes the
personal build to be triggered by iago, instead of cordelia.
The webhook view used a default value for the email, which gave
non-informative errors when the webhook is incorrectly configured without
the email parameter.
The event name seems to have been incorrectly called `todo_due_date_changed`
instead of `todo_due_on_changed`. The API docs for webhooks don't mention
the correct event name, but the TODO json payload[1] seems to contain the
`due_on` field, aside from the fixture actually referring to
`todo_due_on_changed` event type.
[1]: https://github.com/basecamp/bc3-api/blob/master/sections/todos.md
For upgrade-zulip-from-git to work, we need to be able to run
update-prod-static on production systems, which means provision code
like this cairosvg logic needs to be there for now.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).
The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.
Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
For some mobile use cases, 15 seconds is potentially too short for a
busy+slow device to open a browser and fetch the URL. 60 seconds is
plenty, and doesn't carry a materially increased security risk.
The disabled property actually prevented text selection, so it seems
better to use CSS through the `readonly="readonly"` property.
For this, swapped .prop() with .attr() since .prop() was setting it as
`readonly=""`.
`stream_topic_history` is a more appropriate name as this
module will contain information about last message of a
stream in upcoming commits. Function and variable names
are changed accordingly like:
* topic_history() -> per_stream_history()
* get_recent_names() -> get_recent_topic_names()
* name -> topic_name
If a file cannot be added for upload because of restrictions in frontend
we call cancelAll immediately in 'info-visible' callback. This would
prevent files that are already added to be cancelled but does not cancel
files that are yet to be added. So we use break to prevent any more files
from being added.
Calling uppy.cancelAll() when a batch of uploads is completed
result in the cancelation of any other batch of uploads that is
in progress. This case happens when a user uploads some files
and then tries to upload another bunch of files before the existing
upload is completed.
"description: |" supports markdown and is overall better for
writing multiline paragraphs. So use it in multiline paragraphs
and line-wrap the newly formed paragraphs accordingly.
Edited by tabbott to change most single-line descriptions to use this
format as well.
These fixtures were added in 4aa2ac1b52.
The fixture name mentions renewal as the test function name. But we
don't have any function called test_renewal in test_stripe file. This
likely means the fixtures were accidentally added. Also, deleting all
fixtures and running --generate-stripe-fixtures don't result in these
fixtures getting generated as well.
If make_end_of_cycle_updates_if_needed returns None it means the current
plan has been ended. So we should change the value of variable plan_name
only if last_ledger_entry is not None.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.
Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.
Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
If SAML_REQUIRE_LIMIT_TO_SUBDOMAINS is enabled, the configured IdPs will
be validated and cleaned up when the saml backend is initialized.
settings.py would be a tempting and more natural place to do this
perhaps, but in settings.py we don't do logging and we wouldn't be able
to write a test for it.
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.
Fixes#13340.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.
Our reasoning for deprecating support for these releases is as follows:
* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
EOL on the server in about a year.
* Debian Stretch will each EOL in 2020 (the precise date is unclear in
Debian's documentation, but based on past precedent it's in the next
few months, perhaps July 2020).
https://wiki.debian.org/DebianReleases#Production_Releases
* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
Python, which will reach EOL in September 2020 (and we're already
seeing various third-party dependencies that we use drop support for
them).
* While there is LTS support for these older releases, it's not clear it's
going to be worth the added engineering effort for us to maintain EOL
releases of the base OSes that we support.
* We (now) have clear upgrade instructions for moving to Debian Buster
and Ubuntu 18.04.
Removed the condition in the settings_save_discard_widget page which
allowed the page to be rendered only for admins. This change was
important so that this widget can be used on the user settings page
for displaying confirmation messages.
For non-admins, trying to access admin-only settings, behaviour remains
the same as before as these settings are disabled for non-admins.
The form for entering alert words has been moved above the list
of words.
The list of words will be presented alphabetically rather than
time of addition.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.
Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
We stopped using tsearch-extras in Zulip 2.1.0 after Anders figured
out how to achieve its goals with native postgres. However, we never
did a `DROP EXTENSION` on systems thta had upgraded, which meant that
backups created on systems originally installed with Zulip 2.0.x and
older, and later upgraded to Zulip 2.1.x, could not be restored on
Zulip servers created with a fresh install of Zulip 2.1.x.
We can't do this with a normal database migration, because DROP
EXTENSION has to be done as the postgres user, so we add some custom
migration code in the upgrade-zulip-stage-2 tool.
It's safe to run this whenever tsearch_extras.control is installed because:
* Zulip is AFAIK the only software that ever used tsearch_extras.
* The package was only installed via puppet on production servers configured to
run a local Zulip database.
* We'll only run this code once per system, because it removes the
package and thus the control files.
Fixes#13612.
Added UI support for uploding the new profile picture by
clicking on the avatar rather than a button.
Added new spinner for loading indication while uploading
a new avatar over the avatar area.
Fixes#10255
The original commit here was sorting bot owners by
id, which is of course meaningless to users:
444ce74a8e
It was also returning 1/-1 in cases where the bot
owner on both sides of a comparison were missing,
which is a big no-no for sorting algorithms.
We want to avoid creating jQuery objects that just
get turned right back into strings by the list
widget, so we now have our template just include
`last_active_date` instead of kludging it in
after the fact, and we return the template
string in `modifier` rather than wrapping it.
To deal with plain HTML we switch to using
`render_now`.
Calling `render_now` leads to a more simple
codepath than `render_date`, beyond just dealing
with text.
The `render_date` function has special-case logic
that only applies to our time dividers in our
message view, which is why we were passing the
strange `undefined` parameter to it before this
fix.
The `render_date` function was also putting
the dates into `update_list` for once-a-day
updates, which is overkill for an admin screen.
We don't use this logic for drafts or attachments
either. I'm not sure how well tested that logic
is, and it's prone to slow leaks.
This commit sets us up to simplify the list
widget not to have bit-rot-prone code related
to jQuery objects.
We now:
- Skip the broken "Never" case. (The way
we were distinguishing "Unknown" from
"Never" was based on brittle checks that
were just wrong due to bitrot--see Steve
Shank on czo as an example. If we want
to make this distinction rigorous in the
future, we should have a clear mechanism.
If somebody's never actually been active,
we probably want to treat that more like
a dead-on-arrival login, anyway, and make
it easy to clean them up.)
- Use the `presence.last_active_date` instead
of reaching into private data structures.
- Avoid the unnecessary intermediate constants
of LAST_ACTIVE_NEVER and LAST_ACTIVE_UNKOWN.
- Avoid setting `last_active` in `populate_users`.
This commit was modified by @showell:
- I cleaned up the commit message.
- I simplified the diff a bit to avoid
some renaming and lexical moves.
We already know which list widget a `<th>`
tag is associated with when we set up the
event handler, so it's silly to read data
from the DOM to find that widget again
when the handler runs.
This commit eliminates a whole class of possible
errors and busy work.
For some widgets we now avoid duplicate redraw
events from this old pattern:
widget = list_render.create(..., {
}).init();
widget.sort(...);
The above code was wasteful and possibly
flicker-y due to the fact that `init` and
`sort` both render.
Now we do this:
widget = list_render.create(..., {
init_sort: [...],
});
For other widgets we just clean up the need
to call `init()` right after `create()`.
We also allow widgets to pass in `sort_fields`
during initialization (since you may want to
have `init_sort` use a custom sort before the
first render.)
Finally, we make the second and third calls
eliminate the prior updates from the previous
widget. This can prevent strange bugs with
double-reversing columns (although that's
been prevented in a better way with a recent
commit), as well as avoiding double work
with sorting.
This code has always been kind of convoluted
and buggy, starting with the first
sorting-related commit, which put filtering
before sorting for some reason:
3706e2c6ba
This should fix bugs like the fact that
changing filter text would not respect
reversed sorts.
Now the scheme is simple:
- external UI actions set `meta` values like
filter_value, reverse_mode, and
sorting_function, as needed, through
simple setters
- use `hard_redraw` to do a redraw and
trigger external actions
- all filtering/sorting/reverse logic on
the *data* happens in a single, simple
function called `filter_and_sort`
We don't use this anywhere. You can do
`git grep -A 40 list_render.create` to verify
this (with a little bit of noise in the grep).
A better strategy for generalizing
this code is to extract the useful logic
into a function that callers can use in their
own custom event handlers, which I'll do
in an upcoming commit.
We put this in `scroll_util` to make it more likely
we will eventually unify this with other scrolling
logic. (A big piece to move is ui.get_scroll_element,
but that's for another PR.)
And then the other tactical advantage is that we get
100% line coverage on it.
I changed the warning to an error, since I don't
think we ever expect scrolling at the `body` level,
and I don't bother with the preview node.
This commit moves /rest-error-handling examples to components section so
that they can be re-used in individual endpoints where it's example can
be highlighted more easiy.
The Redis-based rate limiting approach takes a lot of time talking to
Redis with 3-4 network requests to Redis on each request. It had a
negative impact on the performance of `get_events()` since this is our
single highest-traffic endpoint.
This commit introduces an in-process rate limiting alternate for
`/json/events` endpoint. The implementation uses Leaky Bucket
algorithm and Python dictionaries instead of Redis. This drops the
rate limiting time for `get_events()` from about 3000us to less than
100us (on my system).
Fixes#13913.
Co-Author-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
Co-Author-by: Anders Kaseorg <anders@zulipchat.com>
We extract a general purpose widget to create dropdown lists with
search. This widget is used for default code block language, but can
be easily extended to cover notifications_stream and similar options.
The current usage is:
```js
const widget = DropdownListWidget({
setting_name: 'realm_alpha_beta',
data: [{name: 'hello', value: 'world'}, {...}, ...],
subsection: 'msg-editing',
default_text: 'Nothing is selected',
});
```
and
```handlebars
{{> dropdown_list_widget
setting_name="realm_alpha_beta"
list_placeholder=(t 'Filter the data')
reset_button_text=(t '[Unset]')
label=admin_settings_label.realm_alpha_beta }}
```
This can further be refined by shifting more variables from handlebars
to javascript in the future.
By taking these functions out of exports.build_page, we can
reuse them for handling other widgets. We also declare
default_code_language_widget after the helper functions to
avoid the linter complaining.
I pushed this risk commit to the end of
a PR that had a bunch of harmless prep
commits at the front, and I didn't make
it clear enough that the last commit (this
one) hadn't been tested thoroughly.
For the list_render widget, we can simplify
the intialization pretty easily (avoid
extra sorts, for example), but the cache aspects
are still tricky on subsequent calls.
Changes .data() Jquery methods to .attr() to prevent unnecessary data
type conversions of the emoji name.
Tested the fix manually and verified the test-js-with-node test suite.
Fixes: #14377
For some widgets we now avoid duplicate redraw
events from this old pattern:
widget = list_render.create(..., {
}).init();
widget.sort(...);
The above code was wasteful and possibly
flicker-y due to the fact that `init` and
`sort` both render.
Now we do this:
widget = list_render.create(..., {
init_sort: [...],
});
For other widgets we just clean up the need
to call `init()` right after `create()`.
We also allow widgets to pass in `sort_fields`
during initialization (since you may want to
have `init_sort` use a custom sort before the
first render.)
The get() logic here was broken, because
when you enter the settings panel for invites
on the 2nd or 3rd time, the text filter
would not work.
This commit doesn't intend to fix the problem; it
just simplifies the code for a later commit
that fixes this holistically.
The way that we update `list_render` objects
is janky with respect to events, so we can end
up double-sorting lists (which puts them back
to normal) and strange things like that.
This is all cosmetic.
Instead of:
const.widget = {
foo: function () = {
},
bar: function () {
},
};
We have:
const widget = {};
widget.foo = function () = {
};
widget.bar = function () {
};
Before this fix, we'd get a traceback if you looked
at invites in the settings (and if one of them was
a multi-user invite link). This commit fixes
that problem by adding a custom sort.
We also rename the "Email" column to "Invitee",
since it's often the case the invitee isn't an
actually an email, but it's instead a multi-use
link.
Note that the invites UI only works the first time you
enter settings. Many of the controls break the second
time you enter it. You can't sort by column header
or use the text filter.
I'll fix that in a subsequent commit.
Giving these functions a name and moving them to
the top-level scope has a couple tactical advantages:
- names show in tracebacks
- code is less indented
- setup code is less cluttered
- will be easier to add unit tests
- will make some upcoming diffs nicer
These are technically more `compare_foo` than `sort_foo`,
but we already had a naming convention that was sort of
in place.
This defends against cross-origin session fixation attacks. Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Instead of sneakily injecting HttpOnly into the cookie via the path
setting, use the setting that was designed for this purpose.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Since commit 1d72629dc4, we have been
maintaining a patched copy of Django’s
SessionMiddleware.process_response in order to unconditionally ignore
our own optional cookie_domain setting that we don’t set.
Instead, let’s not do that.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Right now, the message is "Invalid characters in emoji name" when
the emoji_name is empty. Changing check_valid_emoji_name() in
zerver/lib/emoji.py which validates the name to accomodate the case
of missing name. The new message is "Emoji name is missing".
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.
Closes#14405
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
We had a bug where if your peer mentioned you in
message, but then edited the message not to mention
you, the latter wouldn't reset your unread counts
for "Mentions". And the same problem would happen
vice versa.
The fix basically extracts `update_message_for_mention`
and makes sure it handles all combinations of
unread/mentioned flags, instead of assuming
any invariants about which directions of change
are possible.
And then we call that new function from
`message_events.js` whenever we get message
edit events.
Fixes#14544
We use a somewhat more realistic message, mostly
to prep for testing some mention/unread stuff in
a subsequent commit.
We also set message booleans.
Unfortunately, `recent_senders` is kind of awkward
for checking a single message, since its only
public API is for sorting. I don't bother with it.
But I do check the `topic_data` interaction.
We figure out the dev host using the same logic as
dev_settings.py, so that we don't use wrong things
like 127.0.0.1 for droplet users.
And we display the link in cyan.
Breaking the Casper tests into smaller tests
will make it a lot easier in the future to
hone in on test flakes.
Having small tests adds little overhead--most
of the slowness comes from starting the server.
The only extra steps here are logging in and
entering "Manage Organization", which is two
lines of code.
We split out the custom profile test first,
since the code for custom profiles has the
annoying property that it can only run once
before failing, as it has the side effect
of creating a field name that can't be reused.
We only need to run loops to test flakes, so
this isn't an immediate blocker.
Earlier, the non-editable text-boxes(on clicking view source/edit
topic) were not so apparent due to absence of `disabled` attribute.
Adding the `disabled` attribute makes them consistent with the approach
for non-editable text-boxes and text-areas in organization settings
(for non-admins).
Fixes: #14375
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The `send_message_ajax` function was a relic
of us having an alternative way to send messages
(web sockets) to the server, but now the indirection
is more confusing than helpful.
This also fixes trying to cancel a resend of a
local message.
The problem were was type confusion between
strings and ints.
The function in `rows.js` may feel like overkill,
but I really want to enforce type safety here,
as we usually treat message ids as floats, but
for the local-echo case we're gonna get
strings. I put it in `rows` because we mostly
do a good job of encapsulating the "zid" role
in the DOM there.
By going directly to the DOM here, we avoid
parsing a string to a float and then converting
it right back to a string, which always make
me queasy about float rounding, so one less
moving part.
Due to type confusion, we were silently failing
to delete local_id values for messages that were
being acked by the server.
This used to work when we kept values in our
old Dict data structure, since client_message.id
and message.local_id are really the same value,
just the former is a float and the latter's a
string, and Dict never cared.
We can avoid all this confusion, though, by just
consistently using `local_id`, which I extract
to a local var.
The function message_send_error was messing up
on calls to message.get when we were passing in
string versions of `local_id`. Now we pass in
float ids.
This fixes a traceback where we tried to set
`.failed_request` on to an `undefined` value
that we had instead expected to be a locally
echoed message from our message store.
This will allow us access to the float version of the
message's id in an upcoming commit, without us having
to do possibly brittle string-to-float translations.
We stop using `local_id_counter`, which was just noise,
and instead we just make the test more realistic:
- Use 123.04 for our local id on the message that
we're simulating sending.
- Use 127 as the message id that the server gives
us back in the success payload.
We still stub echo functions, but for
one of our stubs (`try_deliver_locally`)
we now exercise one its actual callees
in the stub (`echo.insert_local_message`).
And we're still stubbing some callees
of `echo.insert_local_message`, since
that has all kinds of unwanted side
effects, too.
The main piece we want from
`insert_local_message`, for now,
is somewhat realistic handling of
our local message ids.
We also add a little sanity check
that our timestamp does get plumbed
through to `local_message.insert_message`.
Option is added to video_chat_provider settings for disabling
video calls.
Video call icon is hidden in two cases-
1. video_chat_provider is set to disabled.
2. video_chat_provider is set to Jitsi and settings.JITSI_SERVER_URL
is none.
Relevant tests are added and modified.
Fixes#14483
This adds a new realm setting: default_code_block_language.
This PR also adds a new widget to specify a language, which
behaves somewhat differently from other widgets of the same
kind; instead of exposing methods to the whole module, we
just create a single IIFE that handles all the interactions
with the DOM for the widget.
We also move the code for remapping languages to format_code
function since we want to preserve the original language to
decide if we override it using default_code_clock_language.
Fixes#14404.
The filter-linkifier input box was disabled which prevented users from
filtering through the linkifiers list. Removed the part of code which
caused the input box to be disabled. This allows users to edit the input
and so filter linkifiers.
With EMAIL_ADDRESS_VISIBILITY_NOBODY (or as a non-admin with
EMAIL_ADDRESS_VISIBILITY_HIDDEN), we were incorrectly generating
zuliprc files containing the shareable email address, which naturally
didn't work.
We use retry_event in queue_processors.py to handle trying on failures,
without getting stuck in permanent retry loops if the event ends up
leading to failure on every attempt and we just keep sending NACK to
rabbitmq forever (or until the channel crashes). Tornado queues haven't
been using this, but they should.
Semaphore has currently has two different versions of their product -
Classic and 2.0. This commit adds support for Semaphore 2.0, along side
Semaphore Classic, using the same webhook. This would let the integration
work seamlessly for users who have already configured a Zulip integration in
their Semaphore 2.0 projects.
Semaphore 2.0 currently only supports GitHub and their payloads do not
contain URLs for common entities like commits, pull requests and tags. We
construct URLs for them using templates, but also try to support other
services by providing notifications without URLs.
Closes#14171
Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in>
Because Franz has been to fix their broken Zulip recipe, and as a
result, currently the Franz/Fedri Zulip integration throws an
exception on every new message (fixed in
https://github.com/adambirds/recipe-zulip/pull/6/files), it is
probably worth re-introducing a version of the element it's looking
for to unbreak it.
The change in 180d8abed6, while correct
for the Django part of the codebase, had the nasty side effect of
exposing a failure mode in the process_notification logic if the users
list was empty.
This, in turn, could cause our process_notification code to fail with
an IndexError when trying to process the event, which would result in
that tornado process not automatically recovering, due to the outer
try/except handler for consume triggering a NACK and thus repeating
the event.
Most failures result from invalid emoji names, so this makes it easier
to recover without re-uploading a file.
Previously, this model would have been problematic, but now that we
have the visual preview, this is clearly better behavior.
This adds a preview of the uploaded emoji image while uploading custom
emoji right below the upload form.
Modified upload_widget.build_widget() to take in the preview
span text and image. In case a parameter isn't passed
for preview text, it defaults to null and the snippets in
build_widgets() related to preview don't run.
Fixes#9229.
Styling tweaked by tabbott.
We now only compute idx on the outbound side,
instead of spreading out the responsibility.
We just iterate through all our items to find
the next available number.
The name here is a bit more precise, as we're
not checking whether a task exists so much
as whether just a particular name is in use.
We also move the function out of the `check_task`
layer, which feels a bit overkill in terms
of nesting (plus, we're gonna remove the other
function inside of `check_task` soon).
If a request fails the tool sleeps for some time before making
further requests. The sleep time is a random number between
0 and 2^failures capped at 64 seconds. More details about the
algorithm can be found at https://chat.zulip.org/#narrow/stream/
92-learning/topic/exponential.20backoff.20--.20with.20jitter
This is a prep commit for changes to the top navbar, it adds helpers
to filter.js which will help control the behavior of some aspects of
the redesigned navbar.
Modified by tabbott to add comments, internationalization tags on the
strings, support streams:public, and change various title strings.
This is a prep commit to the change of the navbar UI, in the new UI
the navbar and search box are toggled by icons and exist in the same
space on the UI. This commit only moves the search_box styles to be
near the tab_bar styles, so that future changes are easier to make,
read and maintain.
We fix this by adding a more expressive data function, with tests, for
whether a filter is on UserMessage data, which would mean that
streams:public could never add additional matches.
If we had a rule like "max 3 requests in 2 seconds", there was an
inconsistency between is_ratelimited() and get_api_calls_left().
If you had:
request #1 at time 0
request #2 and #3 at some times < 2
Next request, if exactly at time 2, would not get ratelimited, but if
get_api_calls_left was called, it would return 0. This was due to
inconsistency on the boundary - the check in is_ratelimited was
exclusive, while get_api_calls_left uses zcount, which is inclusive.
time_reset returned from api_calls_left() was a timestamp, but
mistakenly treated as delta seconds. We change the return value of
api_calls_left() to be delta seconds, to be consistent with the return
value of rate_limit().
The information used to be stored in a request._ratelimit dict, but
there's no need for that, and a list is a simpler structure, so this
allows us to simplify the plumbing somewhat.
That's the value that matters to the code that catches the exception,
and this change allows simplifying the plumbing somewhat, and gets rid
of the get_rate_limit_result_from_request function.
We now use `assert.throws()` to test that we're
properly calling `blueslip.fatal`.
In order to not break line coverage here, we have
to remove an unreachable `return` in `stream_data.js`.
Usually we test `fatal` for line coverage reasons.
Most places where we use `blueslip.fatal` fall in
these categories:
* the code is theoretically unreachable, but
we have `blueslip.fatal` for defensive reasons
* we have some upstream bug that we should just
fix
* the code should recover gracefully and just
use blueslip.errors()
It's possible that we should eliminate `blueslip.fatal`
from our API and just throw errors when really important
invariants get broken. This will make it more obvious
to somebody reading the code that we're not going to
continue after the call, and `blueslip` already knows
how to catch exceptions and report them.
The todo_widget was using the using a counter to store the key value of
every task. This would cause assiging multiple tasks the same key value
in a race condition. To avoid this we make "sender_id" a part of the key
along with the counter.
Also the `key` now not being a integer value, we can't use it to find the
index of the task using it. Thus, a function is made that will find the
index of task whose key is sent by the user to strike.
This commit contains a few clean ups:
* In order to scale better for adding multiple commands,
the message formatting and setting switch logic was
extracted to its own function.
* The command lists were removed, as the frontend parses
the slash command from the compose box, and only sends
a single command to the backend for any given command
alias typed.
* The `switch_command` logic was removed because, given
the aforementioned fact, the index of the command will
always be the same. Thus the switch command will always
be the same.
* Switched to using early returns as opposed to nested
conditionals. Along with removing single use variable
declarations.
This is a prep commit for generating /team page data
using cron job. zerver/tests directory is not present in
production installation. So moving the file from the directory
tests to tools.
This is part of #6427, adding support for live-updating the Zulip UI
to move messages to a new topic.
As noted in the comments, there is still a bug to be fixed here
involving guest users, but the overall implementation is pretty well
tested manually (which is how we test most message-edit UI work since
there's so much complexity involved).
Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
This commit reuses the existing infrastructure for moving a topic
within a stream to add support for moving topics from one stream to
another.
Split from the original full-feature commit so that we can merge just
the backend, which is finished, at this time.
This is a large part of #6427.
The feature is incomplete, in that we don't have real-time update of
the frontend to handle the event, documentation, etc., but this commit
is a good mergable checkpoint that we can do further work on top of.
We also still ideally would have a test_events test for the backend,
but I'm willing to leave that for follow-up work.
This appears to have switched to tabbott as the author during commit
squashing sometime ago, but this commit is certainly:
Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
This commit corrects the test_change_stream_policy_requires_realm_admin
by setting the date_joined of user in the tests itself.
test_non_admin is added to avoid duplication of code.
Code is added for checking success on changing stream_post_policy
by admins.
We no longer delete existing drafts if you happen
to clear the text in your compose box for a message
that was restored from an existing draft. This
prevents folks from losing drafts when they accidentally
delete selected text.
There are still two ways to delete a draft:
* send the message (obviously not always desirable)
* use the drafts UI (with `d` as a shortcut to bring it up)
See https://chat.zulip.org/#narrow/stream/9-issues/topic/lost.20draft
for more discussion.
This commit modifies the padding and margin of the input selector so
that is uses sane values such as 25ps top margin, 5px bottom margin
and 10px top and bottom padding rather than trying to make uneven
values balance each other out. (old values are 25px top margin, 4px
bottom margin, 9px top padding, 11px bottom padding)
As described in the commit that added this function, this fixes one
quite annoying bug and one at least in-principle bug:
* On Windows, the simple version (lacking `git update-index
--refresh`) routinely gives false positives, making the tools
that rely on it basically unusable.
* If you have uncommitted changes in the index but manage to have
the worktree nevevertheless match HEAD, the simple version will
give a false negative and we'd blow away those changes.
This is verbatim from Git upstream, at an older version. (The one
change since then is to add localization for the messages like "You
have unstaged changes" -- which complicates the code, is important and
worth it for Git itself, but for our tools we can do without.)
This function will replace our use of `git diff-index --quiet HEAD`
in several scripts. The key differences in behavior are:
* The `git update-index --refresh`. Without this, on Windows
apparently `git diff-index` routinely (but not all the time!)
reports that tons of files have changed. See report:
https://chat.zulip.org/#narrow/stream/9-issues/topic/.2E.2Ftools.2Ffetch-pull-request.20issue/near/834435
* Instead of one command comparing the worktree to HEAD, we
separately compare the worktree to the index and the index to
HEAD, and abort if either diff is nonempty. This one is obvious,
but rather an edge case (it matters only if you've managed to
make the worktree and HEAD agree while the index has some
changes), and the extra code is annoying if written out in every
script that needs it. But that's what a subroutine is for. :-)
We'll make a few tweaks before actually switching to use this.
The Git commands we're invoking to do the real work are useful to
print, for transparency to see what's happening and that there's no
magic here.
The boring shell stuff like `remote=${2:-"upstream"}` is not so
helpful, and nor is the rather arcane and in any case read-only
command `git diff-index --quiet HEAD`. Those only add noise that
obscures the interesting parts. So, move the `set -x` down to when
we're done with the boring preparatory stuff and ready to perform
the commands that do the work.
If the subscription data was changed from the left sidebar,
we previously would attempt to display the savings indicator
in the stream edit page which wasn't rendered yet. The bug was
introduced in commit 39577b58ba.
This approach is used to harden the codepath against bugs by
keeping the expectOne check in `settings_ui.do_settings_change`
function.
Fixes#14467.
Enhance visibility of "x" to dismiss the dialog box of "You have nothing
to send!" message.
To achieve this:
Added class 'compose-send-status-close' with new color attribute in
file night_mode.scss.
Fixes: #14459
Co-authored-by: @MariaGkoulta <43913366+MariaGkoulta@users.noreply.github.com>
When we redraw the left sidebar, we need to tell the
topic list to clear its data structures (and do other
stuff like hiding its popover), since we are clearing
its parent container.
The commit f0e18b3b3e
introduced this regression in late January 2020.
That commit made topic_list use a vdom to avoid
unnecessary updates. Before that, topic_list did
a lot of brute-force redraws, which covered up the
fact that we weren't having stream_list telling it
when the rug was being pulled out from under it.
The boundary between stream_list and topic_list
has always been kind of complicated code, since
topic lists get embedded into the stream list.
The main interactions, though, are basically:
* topic_zoom.clear_topics() - you're leaving
a narrow that may or may not be zoomed
* topic_list.clear() - you're about to redraw
stream items in the unzoomed stream list
* topic_list.rebuild(stream_li, stream_id) -
you're building or updating a topic list
for the newly active stream
Fixes#14465
Users are unable to modify organization's profile picture, but
disabled buttons for the same are being shown to the user on the
organization profile settings page. This commit removes those
buttons. The file realm-logo-widget.hbs renders those buttons only
if the user is an admin and realm_logo.js has been updated to allow
operations(like click) on the buttons only to admins.
Users are unable to modify organization's logos, but disabled
buttons for the same are being shown to the user on the organization
settings page. This commit removes those buttons. The file
realm-logo-widget.hbs renders those buttons only if the user is an
admin and realm_logo.js has been updated to allow operations
(like click) on the buttons only to admins.
The nginx ‘add_header’ directive doesn’t inherit the way you’d
want (https://trac.nginx.org/nginx/ticket/854), so we need to manually
simulate inheritance using ‘include’, like we previously did with
api_headers.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The `options` parameter is not being passed in any call
of `lightbox.open()` and it uses the same option i.e.
`lightbox_canvas` everytime which is now computed inside
`display_image()` directly.
`image` passed to lightbox.open() is already a jQuery object,
so we don't need to convert it explicitly. Also, the parameter
is renamed from `image` to `$image`.
Previously, lightbox.open() was responsible for retrieving
the image data from the DOM, saving it in `asset_map` and
finally displaying the image using that data. This
implementation wasn't correct for image list at bottom of
the lightbox because the `image` parameter passed to
lightbox.open() could contain more than one instances of
the image that had to be opened.
Now, the metadata of all the images in image-list is stored
in the `asset_map` while rendering the `image-list` inside
`render_lightbox_list_images()` and `lightbox.open()` only
looks for the metadata from `asset_map`.
Fixes#14152.
In case of video embeds, the previous logic used
`data-src-fullsize` or `src` as a key to look
for the metadata of video in `lightbox.open()`,
but while parsing, the key used while storing
the metadata was the video ID.
This doesn't make any sense because video's data
could never be accessed from `asset_map` and we
always needed to lookup the DOM for this.
This commit fixes this by using $img.attr('src')
as a key for `asset_map` for both, images and
videos. Since `src` is the link of preview image
in case of video embeds, it will always uniquely
determine the video ID and we won't loose
anything with the change in how videos handle
things.
Part of #14152.
The value of `canvas.parentNode` in `sizeCanvas()`
appears to be `null` sometimes and it throwed an
exception specially when you switch images from
the images-list quickly.
This used to show a blank page. Considering that the links remain valid
only for 15 seconds it's important to show something more informative to
the user.
This breaks provisioning because running this as import time would
require language_name_map.json to be generated by `manage.py
compilemessages` before we can run any management commands :(.
We could potentially fix this in the future by changing the generate
language files to be things we commit to the project.
This is a prep commit for making use of same choices for
create_stream_policy and invite_to_stream_policy as both fields
have same set of choices.
This will be useful as we add other fields using these same types.
This commit replaces the WAITING _PERIOD with FULL_MEMBERS from
create_stream_policy and invite_to_stream_policy choices to
achieve consistency and making the variables more descriptive.
This simplifies the update_display_settings endpoint to use REQ for
validation, rather than custom if/else statements.
The test changes just take advantage of the now more consistent
syntax.
The value of start would be set to realm creation date or installation
date unless a value is explicitly provided by the user. We don't
want to show analytics data missing error in both these cases if
start is less than 24.5 hours from timezone_now() since some of the
stats are populated by a daily cron job that may take a couple of
minutes to run.
The same case applies if the value of the start is passed in the
request.
This changes the payload that is used
to populate `page_params` for the webapp,
as well as responses to the once-every-50-seconds
presence pings.
Now our dictionary of users only has these
two fields in the value:
- activity_timestamp
- idle_timestamp
Example data:
{
6: Object { idle_timestamp: 1585746028 },
7: Object { active_timestamp: 1585745774 },
8: Object { active_timestamp: 1585745578,
idle_timestamp: 1585745400}
}
We only send the slimmer type of payload
to clients that have set `slim_presence`
to True.
Note that this commit does not change the format
of the event data, which still looks like this:
{
website: {
client: 'website',
pushable: false,
status: 'active',
timestamp: 1585745225
}
}
Next to the checkbox of "Show previews of linked websites" added a
documentation link in order to help users not confuse its function.
To do this:
- Added the field 'help' which is the link for the documentation in
organization_settings_admin.hbs
- Added the if statement in settings_checkbox.hbs to check if the above
field exists.In case it exists, a help icon which leads to the documentation
link appears.
The specific field was added in order to be able to add a help link in
other settings as well.
Co-authored-by: Katerina Perikou
<44238834+kPerikou@users.noreply.github.com>
Fixes: #13450
This adds a new test_realm_integer, replacing test_realm_boolean for
testing integer fields like realm_create_stream_policy,
realm_invite_to_stream_policy, and realm_invite_required in dispatch.js.
Fixes#12284
Fields like realm_email_address_visibility and realm_bot_creation_policy
were strings instead of integers in page_params obeject in
settings_org.js tests.
Also use values struct defined in settings_config.js and setting_bots.js
instead of direct values for improving readability.
Fixes the problem of recipient_bar_icons being too close to each
other. To improve spacing between them, classes are added, namely
recipient_bar_icon_link (for link icon) and reciepient_bar_icon
(for other icons). CSS for spacing these classes correctly, using
padding-left and padding-right, has been added zulip.scss
Manually tested for cases with single and multiple links present.
Fixes#14364.
When we tried to copy/paste multiple rows up to
and including the last row in our view, we'd have
a blueslip error when the `for` loop checked the
condition `rows.id(row) <= ...` after we had
called `row = rows.next_visible(row)` on the last
row. Basically, `rows.id()` would complain
about a non-existent row.
Now we extract that code into `visible_range`, so
that our `while` loop can exit as soon as we found
the last row in the range.
This commit migrates zulip outging webhook payload to
/zulip-outgoing-webhook:post in OpenAPI.
Since this migrates the last payloads from api/fixtures.json to
OpenAPI, this commit removes api/fixtures.json file and the functions
accessing the file.
Tweaked by tabbott to further remove an unnecessary conditional.
The selector we were passing to `condense_and_collapse`
included rows from our drafts UI, which don't have
zids and don't play nice with condense/collapse code
(which expects message ids for settings things like
`.condense` flags).
Now we just use a better selector.
If folks use an overly broad selector for message rows,
they will accidentally include drafts from the drafts
dialog, which won't have zids. More specific selectors
will be more efficient and possibly prevent strange
behaviors.
For testing convenience, we extract the message.
This is critical for importing the very first realm into an empty
server, since in 27b15a9722, we changed
the model to create the internal realm when the first real realm would
be created, but neglected the data import code path.
We now handle the esc key completely within the
keydown handler that we already have for message
editing. We allow escape to work no matter what
the focused element is within an edited message,
and we blur that element properly and end the
edit.
We remove all the strange, duplicated logic
from hotkey.js.
This should also fix a blueslip error where the
hotkey code was passing message_edit a jQuery
element with zero length.
Fixes the traceback reported in #14151, though we should still look at
the DOM cleanup discussed there.
Add sgrep (sgrep.dev) to tooling and include simple rule as
proof of concept. Included rule detects use of old django render
function.
Also added a rule that looks for if-else statements where both
code paths are identical.
The distinction between ValueError and TypeError
is not useful in these functions:
- extract_stream_indicator
- extract_private_recipients (or its callees)
These are always invoked in views to validate
user input.
When we use REQ to wrap the validators, any
Exception gets turned into a JsonableError, so
the distinction is not important.
And if we don't use REQ to wrap the validators,
the errors aren't caught.
Now we just let these functions directly produce
the desired end result for both codepaths.
Also, we now flag the error strings for translation.
The UI in the `#settings/notifications` page is updated similarly
to what is done in the `update_global_notifications` path present
in the `server_events_dispatch` file.
We have an alert for when the stream name is changed.
This also adds an alert when subscription settings
are updated and the widget is similar to that used in
the settings page.
This is also necessary because the stream specific
notification settings UI updation goes through this
path and it is necessary to display a confirmation
to match with other settings confirmation pattern.
Since each element containing the `.alert-notification`
class has a predefined area that wont overlap with any
other element, we make changes to the CSS so that it
just stays hidden until the text appears.
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
While we could fix this issue by changing the markdown processor,
doing so is not a robust solution, because even a momentary bug in the
markdown processor could allow cached messages that do not follow our
security policy.
This change ensures that even if our markdown processor has bugs that
result in rendered content that does not properly follow our policy of
using rel="noopener noreferrer" on links, we'll still do something
reasonable.
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Zulip's modal_link markdown feature has not been used since 2017; it
was a hack used for a 2013-era tutorial feature and was never used
outside that use case.
Unfortunately, it's sloppy implementation was exposed in the markdown
processor for all users, not just the tutorial use case.
More importantly, it was buggy, in that it did not validate the link
using the standard validation approach used by our other code
interacting with links.
The right solution is simply to remove it.
This makes it relatively easy for a system administrator to
temporarily override these values after a desktop app security
release that they want to ensure all of their users take.
We're not putting this in settings, since we don't want to encourage
accidental long-term overrides of these important-to-security values.
If we can't find data on a mentioned user to update its full_name to
the current value, we'll have to go with the value in the message
itself.
This can happen if e.g. we hard-deleted the originally mentioned user
from the database (which can sometimes happen after a "delete my
account completely" request).
The user has an option for setting global
notification settings as well as the same settings
for individual streams. Currently the user has to
keep track of each unmatched stream and then visit
each individual stream whose settings he wants to
update.
Thus this adds a dedicated UI table allowing the user
to view and update the notifications of the specific
streams which differs from the global settings.
It is located on the same page where the user defined
global notification settings can be modified.
Fixes#9228.
Currently we are updating the checkbox UI as soon as the user clicks.
This block is removed to match with the pattern of rest of the
properties in the stream edit page where `stream_events.update_property`
is responsible for updating the UI after a successful server response.
This function returns a list of objects to create a
list_render object, and each item contains the streams
whose atleast one notification setting differs from the
default set by the user.
This is done by comparing the global settings in the
`#settings/notifications` page with those settings
present in the subscribed streams.
Work towards #9228.
This flag was used to delay unread count updates while the bankruptcy
modal was visible. Now that bankrupcty is no longer a modal, we don't
need this flag at all.
Switched to top-of-page prompt to make it natural to fit in with other
notifications. As we switch to panel-based prompt, templates for the
bankruptcy modal are moved along with its usage in application's
homepage.
We include a bit of delay before reloading to make it easy for the
user to read the "Marking all messages as read" banner before it is
covered by the "Reloading..." notice in environments where the reload
is fast.
Fixes#3347.
When stream_post_policy modal is closed either after saving or using
cancel button or cross button, the pointer-events is set to none which
does not allow to close the stream settings overlay on one click.
Added overlay.close_modal on saving such that pointer-events:none is
removed.
Added line which removes pointer-events:none again on clicking cancel
button or close icon.
This is a prep commit which extracts the part of the code in open_modal
and close_modal to separate methods which adds inline style of
pointer-events to enable/disable the background mouse events.
Block comments are added for easy understanding of reader.
Previously, we only printed the test-case when we had an assertion error.
With this change, we also include timeout errors as well as any other
causes for failure.
Before this we were monkey-patching in the
function `waitForSelectorText` into the
`casper` namespace, but only if you called
`common.initialize_casper`.
This would cause confusion if you expected
that function to be documented by Casper.
Now we just add the helper to `common` in
the `common` namespace.
We also avoid having to reason about what
`this` means by just using `casper` inside
the implementation of `wait_for_text` now.
And we don't bother with a return code that
none of our callers were using, anyway.
We removed the phantom_page_loaded logic in
b13265d135
(July 2017).
Now we just say that the page is loaded
to the console, which can possibly help
us debug glitches where the tests are
executing too early.
We added a really nice feature recently,
called `--interactive`, which lets you loop
through Casper tests without having to restart
it every time.
I am renaming it to `--loop` and adding a few
features:
- The first loop will just run without you having
to tell it to start. (This means you don't have
to sit there while waiting for webpack to finish
and for the server to start, just to launch
the tests again.)
- You specify how many loops you want to run,
which means in the success case, it won't
just keep going forever--it will eventually
stop, giving you an opportunity to refine
the test further without re-launching.
We now trim the headers inside of
`get_rendered_messages`, since any
sane caller of that function just
wants nicely trimmed headers.
(Note that we're now doing the
string manipulation inside of
Zulip code, not Casper code, which
is why I didn't reuse normalize_spaces.)
This commit removes "font-weight: 500;" from landing-page.scss so as
to fix a bug on landing pages that used the `markdown` class to
format content. The bug was caused by "a:hover" from landing-page.scss
overriding the font-weight (600) on links as set by the markdown
class, this caused the text to seem jumpy when one hovered over links.
Note from tabbott: The original code was added in
d7f5f31f6a, which doesn't explain it's
purpose, but it predates the more complete "markdown" CSS, was part of
an early prototype that had unfortunate hover behavior more generally,
and makes sense to remove.
Fixes: #14387.
We now have Hamlet, not Othello, send the message
to Othello's bot, since that's a more interesting
test and less likely to lead to a false positive.
And then we simplify the recipient check to avoid
the strange mypy mess as well as possible false
negatives.
When more than one outgoing webhook is configured,
the message which is send to the webhook bot passes
through finalize_payload function multiple times,
which mutated the message dict in a way that many keys
were lost from the dict obj.
This commit fixes that problem by having
`finalize_payload` return a shallow copy of the
incoming dict, instead of mutating it. We still
mutate dicts inside of `post_process_dicts`, though,
for performance reasons.
This was slightly modified by @showell to fix the
`test_both_codepaths` test that was added concurrently
to this work. (I used a slightly verbose style in the
tests to emphasize the transformation from `wide_dict`
to `narrow_dict`.)
I also removed a deepcopy call inside
`get_client_payload`, since we now no longer mutate
in `finalize_payload`.
Finally, I added some comments here and there.
For testing, I mostly protect against the root
cause of the bug happening again, by adding a line
to make sure that `sender_realm_id` does not get
wiped out from the "wide" dictionary.
A better test would exercise the actual code that
exposed the bug here by sending a message to a bot
with two or more services attached to it. I will
do that in a future commit.
Fixes#14384
If we have an old event that's missing the field
`sender_delivery_email`, we now patch it at the top
of `process_message_event`, rather than for each call
to `get_client_payload`. This will make an upcoming
commit a bit easier to reason about. Basically, it's
simpler to shim the incoming event one time rather
than doing it up to four times. We know that
`get_client_payload` is non-destructive, because it
does a deepcopy.
We now validate the message data explicitly, rather
than comparing it to the event data. This protects
us from false positives where we were only validating
that the request data was a mutated version of the
event message data. (We'll have a commit soon that
fixes a mutation-related bug.)
This code is only used in one test, and having
the indirection of setUp partly obscured a
problem with the fact that our event message
is actually a wide dict that gets mutated
by `build_bot_request`. We'll fix that soon,
but this is a pure code move for now.
The `event` parameter is never used by `process_success`,
and eliminating it allows us to greatly simplify tests
that are just confusingly passing in events that are
totally ignored.
Folks can have issues connecting to Casper
as zulipdev.com when they are not connected to
the internet or just have a bad connection, since
the DNS record is on the internet. Folks can
work around this by just creating an /etc/hosts
entry for zulipdev.com, but people don't always
know.
This fix moves the symptom slightly earlier in
the process--we don't advertise that the server
is "up" if you can't actually connect to it as
"zulipdev.com".
In zulip.yaml simple json success response which only contains 'msg'
and 'result' properties has been described repeatedly in multiple
endpoints. Instead, use SimpleSuccess template for such responses
to increase code modularity and reusablility.
If a non-author user clicked on view source in a poll and then close it,
the edit question icon would incorrectly get visible. This made changing
the question in local echo possible for non-author users.
Fixes: #14299
Migrate "call_on_each_event" from api/arguments.json to
/events:real-time in OpenAPI.
This is a bit of a hack, but it lets us eliminate this secondary
arguments.json file, which is probably worth it.
Tweaked by tabbott to fix various formatting issues in the original
documentation while I was looking at it.
Most part of "/message/{message_id}" is migrated to OpenAPI. This commit
migrated the remaning payload "update-message-edit-permission-error"
from "api/fixtures.json" to OpenAPI. This commit also fixes an error
schema in "zulip.yaml" for this payload.
We've had a bug for a while that if any ScheduledEmail objects get
created with the wrong email sender address, even after the sysadmin
corrects the problem, they'll still get errors because of the objects
stored with the wrong format.
We solve this by using FromAddress placeholders strings in
send_future_email function, so that ScheduledEmail objects end up
setting the final `from_address` value when mail is actually sent
using the setting in effect at that time.
Fixes#11008.
This refactors get_single_user to use get_user_by_id instead of
call_endpoint. Doing so is only possible now that we've upgraded
python-zulip-api to a version with the new function.
Overall, this change eliminates a lot of
optional parameters and conditionals, plus
some legacy logic related to caches.
For all the places we are just editing topics,
we now just call `check_topic` to see that
the topic got updated.
For places where the topic edit failed, we
just inline the checks that message still
has the old topic and content.
And then for successful **content** edits,
we now do a more rigorous, more sane check
that the messages are properly cached. The
old code here had evolved from 2013 into
something that didn't really make much sense
in the context of editing topics.
Now we are literally pulling data from the
cache and making sure it's valid, rather
than trying to poorly simulate the two
codepaths related to dispatching message
events and fetching messages. Some of the
history here was that when I introduced
`MessageDict` several years ago, I did a
lot of code sweeping and didn't analyze every
single test to make sure it's still valid,
plus some of the tests still had some value
for catching regressions. A recent commit
now gets us coverage on that a lot more
explicitly, rather than in passing.
See the comment in the test for a thorough explanation.
In brief, this test makes sure that the events codepath
for messages produces the same results as the fetch
codepath.
And this sets us up to simplify another test that kind
of poorly tried to do the same thing in passing. (In
fairness the test was really ancient and preceded a lot
of later work that we did here.)
When we are fetching messages, we need to hydrate
stream names into the messages for legacy reasons.
(Ideally, we could skip this step for the webapp
and modern mobile clients, since they really only
need stream_ids, but we're not there yet.)
We keep a recipient cache that maps recipient ids
to stream names.
When we populate that cache, we now use `values(...)`
to avoid fat objects and extra DB work.
Note that we are already using a similar technique
for hydrating PM/huddle recipients.
After some testing, I've confirmed that this seems to behave
significantly better in terms of the number of failed requests due to
Tornado being the process of restarting compared with the previous
version, as each individual process is only down for a short time,
rather than all of them being down at once.
For event types that we don't yet support, like worklog_created (and
likely many more in the future), it doesn't make sense to call a
function that only parses issue events correctly.
Apparently, we didn't have any database indexes on Confirmation, which
meant that on servers with large numbers of users like zulipchat.com,
new account registration could spend a ton of time effectively doing a
table scan on this table.
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).
Fixes#14350.
Starred messages from muted topics were not shown in the starred
messages view. Condition for muting_enabled is modified accordingly
such that the starred messages from muted topics is shown in the
starred messages narrowed view.
Node tests are updated accordingly.
Fixes#13548
The previous system for documenting arguments was very ugly if any of
the examples or descriptions were wrong. After thinking about this
for a while, I concluded the core problem was that a table was the
wrong design element to use for API parameters, and we'd be much
better off with individual card-type widgets instead.
This rewrites the API arguments documentation implementation to use a
basic sort of card-like system with some basic styling; I think the
result is a lot more readable, and it's a lot more clear how we would
add additional OpenAPI details (like parameter types) to the
documentation.
In the past it has blocked Python library security updates with overly
strict version bounds, and we don’t use it as a library, only as a
binary.
Skip the PROVISION_VERSION bump because we can use the tx binary from
either location.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This reverts commit 36a8e61e67 (#13934).
The Django 2.2 autoreloader works by forking into a child process that
exits with status 3 when a file changes, and a parent process that
restarts the child when it exits with status 3. Setting this
environment variable had the effect of pretending we were already the
child process, without a parent process to restart it. Therefore,
changing any code used by the queue processor caused it to exit rather
than restart.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I verified there are no other endpoints we've added since Zulip 2.1,
so at least this part of our documentation is accurate.
We added the `/realm/export` endpoints in Zulip 2.1, but those don't
have documentation yet, so we'll just need to remember to cover this
when they are added.
Automatic inclusion is for files with global declarations, as opposed
to files declared as modules. typeRoots already had the side effect
of disabling the default automatic inclusion for node_modules. Since
we also don’t need it for static/js/types, we may as well disable it
completely.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The previous logic avoided updating the setting for
non-administrators, because their value was always true, but removing
those if statements results in better test coverage and is more likely
correct if we ever try to support live-update for whether the user is
an administrator.
Comment out any Focal jobs in circleci config file.
This is due to a recent bug that is causing some trouble
while installing python-pip in Focal.
So we are pausing Focal tests until we get stability.
We've noticed that many production organizations don't set either an
organization description or profile picture, even large open source
organizations that could definitely take advantage of this feature.
This adds a top-of-page banner that bugs organization administrators
to add an organization description and profile picture, generally
starting on the second login (as we only do it on page load after
notifications are configured).
Significantly tweaked by tabbott to get the right user experience.
Fixes#14019.
The original implementation of panels.js was just for notifications,
and ended up running a bunch of notifications-specific code, including
registration click handlers and some localstorage-related
notifications logic, every time a panel was supposed to be opened.
This refactoring makes the panels library make sense -- we now
initialize all click handlers in the initialize() method, and do the
notifications check in a single, coherent place scoped to notifications.
We now use `wait_for_message_fully_processed`
to check that messages are fully rendered.
Before this, we had loopholes where messages
sent outside the view were effectively ignored.
Now we explicitly ignore the check for the
one place we do that.
The more important behavior is for messages
that get sent to the current view.
Before this change, the older version of this
function declared victory as soon as we put the
server version of a locally echoed message into
the current message list's data.
This fixes flaky behavior with 07-stars in
particular, since we need the star icon
on our last message to be there before
we click on it.
Because this function is more robust now, we
can remove some redundant checks in 08-edit.js.
Commit 68335d9124 removed the ability to tab
into this field, since it was a hidden field. This field is no longer
hidden, and this commit restores the ability to tab into it.
I think we could write this test better, but it's not a big deal for
this to break in the rare even that we change/remove one of the 2
strings it interacts with.
In continuation to #13250
CHANGES:
-the stream name edit button is now visible for long names too.
-ellipsis are removed when you click on edit name option.
-added border while editing name to give a text-box feel.
REASONS:
-added border while editing the name to give a textbox-esque feel.
-text overflow was changed from ellipsis to clip (while editing) as
ellipsis prevented editing the entire name (clip provides better
functionality).
The last two changes are reverted back to original (i.e. ellipsis and
no border) once you finish editing the stream name.
P.S.- clicking on anywhere else updates the new name perfectly
Here we have migrated checkboxes of all general notifications to the table.
By general notifications we mean, Mobile, Email, Desktop audio, and visual
notifications.
This is a part of a bigger migration to simply our notifications setting
changing infrastructure for all streams and individual streams. Later we
will add more row to this for different categories of notifications in
addition to the current ones ("Streams" and "PMs, mentions, alerts").
Fixes: #12182.
Missing commas in the definition of all the queues to check meant that it would be looking for queues with concatenated names, rather than the correct ones. Added the commas.
When you select a typeahead, it shouldn't
immediately do the action for you; you should
have to hit enter first. Even though 99% of
the time you're gonna confirm the typeahead,
it's jarring when you don't expect it.
You can still add a bunch of default streams
quickly, using only the keyboard, because
we have always had support for the enter
key saving. (and tab and enter also works)
This is a full-stack change:
- server
- JS code
- templates
It's all pretty simple--just use stream_id instead
of stream_name.
I am 99% sure we don't document this API nor use it
in mobile, so it should be a safe change.
We now only use `page_params.realm_default_streams` during
initialization, and then after that we use `stream_data`
APIs to get default stream ids and related info. (And
for the event that replace the data, we just update our
internal data structures as well.)
Long term we should have the server just send us ids here,
since we are now hydrating info from stream data in all places.
This code is a bit simpler.
The previous code was concatenating two lists
and then removing duplicates by calling filter().
Now we just have two loops that append to a single
list, and the second loop detects duplicates
before inserting into the list.
We also now use `default_stream_ids` instead of
`page_params` data, which is convenient for two
reasons:
- working with sets of ids is convenient
- we don't need to maintain `page_params`
data any more
We now use the up-to-date info from stream_data
to hydrate the default stream ids. All we need
here in the template is `invite_only` and `name`.
Since we are no longer using data from `page_params`,
we can remove `maybe_update_realm_default_stream_name`.
(If you are wondering if we still get live updates,
we get that via a more upstream call to
update_default_streams_table in the event
dispatching codepath.)
We only used get_default_stream_names() in a
test, so now it's being replaced with a function
that just gets ids.
We'll have use for get_default_streams_ids()
in an upcoming commit.
Now if a default stream gets deleted, we just
redraw the table. We always have a small number
of default streams, and the way that we were removing
rows without the actual consent of `list_render` was
really janky (and just a vestige of pre-list-render
code that never got fully ported).
This also makes us consistent with how we handle
added streams (i.e. just call
`update_default_streams_table`).
ASIDE:
Ideally we will update `list_render` at some point to
have an API for adding and removing elements. It does
allow you now to call `data()` to reset its data, but
for now we just build a new `list_render` object every
time.
This commit modifies 'zerver/lib/bot_lib.py' to decouple the
user-controllable 'service_name' parameter from the value that is
passed in to 'import_module'. This is done as a precautionary
hardening.
This commit introduces two new functions in 'url_encoding.py' which
centralize two common patterns for constructing redirect URLs. It
also migrates the files using those patterns to use the new
functions.
After subscribing a stream email address to a Mailman email list
and receiving a message from it (using the polling configuration
with an Exim + Dovecot mailserver), the following error message
is emitted by Zulip:
Logger zerver.lib.email_mirror, from module zerver.lib.email_mirror line 77:
Error generated by Anonymous user (not logged in) on zulip deployment
Sender: "Foo Bar" <foo@example.com>
To: No recipient found
Missing recipient in mirror email
This is because the To: header on the received email corresponds
to the email list, and there are no other headers to indicate the
final recipient, apart from the "Envelope-To" header added by
Exim. To resolve this problem, the commit adds "Envelope-To" to
the list of headers to check for a match.
python-dev will be depreciated in Focal but can be used as python2-dev
so removed it from common dockerfile.template and added it
as an extra package in .circleci/config.yml.
python 3.8 support for python-api-bindings was fixed in commit
63bc9b8a4f
so upgraded python-api-bindings to tag 0.6.3 which included this fix.
Bumped PROVISION_VERSION.
The function `prepare_login_url_and_headers` returns a register
link for any value of `is_signup` unless it's not none.
This commit changes it to a boolean for that function and other
functions using it so that it becomes much clearer when a
register link will be returned.
Also, all occurrences of `is_signup='1'` are changed to
`is_signup=True` to make the code consistent with the above change.
This will otherwise throw error when the billing page is accessed
after a plan is expired. A proper followup would be to not show
licenses, licenses_used etc if the current plan is on Limited.
Commit 03393631bd (#14142) regressed the
keyboard accessibility of the keyboard shortcuts modal. Fix it by
moving tabindex="0" to the scrolling element of the SimpleBar.
Fixes#14320.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We stopped needing this with
0329b67048
(Dec 2016).
The function sets `bot.can_admin`,
which was only used in `bot_data.get_editable`.
We removed two tests (and then put back
some test setup that needed to leak down
to the last test).
This is code simplification motivated
by a recent bug that we fixed with some
server changes, but which was really
caused in some sense by our client code
using an overly finicky
condition to check falsiness.
For cross-realm bots, the value of
`user.bot_owner_id` may be `null`, or it
may simply be `undefined`, depending
on whether the server passes `None`
or simply omits the field.
We don't want out client code to be
coupled to that rather arbitrary
decision.
We were doing a `!== null` check instead
of checking for falsiness, which led to
blueslip errors in the past. Because a
bot owner id could be plausibly 0, a falsiness
check would be brittle in a different way.
Now we avoid that ugliness by calling
`get_bot_owner_user`, which either returns
an object or `undefined`.
And then the caller can just do a concise
check for whether `bot_owner` exists.
And we also fix up the crufty code that
was putting `bot_owner_full_name` on to
the object instead of using a local.
We have a bug report for this again, although
it might be on an old branch.
Fixes#13621.
This allows us to block use of the desktop app with insecure versions
(we simply fail to load the Zulip webapp at all, instead rendering an
error page).
For now we block only versions that are known to be both insecure and
not auto-updating, but we can easily adjust these parameters in the
future.
We know that by default the value of `enable_stream_push_notifications` is
true, so on some server if push notifications are disabled then due to
```
{{#unless show_push_notifications_tooltip}}
```
the `enable_stream_push_notifications` checkbox is rendered as unchecked
and when we update any other setting in the "Stream notifications" section,
due to `settings_notifications.update_page()` all checkboxes are changed to
current values which means above checkbox get checked.
We can deal with this bug either making changes in
`settings_notifications.update_page()` but we should show check/uncheck
state of checkboxes anyway so removing this `unless` condition fixes this
bug.
Instead of having logical expressions in templates, it's always preferred
to calculating them in javascript and pass the results as a context. It
also enhances the readability of templates and testing of such logic is
easier in js over templates.
`all_notification_settings_labels` is misleading that this variable is a
list of notifications setting labels so changed it to
`all_notification_settings`.
The reason for extracting this function is that getting the text, integer,
boolean value from the input elements (like checkboxes, dropdowns) is a
common task, and later we can use this function to get the input element
value in `settings_notifications` in the upcoming commit.
This is a bug fix where, if a list_render
object with the given name exists and it's items
have been sorted, then the filtered_list's data
does not get updated on re-rendering.
This line was present in the original commit
9576d5caef.
The use case for this are small or fixed tables, which do not need
filtering support. Thus we are able to not include the unnecessary
search input inside the html parent container.
It is not used at present, but will be required when we refactor
the settings pages.
We also split out exports.validate_filter function for
unit testing the above condition.
This improves the error handling for invalid values of the
propagate_mode parameter to our message editing endpoints.
Previously, invalid values would just work like change_one rather than
doing nothing.
As a consequence of too many options in the bottom `Other permissions`
subsection, the `Save` button could end up too far up from the bottom,
such that it might appear offscreen on low-height laptops.
We fix this by reorganizing the settings in a way that is both more
intuitive and also ensures that none of the subsections are too tall.
Fixes: #14274.
setup_event_queue() generates some logs about loaded event queues, and
it's good for the logging system to have access to the port at that
point already.
In development env, we use `get_secret` to get
`SOCIAL_AUTH_GITLAB_KEY` from `dev-secrets.conf`. But in
production env, we don't need this as we ask the user
to set that value in `prod_settings_template.py`.
This restricts the code from looking `zulip-secrets.conf`
for `social_auth_gitlab_key` in production env.
Before this commit, the reactions code would
take the `message.reactions` structure from
the server and try to "collapse" all the reactions
for the same users into the same reactions,
but with each reaction having a list of user_ids.
It was a strangely denormalized structure that
was awkward to work with, and it made it really
hard to reason about whether the data was in
the original structure that the server sent or
the modified structure.
Now we use a cleaner, normalized Map to keep
each reaction (i.e. one per emoji), and we
write that to `message.clean_reactions`.
The `clean_reactions` structure is now the
authoritatize source for all reaction-related
operations. As soon as you try to do anything
with reactions, we build the `clean_reactions`
data on the fly from the server data.
In particular, when we process events, we just
directly manipulate the `clean_reactions` data,
which is much easier to work with, since it's
a Map and doesn't duplicate any data.
This rewrite should avoid some obscure bugs.
I use `r` as shorthand for the clean reaction
structures, so as not to confuse it with
data from the server's message.reactions.
It also avoids some confusion where we use
`reaction` as a var name for the reaction
elements.
Previously, we only did apt updates when our sources.list files or
keys changed, which could result in provisioning errors for
development systems that don't routinely update their apt cache
(probably including ~all Vagrant environments).
importlib-metadata and importlib-resources are dependent packages for jsonschema
and cfn-lint respectively. They are built-in modules in later versions
of python (3.8, 3.7). When update-locked-requirements is run within python3.7 or
3.8 they will generate difference in locked files so we build these modules separately
to avoid such conflicts.
transifex-client0.13.4 did not support python3.8 so updated
it to the latest version. Earlier we kept transifex-client version to
0.13.4 as transifex-client0.13.5 added overly strict version bounds
on six and urllib3. With the latest version this is not the case.
Bumped provision version.
cgi.escape is deprecated in python3.2 and removed in python3.8.
This function was unsafe because quote is false by default, hence
removed and replaced with a safer html.escape.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
In virtualenv ≥ 20, the site_packages variable was removed from
activate_this.py. To avoid a KeyError, replace
activate_locals['site_packages'] with os.path.join(venv, 'lib',
python_version), where python_version is the 'pythonX.Y' name of the
directory where site-packages resides in the virtualenv.
Fixes#14025.
I'm not sure what causes some Jira webhook events to not include the
metadata that other events do, but it's definitely a format sent by
real installations of Jira (likely a very old version, since this has
fields missing from what modern Jira does) and we've seen it in
production.
The best we can do is encourage users to upgrade Jira for better data.
The previous starred_messages race handling did not correctly consider
the possibility that an event queue might have been registered without
starred_messages.
Instead of operating on RateLimitedObjects, and making the classes
depend on each too strongly. This also allows getting rid of get_keys()
function from RateLimitedObject, which was a redis rate limiter
implementation detail. RateLimitedObject should only define their own
key() function and the logic forming various necessary redis keys from
them should be in RedisRateLimiterBackend.
type().__name__ is sufficient, and much readable than type(), so it's
better to use the former for keys.
We also make the classes consistent in forming the keys in the format
type(self).__name__:identifier and adjust logger.warning and statsd to
take advantage of that and simply log the key().
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
When a user in login flow using github auth chooses a email that is
not associated with an existing account, it leads to a "continue to
registration" choice. This cannot be tested with the earlier version
of `stage_two_of_registration`.
Also added the test.
Thanks to Mateusz Mandera for the solution.
Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
The previous model for GitHub authentication was as follows:
* If the user has only one verified email address, we'll generally just log them in to that account
* If the user has multiple verified email addresses, we will always
prompt them to pick which one to use, with the one registered as
"primary" in GitHub listed at the top.
This change fixes the situation for users going through a "login" flow
(not registration) where exactly one of the emails has an account in
the Zulip oragnization -- they should just be logged in.
Fixes part of #12638.
URLs for config errors were configured seperately for each error
which is better handled by having error name as argument in URL.
A new view `config_error_view` is added containing context for
each error that returns `config_error` page with the relevant
context.
Also fixed tests and some views in `auth.py` to be consistent with
changes.
We've often gotten the complaint that Zulip's emoji are a bit too big;
this should address the worst consequences of that (line-wrapping
being off with large emoji present) while still making it possible to
easily see what a given emoji is.
The right place to change this is in rendered_markdown.scss, not the
main emoji definition in zulip.scss, as the latter is also used in
places like the emoji picker where a larger size is valuable.
Closes#12731, an older PR that did this with slightly different
parameters (and without a comment).
Saying `foo.lstrip('# ')` does more than just remove
a '# ' prefix. It removes any combination of '#' and
spaces.
We now make the intention slightly more clear.
We would strip these as you'd expect:
# foo
## foo
### foo
but for this we now only strip the first "#":
# # # # # foo
Thanks to @minusworld for catching this--see #14264, which
points out that lstrip() doesn't do what your intuition
might tell you it does.
Now we properly remove the "HTTP_" prefix.
It's not clear to me why we need these prefixes for Django
purposes in the fixtures, but I didn't want to go down
the rabbit hole of fixing those.
To test:
got to http://YOUR-DEV_SERVER/devtools/integrations/
select "bitbucket3" for the integration.
select "diagnostics_ping.json" for the fixture.
see "X_EVENT_KEY" in "Custom HTTP Headers"
Fixes#14264
Fixes#14254
You can test this on dev:
* do "-stream:Verona" in the search bar (the minus
sign negates the search here)
* reload the browser
You should see the same search (all streams besides Verona).
We simplify the code for deciding whether
we show a subscribe button or not, and in
doing so avoid a blueslip error where we
were passing `undefined` into `get_sub()`.
Apparently, this test was not allowing the browser to run between the
keypress to start edit and checking to see if message_edit_content appeared.
I'm not sure if this is what has been causing recent flakes, but it
was definitely wrong Casper code.
We had this API:
people.add_in_realm = full-fledged user
people.add = not necessarily in realm
Now the API is this:
people.add = full-fledged user
people._add_user = internal API for cross-realm bots
and deactivated users
I think in most of our tests the distinction between
people.add() and people.add_in_realm() was just an
accident of history and didn't reflect any real intention.
And if I had to guess the intention in 99% of the cases,
folks probably thought they were just creating ordinary,
active users in the current realm.
In places where the distinction was obviously important
(because a test failed), I deactivated the user via
`people.deactivate`.
For the 'basics' test in the people test suite, I clean
up the test setup for Isaac. Before this commit I was
adding him first as a non-realm user then as a full-fledged
user, but this was contrived and confusing, and we
didn't really need it for test coverage purposes.
We want to move more logic to stream_data to facilitate
testing.
Both before and after this commit, we essentially build a
new list of users for typeahead, but now the new list
excludes subscribed users. We can do even better than
this in a follow-up commit.
Before this commit, presence used get_realm_count()
to determine whether a realm was "small" (and thus
should show all human users in the buddy list, even
humans that had not been active in a while).
The `get_realm_count` function--despite a very wrong,
misleading comment--was including bots in its count.
The new function truly counts only active humans
(and no bots).
Because we were overcounting users before this change,
we should technically adjust `BIG_REALM_COUNT` down
by some amount to reflect our original intention there
on the parameter. I'm leaving it alone for now, though,
since we've improved the performance of the buddy list
over time, and it's probably fine if a few "big" realms
get re-classified as small realms (and show more users)
by virtue of this change.
(Also note that this cutoff value only affects the
"normal" view of the buddy list; both small realms
and large realms will show long-inactive users if you
do searches.)
Fixes#14215
This is a prep commit for the new navbar, since the new navbar switches
between a search bar and stream descriptions, it's easier to have the
border defined in an outer div. Due to the way the changesets is
generated, this may seem like a large diff, however, the only change to
navbar.html is to add an opening div with the ".top-navbar-border" class
and a corrseponding closing div to wrap around "#search_box" and
"#search_box_legacy". Apart from this, a few styles have been edited in
zulip.scss and night_mode.scss.
This is a prep-commit for the new navbar style, seperating
navbar_alerts.html from navbar.html in order to make the structure and
styles of navbar.html easier to tweak.
These templates have very little to do with each other to begin with
appart from the fact that they are both rendered at the top of the app.
This is a bit more rigorous than just
dereferencing the first element of
a list comprehension, as it will give a
ValueError if more matches are found than
the test was expecting.
We don't need `do_create_user` to send a partial
event here for bots. The only caller to `do_create_user`
that actually creates bots (apart from some tests that
just need data setup) is `add_bot_backend`, which
sends the more complete event including bot "extras"
like service info.
The modified event tests show the simplification
here (2 events instead of 3).
Also, the bot tests now use tuple unpacking, which
will force a ValueError if we duplicate events
again.
We now restrict emails on the zulip realm, and now
`email` and `delivery_email` will be different for
users.
This change should make it more likely to catch
errors where we leak delivery emails or use the
wrong field for lookups.
We were going back to the database to get all
the users in the realm, when we had them right
there already. I believe this is a legacy
of us running on a very old version of Django
(back in early days), where `bulk_create`
didn't give you back ids in a nice way.
In the interim we added the `RealmAuditLog`
code, which does take advantage of the
existing profiles (and proves we can rely
on them).
But meanwhile we were still
doing a query to get all N users in the
realm. With `selected_related`!
To be fair, bulk_create_users() is by
its very nature a pretty infrequent
operation. This change is more motivated
by code cleanup.
Now we just loop through user_ids for
the Recipient/Subscriber foreign key rows.
I also removed some fairly convoluted code mapping
emails to user_ids and just work in user_id
space.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.
(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)
In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.
A couple random things:
- I fixed any error messages that were leaking
the wrong email
- a test that claimed to rely on the order
of emails no longer does (we sort user_ids
instead)
- we now use user_ids in some place where we used
to use emails
- for IRC mirrors I just punted and used
`reset_emails_in_zulip_realm` in most places
- for MIT-related tests, I didn't fix email
vs. delivery_email unless it was obvious
I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug. (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)
In particular, I just reset data for the analytics
and corporate tests.
We specifically give the existing user different
delivery_email and email addresses, to prevent false
positives during the test that checks that users
signing up with an already-existing email get
an error message.
(We also rename the test.)
The matrix integration folder in python-zulip-api changed its name
from matrix to bridge_with_matrix but this was not reflected in
the docs. Change the docs accordingly.
Solves issue #545 in the python-zulip-api directory.
The simplebar is the default scrollbar throughout majority of Zulip but
it was missing in "Keyboard shortcuts", "Message formatting" and "Search
operators" modals. Added simplebar in the 3 modals.
I guess `test_classes` has 100% line coverage
enforcement, which is a bit tricky for error
handling.
This fixes that, as well as making the name
snake_case and improving the format of the
errors.
This test was using the anti-pattern of doing an
assertion inside a conditional.
I added the `findOne` helper to make it easier
to write robust tests for scenarios like this.
Tests for these links often result in rate-limiting from GitHub,
leading to the builds failing in Circle CI. We temporarily mark
github.com/zulip links as external to keep the builds passing.
Those docs were outdated and no longer represantative of how things
work. upgrade-python-dependencies and unupgradable.json are no longer a
thing, so the entire paragraph about them should be removed.
Then the requirements/README.md file is refreshed a bit to be more
accurate.
We had a bunch of ugly hacks to monkey patch things due to upstream
being temporarily unmaintained and not merging PRs. Now the project is
active again and the fixes have been merged and included in the latest
version - so we clean up all that code.
Punctuate marketing headings with a period. Fix a couple of
title-cased headings to sentense case. Consistently use curly
apostrophes, curly quotation marks, and Unicode ellipses.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
If I send a message from a normal Zulip client, it is
considered to be "read" by me. But if I send it via
an API program (using my human account), the message
is not immediately "read" by me.
Now we handle this correctly in `get_raw_unread_data`.
The symptom of this was that these messages would get
"stuck" in "Private Messages" narrows until the next
time you reloaded your app.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
There were two problems with the previous code-
1) The code glob.glob("scripts/lib/build-") should be
glob.glob("scripts/lib/build-*) otherwise it would always return [].
2) The part of the code where we included scripts/lib/build-* for sha1 sum
check would only run when debian is not in os_families(). This wasn't
correct as we could have a situation where we have to build pgroonga
from source even in case of debian and so we need to improve the
condition on it.
Now since we only have build-pgroonga there its better to just directly hash
its content with the condition of BUILD_PGROONGA_FROM_SOURCE.
I've always thought of distributed teams as the place where Zulip
really shines over other tools, because chat is much more important in
that context.
And I've always been kinda unhappy with "most productive team chat" as
a line.
There's a lot more we should do here, but this is a start.
This should fix spurious failures, where test-run-dev would occasionally
freeze. What exactly about these changes was causing that is still to
be potentially investigated. This is merely meant as a fix to the
failures.
This reverts commit 19429c3ad7.
Given that can_mark_messages_read is called whenever the blue box
cursor stops on a message and that it is calculated purely on the
basis of sorted_term_types, it makes sense to cache the result.
Previously, when list_render.create was called, if a list_render
object with the given name existed, it returned the existing
list_render object with the previous properties, without the property
to sort the lists added. The root cause of the bug was that when we
added the sorting click handlers, we put them just in the constructor,
not in __set_events, the function we call from appropriate code paths
to add the other necessary click handlers.
Fix this by moving the code to add the sorting properties into
__set_events().
Fixes#14175.
Using an Exists subquery to avoid scanning the entire Subscription
table seems to speed things up greatly.
Set up with:
./manage.py populate_db --extra_users 2000 --extra-streams 1000
Tested on my computer, the original function was taking ~1.2seconds,
the optimized version only ~0.05-0.06.
Likely fixes#13874; we can re-open if after production testing we
feel more work is warranted.
If you were in the "Starred messages" narrow and
your pointer was on a message with the stream/topic
of "social/lunch", we wouldn't move you to the unread
messages for that topic.
I fixed this by removing the code that looked at
the current message's topic. Instead, we only look
at the active narrow to figure out the "next" topic
to go to.
Fixes#14120.
This ensures that even if it were possible to create an MIT Kerberos
account with a malicious username and/or hack webathena to pretend
that's the case, one couldn't do anything malicious.
This security improvement only impacts a single installation of Zulip
where Zephyr mirroring is in use that has already had the fix applied,
so there's no reason to do a security notice for it.
Found by Graham Bleaney using pysa.
In 220c2a5ff3 I
introduced a query to find invites by delivery_email
but was still using email as the key.
For most realms `email` and `delivery_email` are
synonymous, so this temporary bug would not affect
them. For realms that restrict emails, the invite
would have probably failed for other reasons, but
the symptom would have been less clear.
The user can pass description along with the task name by splitting the input string with hyphen.
Eg: Task Title - Task Description
todo_list: Add index numbers to task.
We now have this API...
If you really just need to log in
and not do anything with the actual
user:
self.login('hamlet')
If you're gonna use the user in the
rest of the test:
hamlet = self.example_user('hamlet')
self.login_user(hamlet)
If you are specifically testing
email/password logins (used only in 4 places):
self.login_by_email(email, password)
And for failures uses this (used twice):
self.assert_login_failure(email)
This reduces query counts in some cases, since
we no longer need to look up the user again. In
particular, it reduces some noise when we
count queries for O(N)-related tests.
The query count is usually reduced by 2 per
API call. We no longer need to look up Realm
and UserProfile. In most cases we are saving
these lookups for the whole tests, since we
usually already have the `user` objects for
other reasons. In a few places we are simply
moving where that query happens within the
test.
In some places I shorten names like `test_user`
or `user_profile` to just be `user`.
We want a clean codepath for the vast majority
of cases of using api_get/api_post, which now
uses email and which we'll soon convert to
accepting `user` as a parameter.
These apis that take two different types of
values for the same parameter make sweeps
like this kinda painful, and they're pretty
easy to avoid by extracting helpers to do
the actual common tasks. So, for example,
here I still keep a common method to
actually encode the credentials (since
the whole encode/decode business is an
annoying detail that you don't want to fix
in two places):
def encode_credentials(self, identifier: str, api_key: str) -> str:
"""
identifier: Can be an email or a remote server uuid.
"""
credentials = "%s:%s" % (identifier, api_key)
return 'Basic ' + base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
But then the rest of the code has two separate
codepaths.
And for the uuid functions, we no longer have
crufty references to realm. (In fairness, realm
will also go away when we introduce users.)
For the `is_remote_server` helper, I just inlined
it, since it's now only needed in one place, and the
name didn't make total sense anyway, plus it wasn't
a super robust check. In context, it's easier
just to use a comment now to say what we're doing:
# If `role` doesn't look like an email, it might be a uuid.
if settings.ZILENCER_ENABLED and role is not None and '@' not in role:
# do stuff
Instead of trying to set the _requestor_for_logs attribute in all the
relevant places, we try to use request.user when possible (that will be
when it's a UserProfile or RemoteZulipServer as of now). In other
places, we set _requestor_for_logs to avoid manually editing the
request.user attribute, as it should mostly be left for Django to manage
it.
In places where we remove the "request._requestor_for_logs = ..." line,
it is clearly implied by the previous code (or the current surrounding
code) that request.user is of the correct type.
This refactors remove_reaction in python_examples.py to validate the
result with validate_against_openapi_schema. Minor changes and some
additions have been made to the OpenAPI format data for
/messages/{message_id}/reactions endpoint.
This refactors add_reaction in python_examples.py to use the
openapi_test_function decorator and validate result with
validate_against_openapi_schema. Minor changes have been made to the
OpenAPI format data for /messages/{message_id}/reactions endpoint.
This also adds add-emoji.md to templates/zerver/api and adds
add-emoji to rest-endpoints.md (templates/zerver/help/include).
This refactors get_members_backend to return user data of a single
user in the form of a dictionary (earlier being a list with a single
dictionary).
This also refactors it to return the data with an appropriate key
(inside a dictionary), "user" or "members", according to the type of
data being returned.
Tweaked by tabbott to use somewhat less opaque code and simple OpenAPI
descriptions.
Previously, get_client_name was responsible for both parsing the
User-Agent data as well as handling the override behavior that we want
to use "website" rather than "Mozilla" as the key for the Client object.
Now, it's just responsible for User-Agent, and the override behavior
is entirely within process_client (the function concerned with Client
objects).
This has the side effect of changing what `Client` object we'll use
for HTTP requests to /json/ endpoints that set the `client` attribute.
I think that's in line with our intent -- we only have a use case for
API clients overriding the User-Agent parsing (that feature is a
workaround for situations where the third party may not control HTTP
headers but does control the HTTP request payload).
This loses test coverage on the `request.GET['client']` code path; I
disable that for now since we don't have a real use for that behavior.
(We may want to change that logic to have Client recognize individual
browsers; doing so requires first using a better User-Agent parsing
library).
Part of #14067.
The "sender" property in `send_message_backend` is meant to only do
something when doing Zephyr mirroring (or similar). We should help
clients behave correctly by banning this property in requests that are
not specifically requesting mirroring behavior.
This commit requires changes to a number of tests that incorrectly
passed this parameter or didn't use the right setup for mirroring.
The special Zephyr mirroring logic is only intended to be used via the
API, so this sets up a more effective test. It also allows us to
remove certain Client parsing logic for the /json/ views using session
authentication.
Original email address is shown to admin users in subscriber list when
email_address_visibilty is set to "Admins only" by passing delivery_email
at required places. Email address are not shown to non-admin users when
visibility is set to "Admins only".
Tweaked by tabbott to fix a few bugs and dead code.
Fixes a part of #13541.
User IDs are more robust than email addresses as they don't change
with time, and also don't have complications with
different email_address_visibility settings.
The email domain restriction to @zulip.com is annoying in development
environment when trying to test sign up. For consistency, it's best to
have tests use the same default, and the tests that require domain
restriction can be adjusted to set that configuration up for themselves
explicitly.
This will hide all the download buttons on the initial render of the `/apps`
page. It'll hide them until the JavaScript is loaded and calls `update_page`
method to render appropriate button.
We are not using JS to hide the buttons as it still will result in displaying the
buttons and not hide them until JS kicks in. Optimal solution is to set them as
hidden HTML elements and let jQuery override it's display attribute later.
Fixes#14134.
This uses the better, modern, user ID based API for sending messages
internally in the test suite, something that's convenient to do as a
follow-up to the migration to pass UserProfile objects to these
functions.
This commit mostly makes our tests less
noisy, since emails are no longer an important
detail of sending messages (they're not even
really used in the API).
It also sets us up to have more scrutiny
on delivery_email/email in the future
for things that actually matter. (This is
a prep commit for something along those
lines, kind of hard to explain the full
plan.)
This is a common UX pattern for forms - a user would expect the
input to be submitted on hitting enter.
So, create a 'keypress' event listener on the input field for the
new status, which calls 'submit_new_status' on enter key press.
This intent is that we'll be able to reuse this when editing streams
as well.
* Rename method: filter_with_new_topic to filter_with_new_param.
* Fix tests and method calls.
We plan to use these records to check and record the schema of Zulip's
events for the purposes of API documentation.
Based on an original messier commit by tabbott.
In theory, a nicer version of this would be able to work directly off
the mypy type system, but this will be good enough for our use case.
This extends our email address visibility settings to deny access to
user email addresses even to organization administrators.
At the moment, they can of course change the setting (which leaves an
audit trail), but in the future only organization owners will be able
to change that setting.
While we're at this, we rewrite the settings_data.js test to cover all
the cases in a more consistent way.
Fixes#14111.
This isn't the only bug in our testing libraries with
EMAIL_ADDRESS_VISIBILITY; but we don't have a lot of tests that need
to deal with that set of settings.
This updates update the download android and ios app button on
/apps/android and /apps/ios routes respectively to use the official
badges provided by the google and apple.
We also clean up some of the JavaScript implementing the page.
Fixes#14061.
We will cache failed lookups with None. The
use case here is that broken API clients may
continually ask for the same wrong API key, and
we want to handle that as quickly as possible.
The example regexes for linkifier settings are not themed according
to the user's dark theme setting.
Modify 'night_mode.scss' to render all 'code' elements with dark
theme.
We were using `code` to pass around messages.
The `code` field is designed to be a code, not
a human-readable message.
It's possible that we don't actually need two
flavors of messages for these type of validations,
but I didn't want to change that yet.
We **definitely** don't need to put two types of
message in the exception, so I fix that. Instead,
I just have the caller ask what level of detail
it needs.
I added a non-verbose message for the case of
system bots.
I removed the non-translated version of the message
for deactivated accounts, which didn't have test
coverage and is slightly more prone to leaking
email info that we don't want to leak.
In the prep commits leading up to this, we split
out two new helpers:
validate_email_is_valid
get_errors_for_new_emails
Now when we validate invites we use two separate
loops to filter our emails.
Note that the two extracted functions map to two
of the data structures that used to be handled
in a single loop, and now we break them out:
errors = validate_email_is_valid
skipped = get_errors_for_new_emails
The first loop checks that emails are even valid
to begin with.
The second loop finds out whether emails are already
in use.
The second loop takes advantage of this helper:
get_errors_for_new_emails
The second helper can query all potential new emails
with a single round trip to the database.
This reduces our query count.
The main purpose of this new function is to allow
us to validate emails in bulk, which we don't do
yet (still setting the stage for that).
This is still a speedup, though, since in our
caller we grab only three fields now.
And other than that, we're essentially doing
the same query for the single-email case, just
outside the loop.
We are trying to kill off `validate_email`, so
we no longer call it from these tests.
These tests are already kind of low-level in
nature, so testing the more specific helpers
here should be fine.
Note that we also make the third parameter
to `validate_email` non-optional in this commit,
to preserve 100% coverage. This is really just
refactoring noise--we will soon eliminate the
entire function, but I didn't want to do everything
in a huge commit.
This is a prep commit that will allow us
to more efficiently validate a bunch of
emails in the invite UI.
This commit does not yet change any
behavior or performance.
A secondary goal of this commit is to
prepare us to eliminate some hackiness
related to how we construct
`ValidationError` exceptions.
It preserves some quirks of the prior
implementation:
- the strings we decided to translate
here appear haphazard (and often
get ignored anyway)
- we use `msg` in most codepaths,
but use `code` for invites
Right now we never actually call this with
more than one email, but that will change
soon.
Note that part of the rationale for the inner
method here is to avoid a test coverage bug
with `continue` in loops.
We are trying to elminate the version of
`validate_email` that lives in `actions.py`.
Inlining it barely increases the code size, and
it removes some noise related the three-item
tuple that `check_incoming_email` returns.
This has two goals:
- sets up a future commit to bulk-validate
emails
- the extracted function is more simple,
since it just has errors, and no codes
or deactivated flags
This commit leaves us in a somewhat funny
intermediate state where we have
`action.validate_email` being a glorified
two-line function with strange parameters,
but subsequent commits will clean this up:
- we will eliminate validate_email
- we will move most of the guts of its
other callee to lib/email_validation.py
To be clear, the code is correct here, just
kinda in an ugly, temporarily-disorganized
intermediate state.
We now use the `get_realm_email_validator()`
helper to build an email validator outside
the loop of emails in our invite list.
This allows us to perform RealmDomain queries
only once per request, instead of once per
email.
We now query RealmDomain objects up front. This
change is minor in most circumstances--it sometimes
saves a round trip to the database; other times,
it actually brings back slightly more data
(optimistically).
The big win will come in a subsequent commit,
where we avoid running these queries in a loop
for every callback.
Note that I'm not sure if we intentionally
omitted checks for emails with "+" in them
for some circumstances, but I just preserved
the behavior.
Now called:
validate_email_not_already_in_realm
We have a separate validation function that
makes sure that the email fits into a realm's
domain scheme, and we want to avoid naming
confusion here.
Without the fix here, you will get an exception
similar to below if you try to invite one of the
cross realm bots. (The actual exception is
a bit different due to some rebasing on my branch.)
File "/home/zulipdev/zulip/zerver/lib/request.py", line 368, in _wrapped_view_func
return view_func(request, *args, **kwargs)
File "/home/zulipdev/zulip/zerver/views/invite.py", line 49, in invite_users_backend
do_invite_users(user_profile, invitee_emails, streams, invite_as)
File "/home/zulipdev/zulip/zerver/lib/actions.py", line 5153, in do_invite_users
email_error, email_skipped, deactivated = validate_email(user_profile, email)
File "/home/zulipdev/zulip/zerver/lib/actions.py", line 5069, in validate_email
return None, (error.code), (error.params['deactivated'])
TypeError: 'NoneType' object is not subscriptable
Obviously, you shouldn't try to invite a cross
realm bot to your realm, but we want a reasonable
error message.
RESOLUTION:
Populate the `code` parameter for `ValidationError`.
BACKGROUND:
Most callers to `validate_email_for_realm` simply catch
the `ValidationError` and then report a more generic error.
That's also what `do_invite_users` does, but it has the
somewhat convoluted codepath through `validate_email`
that triggers this code:
try:
validate_email_for_realm(user_profile.realm, email)
except ValidationError as error:
return None, (error.code), (error.params['deactivated'])
The way that we're using the `code` parameter for
`ValidationError` feels hacky to me. The intention
behind `code` is to provide a descriptive error to
calling code, and it's not intended for humans, and
it feels strange that we actually translate this in
other places. Here are the Django docs:
https://docs.djangoproject.com/en/3.0/ref/forms/validation/
And then here's an example of us actually translating
a code (not part of this commit, just providing context):
raise ValidationError(_('%s already has an account') %
(email,), code = _("Already has an account."),
params={'deactivated': False})
Those codes eventually get put into InvitationError, which
inherits from JsonableError, and we do actually display
these errors in the webapp:
if skipped and len(skipped) == len(invitee_emails):
# All e-mails were skipped, so we didn't actually invite anyone.
raise InvitationError(_("We weren't able to invite anyone."),
skipped, sent_invitations=False)
I will try to untangle this somewhat in upcoming commits.
We allow folks to invite emails that are
associated with a mirror_dummy account.
We had a similar test already for registration,
but not invites.
This logic typically affects MIT realms in the
real world, but the logic should apply to any
realm, so I use accounts from the zulip realm
for convenient testing. (For example, we might
run an IRC mirror for a non-MIT account.)
I use a range here because there's some leak
from another test that causes the count to
vary. Once we get this a bit more under control,
we should be able to analyze the leak better.
The substantive improvement here is to use
a strange casing for Hamlet's email, which
will prevent future casing bugs.
I also log in as Cordelia to prevent confusion
that the test has something to do with
inviting yourself. It's more typical for
somebody to invite another person to a realm
(not realizing they're already there).
I also made two readability tweaks.
Replaced unique_together with UniqueConstraint in models that
covered nullable fields as in unique_together database indexes
don't work where subgroup=None. So added conditional unique
index handling invalid duplicate Count data.
Added 0015_clear_duplicate_counts migration to handle existing
data that violates the constraints.
Also corrected a test case in test_counts.py which didn't clear its
state properly and thus was accidentally taking advantage of this
database schema bug.
The copy of the styling for users_hover_info:
```
-#users_hover_info {
- left: 25px;
- top: -40px;
-}
```
Looks less good than the common one with #hoverinfo, so we remove it.
Currently, the cursor for the date input field in the settings page
is 'not-allowed' as it has the disabled attribute because we want
users to pick the date from the date picker. But this leads to
confusion whether the field is editable at all.
Change the cursor to 'pointer' to make it clear that the field has
a click action associated with it.
Clicking on the 'Owner' value for a row in the list of bots does
nothing, and causes a blueslip error.
This is because the map object in which we store the users have
integer keys, while we pass the owner id as string.
This is fixed by parsing the owner id to integer before passing it
on.
Fixes#14107.
We need to request access to read:org scope to be able to check org/team
membership. Without it SOCIAL_AUTH_GITHUB_ORG_NAME and
SOCIAL_AUTH_GITHUB_TEAM_ID settings don't work and simply lead to all
auth attempts failing.
Tested manually.
Several of our queues are capable of doing work that includes
rendering markdown (outgoing_webhook, embedded_bots, embed_links, and
email_mirror). As a result, it's essential that these don't cache
per-request data (specifically, realm filters) longer than they
should, making editing/deleting linkifiers potentially use old
settings until the relevant process was restarted.
Flushing these caches is extremely cheap (just clearing two
dictionaries) and thus is reasonable to do after every queue event,
rather than trying to do it only the ~1/3 of queues that specifically
do markdown processing. We do the same in our middleware for
reset_queries.
It's not worth writing a test for this because it's very difficult to
create the test setup situation for this bug with a single test worker
process; one needs to edit the linkifier configuration in a different
process than the one sending the message in order to see the bug.
This was a much larger visible bug on Zulip 2.1.x, where the presence
of the message_sender queue meant that this would apply to messages
sent via a browser.
Fixes#14095.
Previously, the input:
====================
- One
- Two
Two continued
====================
Would produce the same output as:
====================
- One
- Two
```
Two continued
```
====================
This was because our CodeBlockProcessor had a higher priority than
the ListIndentProcessor. This issue was discussed here:
https://chat.zulip.org/#narrow/stream/9-issues/topic/continuation.20paragraphs.20in.20list.20items.
/delete_topic endpoint could be used to request the deletion of a topic,
that would cause do_delete_messages to be called with an empty set in
these cases:
1. Requesting deletion of an empty stream.
2. Requesting deletion of a topic in a private stream with history not
public to subscribers, if the requesting admin doesn't have access to
any of the messages in that topic.
This function slims down the data that we get
from the database in order to create the
streams part of our client payload.
We also fix a typo.
We also clearly distinguish between queries
and lists here.
This new method prevents us from getting fat
objects from the database.
Instead, now we just get ids from the database
to build our subqueries.
Note that we could also technically eliminate
the `set(...)` wrappers in this code to have
Django make a subquery and save a round trip.
I am postponing that for another commit (since
it's still somewhat coupled to some other
complexity in `do_get_streams` that I am trying
to cut through, plus it's not the main point
of this commit.)
BEFORE:
# old, still in use for other codepaths
def get_stream_subscriptions_for_user(user_profile: UserProfile) -> QuerySet:
# TODO: Change return type to QuerySet[Subscription]
return Subscription.objects.filter(
user_profile=user_profile,
recipient__type=Recipient.STREAM,
)
user_subs = get_stream_subscriptions_for_user(user_profile).filter(
active=True,
).select_related('recipient')
recipient_check = Q(id__in=[sub.recipient.type_id for sub in user_subs])
AFTER:
# newly added
def get_subscribed_stream_ids_for_user(user_profile: UserProfile) -> QuerySet:
return Subscription.objects.filter(
user_profile_id=user_profile,
recipient__type=Recipient.STREAM,
active=True,
).values_list('recipient__type_id', flat=True)
subscribed_stream_ids = get_subscribed_stream_ids_for_user(user_profile)
recipient_check = Q(id__in=set(subscribed_stream_ids))
We calculate `max_message_id` for the mobile client.
Our query now no longer joins to the Message table
and just grabs one value instead of fat objects.
We were only checking error handling before, not
the happy path. The structure of the code
made it so that we effectively tested most of the
logic for this use case (since all the other flags
are sort of just filters on top of this), but
obviously we want explicit coverage here. Also,
we weren't testing the is-admin-but-not-api-super-user
error checking until this commit.
The file populates `windows.i18n`, so now
the file name matches our convention.
Note that the module really just initializes
`i18next` and then does this:
window.i18n = i18next;
It doesn't really add any functionality to
third party library.
Before this test, we were validating the behavior
of `i18next`, but we weren't validating our light
layer that sits on top of `i18next`, which currently
resides in the slightly misnamed `translations.js`
file.
The translations module is now so small that I'll
just quote it verbatim here:
import i18next from 'i18next';
i18next.init({
lng: 'lang',
resources: {
lang: {
translation: page_params.translation_data,
},
},
nsSeparator: false,
keySeparator: false,
interpolation: {
prefix: "__",
suffix: "__",
},
returnEmptyString: false, // Empty string is not a valid translation.
});
window.i18n = i18next;
We now just do `zrequire('translations')` to initialize
the `i18next` library, which allows us to have simpler
test setup and to actually exercise the above call to
`i18next.init`.
This change now gives us 100% line coverage of `translations.js`,
which of course isn't that hard to acheive (see above).
Before 2018, we used a feature of i18next where
we would cache translations in local storage
for up to two weeks:
var cacheOptions = {
// ...
prefix: 'i18next:' + page_params.server_generation + ':',
expirationTime: 2*7*24*60*60*1000, // 2 weeks
};
i18next.init({
/// ...
cache: cacheOptions
}
Because `server_generation` would change each time you
upgraded a server, a frequently upgraded server like
chat.zulip.org would cause its active users to start
to accumulate lots of obsolete key/value pairs in local
storage over the two weeks.
See #4443 for more details.
We eventually reduced the cache life to 2 days. And then
on top of that, newer versions of the server would start
to clean up after themselves using this commit from
April 2017:
e3f1d025ae
We then removed the caching option altogether a year
later in May 2018:
cff40c557b
We kept around the code to remove all the old keys, though.
This was particularly important for users who may have
been hitting servers that did an upgrade to the new
version from some older version that didn't have the
key-fixing code.
But mostly the problem takes care of itself after
either two days or two weeks, even on really out-of-date
servers.
The original problem was most likely to affect server
admins that did a lot of upgrades (and possibly only really
affected chat.zulip.org), so as long as those server
admins continued their patterns, it's highly likely that
they've done several upgrades since May 2018 that would
have cleaned these keys out for good.
And, again, even if there is some strange straggler here,
they probably only have one set of keys that will expire
either two days or two weeks after an upgrade, depending
on how long ago the prior upgrade was. (All of their
keys based on older versions of `server_generation` would
have long since expired.)
Finally, any upgrade certainly won't make the problem
worse for any users under this hypothetical situation,
since the new server won't be writing new keys.
So I am removing the cleanup code.
This extracts a new module with three
functions, which we will test with 100%
line coverage:
- show_email
- email_for_user_settings
- get_time_preferences
The first two break several dependencies
in the codebase on `settings_org.js`. The
`get_time_preferences` breaks an annoying
dependency on `page_params` within people.
The module is pretty cohesive, in terms that
all three functions are just light wrappers
around `page_params` and/or `settings_config`.
Now all the modules that want to call show_email()
only have to require `settings_data`, instead of
having a dependency on the much heavier
`settings_org.js` module.
I also make some of the unit tests here be more
full-stack, where instead of stubbing show_email,
I basically just toggle `page_params.is_admin`.
Explicitly stubbing i18n in 48 different files
is mostly busy work at this point, and it doesn't
provide much signal, since often it's invoked
only to satisfy transitive dependencies.
For historical reasons we were creating Recipient
objects at some point in the typing-notifications
codepath. Now we just work with UserProfiles.
This removes some queries, as indicated by
the change to `len(queries)` in a couple of the
tests.
The one subtle thing that changes here is huddles.
If user 10 sends a typing notification that they
are talking to users 20 and 30, there might not
actually be a huddle for users 10/20/30, but
we were actually creating huddles on the fly!
There is no need to create huddles just for
typing notifications, since we don't even
share huddle ids with our clients. The clients
just infer the huddles.
Some of the code that gets killed off here as
somewhat "collateral damage" is some
defensive code related to formerly supporting streams
in typing indicators. The support for streams
was killed off almost as soon as we released
the feature, and the codepath is pretty clearly
user-centric at this point.
I actually like this pattern:
def check_send_typing_notification(...):
typing_notification = check_typing_notification(...)
do_send_typing_notification(...)
It can help divide responsibilities nicely and make it easy
to write detailed unit tests against each of the two helpers.
Unfortunately, the good things didn't really happen here, and
instead we got the worst aspects of the pattern:
- The responsibilities for validation leaked into
the second function.
- Both functions were doing sane things individually
that became not-so-sane in the big picture (namely,
we ended up making Recipient objects for no reason,
but if you read each of the helpers, it was just one
step that seemed reasonable).
- Passing around dictionaries for results can be annoying.
Also, the pattern made a lot more sense when the validation
for typing was a lot more complicated. My prior commit makes
it so that we only ever deal with a list of user_ids.
Anyway, now I'm inlining it. :)
Subsequent commits will clean up the more substantive issue
here, which is that we are building Recipients for no reason.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
Unfortunately, we still have some older versions
of mobile that still send emails.
In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.
One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion). It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.
We also clean up the test structure for the
typing tests:
TypingHappyPathTest.test_start_to_another_user
TypingHappyPathTest.test_start_to_multiple_recipients
TypingHappyPathTest.test_start_to_self
TypingHappyPathTest.test_start_to_single_recipient
TypingHappyPathTest.test_stop_to_another_user
TypingHappyPathTest.test_stop_to_self
TypingValidateOperatorTest.test_invalid_parameter
TypingValidateOperatorTest.test_missing_parameter
TypingValidateUsersTest.test_argument_to_is_not_valid_json
TypingValidateUsersTest.test_bogus_user_id
TypingValidateUsersTest.test_empty_array
TypingValidateUsersTest.test_missing_recipient
TypingValidationHelpersTest.test_recipient_for_user_ids
TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id
TypingLegacyMobileSupportTest.test_legacy_email_interface
Users who are using ZulipDesktop or haven't managed to auto-update to
ZulipElectron should be strongly encouraged to upgrade.
We'll likely want to move to something even stricter that blocks
loading the app at all, but this is a good start.
Before the Django 2.x upgrade, the DatabaseCreation
argument took an integer value. To deal with running
mulitple test instances, we created a random start
range that could count up 100 workers until the next
random id. Arbitrarily limiting the number of workers
to 100.
Post upgrade, we can now use string values. Enabling
the database + worker numbers to be more readable, as
well as removing the cap on the worker count.
This follows the convention of other code calling into
add_sub_to_table of checking whether the stream settings overlay is
open (and thus in the DOM) before trying to rerender it.
This fixes a bug where you can’t open the same overlay twice in a row
in IE 11, which doesn’t support HashChangeEvent.oldURL; it was exposed
by commit 05be16e051 (late 2018).
While here, parse the hash from oldURL in a less ad-hoc way.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We add these two functions to the API,
so that we no longer have `alert_words_ui`
using private data from `alert_word`:
alert_words.has_alert_word()
alert_words.get_word_list()
And to initialize the data, we have a proper
`initialize` method that is passed in only
the parameters that it needs from `ui_init`.
(We also move the step of deleting `alert_words`
from `page_params` to the `ui_init` module.)
Because it's a bit less cumbersome to initialize
`alert_words`, we now just it directly in the
node tests for `alert_words_ui`.
This is follow up to da79fd206a
I accidentally skipped over pm_conversations. Same
ideas as the bigger previous commit--we pass in params
to the initialize function and do the delete cleanup
within ui_init.
Calling a function with hundreds of thousands to millions of
arguments, depending on the browser, can throw a RangeError. This was
true of both ids.push(...a) and the [].concat.apply construction that
it replaced in commit 59d55d1e06,
although the old one was less likely to overflow due to bucketing.
Use a loop instead.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Let's say you have module hello.js like so:
// hello.js
const hello_world = i18n.t('Hello world');
exports.get_greeting = () => hello_world;
And then two modules like this:
// apple.js
const hello = require('hello');
exports.foo = () => {
show_greeting(hello.get_greeting());
};
// banana.js
const hello = require('hello');
exports.foo = () => {
display_greeting(hello.get_greeting());
};
The test for apple.js could look like this,
and it won't crash due to the stub:
set_global('i18n', {t: () => {}});
zrequire('hello');
zrequire('apple');
Now let's say your write this broken version
of a test for banana.js:
zrequire('hello');
zrequire('banana');
If you run `./tools/test-js-with-node`, the
"banana" test will pass, because while it
does require "hello", it won't actually
*execute* the code that happens at require
time for "hello", because it's already in
the cache. Here is the code that gets
skipped:
const hello_world = i18n.t('Hello world');
But then if you try to run the banana test
individually, the above line of code will
cause the test to crash. And it will crash
even before you actually try to test the
meaningful code here:
exports.foo = () => {
display_greeting(hello.get_greeting());
};
This commit fixes this leak scenario by just
aggressively clearing out things from the
require cache.
This slows tests down by about 10%, which I think
is worth the extra safety here.
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.
Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
This field wasn't accessed by any clients and was a less robust
version of the user_id field. Any client hoping to be interested in
who did message edits should be able to handle working with user IDs
rather than email addresses.
This is preparation for supporting moving messages between streams in
some cases.
It doesn't actually have any functional effect, since flush_message
clears the message unconditionally anyway.
This cleans up the handoff of page_params
data between ui_init and modules that
take over ownership of page_params-derived
data.
Read the long comment in ui_init for a bit
more context.
Most of this diff is actually test cleanup.
And a lot of the diff to "real" code is
just glorified `s/page_params/params/`
in the `initialize` functions.
One little oddity is that we don't actually
surrender ownership of `page_params.user_id`
to `people.js`. We could plausibly sweep
the rest of the codebase to just use
`people.my_user_id()` consistently, but it's
not a super high priority thing to fix,
since the value never changes.
The stream_data situation is a bit messy,
since we consume `page_params` data in the
initialize() function in addition to the
`params` data we "own". I added a comment
there and intend to follow up. I tried
to mostly avoid the "word soup" by extracting
three locals at the top.
Finally, I don't touch `alert_words` yet,
despite it also doing the delete-page-params-data
dance. The problem is that `alert_words`
doesn't have a proper `initialize()`. We
should clean that up and have it use a
`Map` internally, too.
We should not need so many queries here,
although a couple of the queries are just
standard things that apply to all requests.
I will reduce the number of queries in a
later commit.
This is mostly refactoring, but we also prevent a new
type of value error (list of non-int-or-string). The
new test code helps enforce that.
Cleanup includes:
- Use early-exit for email case.
- Rename helpers to get_validate_*.
- Avoid clumsy rebuilding of lists in helpers.
- Avoid the confusing `recipient` name (which
can be confused with the model by the same
name).
- Just delegate duplicate-id/email-removal to
the helpers.
The cleaner structure allows us to elminate a couple
mypy workarounds.
Credits to @xpac1985 for reporting, debugging and proposing fix to the
issue. The proposed fix was modified slightly by @hackerkid to set the
correct value for max_invites and upload_quota_gb. Tests added by
@hackerkid.
Fixes#13974
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.
Comments on the Octopus emoji added by tabbott.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is not always a behavior-preserving translation: _.defaults
mutates its first argument. However, the code does not always appear
to have been written to expect that.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is not always a behavior-preserving translation: _.extend mutates
its first argument. However, the code does not always appear to have
been written to expect that.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is not always a behavior-preserving translation: $.extend mutates
its first argument. However, the code does not always appear to have
been written to expect that.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Bounce five times, once every 5 seconds, rather than forever every
0.75 seconds. This reduces annoying user distraction and idle CPU/GPU
consumption.
Fixes#13760.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Now the caller simply imports the debug ‘require’ function as a
module, deciding for itself how to expose it and with what name (in
our case, we expose it as ‘require’ with expose-loader). Also, remove
a stray console.log.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This test wasn't particularly high value, was flaky, and would be
better rewritten as a set of node tests verifying the logic that would
run 100x as fast and more reliably for similar testing fidelity.
"Outreach programs" I think is better phrasing than "Internship
programs", since GSoC is the main one we do these days and isn't an
internship program.
This may fail CI since it contains links to anchors on ReadTheDocs
that may race; it should fail ones the ReadTheDocs build completes.
We were incorrectly passing a string version of an integer ID,
e.g. "10", to a function expecting an integer, e.g. 10. Fix this by
using the common get_stream_id function intended for the purpose
rather than hand-written parsing.
This was likely broken in the recent Dict -> IntDict/Map migrations.
This fixes a confusing aspect of how our automated tests worked
previously, where we'd almost all HTTP requests in the unlikely
configuration with no User-Agent string specified.
We need to adjust query counts in a few tests that now are a bit
cheaper because they now can take advantage of a Client object created
in server_initialization.py in `process_client`.
This adds a global require() function that makes JS modules accessible
to the browser console without adding them to the global window
object:
» const typeahead = require("./static/shared/js/typeahead");
» typeahead.popular_emojis
Array(6) [ "1f44d", "1f389", "1f642", "2764", "1f6e0", "1f419" ]
The list of known modules is exposed via the keys of require.ids
object.
This will allow us to migrate more modules to ES6 without losing
access to this debugging functionality.
I’ll probably upload this plugin to NPM at some point, but I figured
I’ll let it bake in-tree first.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
- I fixed a typo with "lowerecase"
- I elaborated on the prefix before elaborating
on the rest of the message (i.e. went in correct
order).
- I split out the provision example (since we
talk about it some depth).
- I added more positive examples.
- I removed the distracting italics around the
good commit messages.
- I moved the "gather_subscriptions" commit to
the bottom of the list, since we elaborate
on that below the list.
So far the conversion was in a very random place -
register_remote_user(). All other codepaths that use
login_or_register_remote_user() call it with the user's email address.
Making remote_user_sso convert remote_username to the email address
before calling login_or_register_remote_user makes this usage consistent
across the board.
finish_desktop_flow is called with the assumption that the request
successfully proved control over the user_profile and generates a
special link to log into the user_profile account. There's no reason to
pass the realm param, as user_profile.realm can be assumed.
Extend the context dictionary with variables `social_backend_name`
and `backend_error` flag which determines if the error should be
shown. Not extended this for ldap, smtp and saml as they have a
different format of block.
In `auth.py` there are three `if` blocks for different backends
to redirect to config error page with similar code. It is better
handled with common code using `get_attr()` function on
constructed setting names.
There was some duplicated code to test config error pages for
different auths which could be handled with less duplicated code
by adding those functions to `SocialAuthBase`.
Also moving the other tests makes it easier to access tests related
to a backend auth when they are in the same file.
We were computing id_of_last_message_sent_by_us
for a valid reason before
fa44d2ea69
was committed in December 2017 to remove the
autoscroll_forever setting.
Since then the only thing that the
conditional for `id_of_last_message_sent_by_us`
short-circuits is a buggy computation of
`id_of_last_message_sent_by_us` itself.
Removing this dead code obviously makes the code
more clear, plus it does save some needless and
possibly bug-prone computation.
In particular, I am trying to lock down `rows.id` to
be more strict about receiving bogus elements, and
removing this code will help with that.
The bug was in complex `if` condition, which should mean that users should
be allowed to create a User group only when they are either admin or user
group creation policy is set to everyone.
Fixes: #13909.
We now no longer do local echo if a user has logged in or visited a
narrow so recently that we are still fetching new messages for them in
their current message list.
Since we want any message list we're displaying to show only
contiguous sequences of messages within that view, it's not correct to
append messages that were just sent at the end unless
fetch_status.has_found_newest shows that we are up to date with the
latest messages from the server.
While we have some logic aimed at correcting our-of-order message IDs
in Zulip, even a brief (few seconds) temporary display of that is a
bug that we should avoid.
This means that we should disable local echo when the user's current
narrow is not up to date. We can be sure that we'll get the message
the user sent from the server either during the catch-up process or
when we receive it back from th server via the events system.
That particular race window can be several seconds in situations where
somebody is in a narrow where their pointer (or equivalent) is far
behind the latest messages.
This commit only fixes the local echo race condition. There's a
related bug where new messages sent by (potentially other) users
delivered to the client via server_events might race with our fetching
until we get the latest messages in a given narrow, which we'll need
to deal with separately.
See https://github.com/zulip/zulip/issues/8989 for more details. It's
possible that we'll close the issue after this fix, since any
additional fixes would add a lot of complexity, and I'm not sure how
much of a problem this will really be in practice after this fix.
Note that we don't have great automated testing for
`try_deliver_locally` (or really `echo.js` in general). For
`try_deliver_locally` the node tests would probably be 8x more complex
than the code itself, since that function is basically "glue" code
touching several external dependencies. It's also kind of hard to
screw up this code without getting pretty obvious failures early in
the QA process.
Fixes#8989.
With the new Map, we want to make sure we
convert the square number into an int.
The symptom here was you'd click on the
square, and the data would get passed
around via the event system, but when
we went to draw the board, the idx value
was a string.
This moves some code from settings_display.js
into the new module settings_config.js.
Extracting this module breaks some dependencies
on settings_display.js (which has some annoying
transitive dependencies, including jQuery).
In particular this isolates stream_data from
from settings_display.js.
Two of the three structures that we moved here
weren't even directly used by settings_display.js,
since we do a lot of rendering in the modules
admin.js and setting.js.
We make get_all_display_settings() a function
to avoid a require-time dependency on page_params.
Breaking the dependencies simplifies a few
node tests.
Most of the node test complexity came from the
following commit in March 2019:
5a130097bf
The commit itself seems harmless enough, but
dependencies can have a somewhat "viral" nature,
where making stream_data depend on settings_display
caused us to modify four different node tests.
This cleans up a few things:
- just yield values so we don't have to do
tedious max logic
- use values() instead of items() for
skin_variations loop
In the ideal world the emoji.json would reduce this
code to `get_square_size = lambda data: data['square_size']`,
but I don't think we can get the square size explicitly.
This commit changes the calculation of the
background-size parameter that we use to
render emojis from sprite sheets.
In particular, it now makes the parameter
match the sizes of our latest sprite
sheets from Twitter/Google.
This should fix the geometry aspect of #13959,
but we also need to fix some issues with the
cache being sticky.
There is also some minor cleanup:
- Remove obsolete -moz/-webkit CSS.
- Remove needless precision in percentages.
- Fix the transposed nrows/ncols names.
- Add extensive commenting.
Finally, we add a minor bump to the provision
number. This commit should be merged in the
same series as the other fix for this issue,
which will probably have a major bump, and we'll
need to rebase this appropriately.
This fixes an issue where the /apps page would have gradient colors
awkwardly overlapping the footer in mobile views.
This was because the /apps page was sharing /hello page gradient HTML
(defined in zerver/gradients.html), and the /apps content isn't tall
enough for the gradient content to be under actual content.
The fix is simple: Just don't include the gradient for /apps. The
design for the page was long ago changed to not use the gradient.
We don't expect a similar bug with the gradients in other pages
because they all have enough content to have the gradients end well
before reaching the footer.
Fixes#13375.
SOCIAL_AUTH_SAML_SECURITY_CONFIG["authnRequestsSigned"] override in
settings.py in a previous commit wouldn't work on servers old enough to
not have the SAML settings in their settings.py - due to
SOCIAL_AUTH_SAML_SECURITY_CONFIG being undefined.
This commit fixes that.
Original idea was that KeyError was only going to happen there in case
of user passing bad input params to the endpoint, so logging a generic
message seemed sufficient. But this can also happen in case of
misconfiguration, so it's worth logging more info as it may help in
debugging the configuration.
Create a new page for desktop auth flow, in which
users can select one from going to the app or
continue the flow in the browser.
Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
While it's a bit of extra complexity to do this check, which I'm not
excited about, we've had multiple folks spend significant time being
confused rebasing past d7d8632525 into
deleting `pygments_data.json`, with provision not rebuilding it, so
this seems worth merging as a transitional fix even if we decide to
remove it in 2 months.
To avoid some hidden bugs in tests caused by every ldap user having the
same password, we give each user a different password, generated based
on their uids (to avoid some ugly hard-coding in a bunch of places).
Without calling cov.erase() the data file seems to persist and even
pollute future test runs if not removed. Registering an atexit handler
seems like a good, and reasonably clean way to ensure the cleanup
happens.
Fixes#13933.
Puppet doesn’t re-run an exec blocks that’s declared as creating an
existing file, even if it’s notified. Remove the creates declaration.
Fixes#13730.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
This refactoring is the first step toward sharing
our markdown code with mobile. This focuses on
the Zulip layer, not the underlying third party `marked`
library.
In this commit we do a one-time initialization to
wire up the markdown functions, but after further
discussions with Greg, it might make more sense
to just pass in helpers on every use of markdown
(which is generally only once per sent message).
I'll address that in follow-up commits.
Even though it looks like a pretty invasive change,
you will note that we barely needed to modify the
node tests to make this pass. And we have pretty
decent test coverage here.
All of the places where we used to depend on
other Zulip modules now use helper functions that
any client (e.g. mobile) can configure themselves.
Or course, in the webapp, we configure these from
modules like people/stream_data/hash_util/etc.
Even in places where markdown used to deal directly with
data structures from other modules, we now use functions.
We may revisit this in a future commit, and we might
just pass data directly for certain things.
I decided to keep the helpers data structure completely flat,
so we don't have ugly nested names like
`helpers.emoji.get_emoji_codepoint`. Because of this,
some of the names aren't 1:1, which I think is fine.
For example, we map `user_groups.is_member_of` to
`is_member_of_user_group`.
It's likely that mobile already has different names
for their versions of these functions, so trying for
fake consistency would only help the webapp. In some
cases, I think the webapp functions have names that
could be improved, but we can clean that up in future
commits, and since the names aren't coupled to markdown
itself (i.e. only the config), we will be less
constrained.
It's worth noting that `marked` has an `options`
data structure that it uses for configuration, but
I didn't piggyback onto it, since the `marked`
options are more at the lexing/parsing layer vs.
the app-data layer stuff that our helpers mostly
help with.
Hopefully it's obvious why I just put helpers in
the top-level namespace for the module rather than
passing it around through multiple layers of the
parser.
There were a couple places in markdown where we
were doing awkward `hasOwnProperty` checks for
emoji-related stuff. Now we use the Python
principle of ask-forgiveness-not-permission and
just handle the getters returning falsy data. (It
should be `undefined`, but any falsy value is
unworkable in the places I changed, so I use
the simpler, less brittle form.)
We also break our direct dependency on
`emoji_codes.json` (with some help from the
prior commit).
In one place I rename streamName to stream_name,
fixing up an ancient naming violation that goes
way back to before this code was even extracted
away from echo.js. I didn't bother to split this
out into a separate commit, since 2 of the 4
lines would be immediately re-modified in the
subsequent commit.
Note that we still depend on `fenced_code`
via the global namespace, instead of simply
requiring it directly or injecting it. The
reason I'm postponing any action there is that
we'll have to change things once we move
markdown into a shared library. (The most
likely outcome is that we'll rename/move both files
at the same time and fix the namespace/require
details as part of that commit.)
Also the markdown code still relies on `_` being
available in the global namespace. We aren't
quite ready to share code with mobile yet, but the
underscore dependency should not be problematic,
since mobile already uses underscore to use the
webapp's shared typing_status module.
This mostly moves logic into people.js.
The people functions added here are glorified
two-liners.
One thing that changes here is that we
are a bit more rigorous about duplicate
names.
The code is slightly awkward, because this
commit preserves the strange behavior
that if 'alice|42' doesn't match on
the user with the name "alice" and user_id
"42", we instead look for a user whose
name is "alice|42". That seems like a
misfeature to me, but there's a test for
it, so I want to check with Tim that it's not
intentional behavior before I simplify
the code.
We add this API to emoji.js, so that markdown
doesn't need to look at internal data structures
(or even need to understand any kind of record
format for results).
Here are the functions:
get_realm_emoji_url()
get_emoji_name()
get_emoji_codepoint()
We use the API now in markdown, which eliminates
the need for the markdown parser to require
the emoji JSON file.
Each function has a simple docstring:
get_emoji_name('1f384') === 'holiday_tree'
get_emoji_codepoint('avocado') === '1f951'
get_realm_emoji_url('shrug') === '/user_avatars/2/emoji/images/31.png'
Also we have simple test coverage for the API
(including tests that verify the docstrings).
This name was misleading, because we weren't
actually setting realm_filters (that's what
`page_params.realm_filters = realm_filters`
is for); we were instead updating our
realm filter rules.
Commit 612b237cec introduced a
regression that broke the “Discard” button, because
get_subsection_property_elements returns a jQuery object rather than
array, and jQuery objects don’t have a forEach method. Change it to
return an array.
[anders@zulipchat.com: Use Array.from instead of .toArray to avoid the
need for extra mocking.]
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This avoids using `.save()` directly for editing stream properties,
and also uses the API in _send_and_verify_message to avoid confusing
logic around which user is doing what request.
Fixes part of #13823
This allows us to collect coverage for Handlebars templates, and also
improves the readability of Handlebars-related stack traces.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In Django 2.2 the autoreload system has changed.
DJANGO_AUTORELOAD_ENV env variable should be set when calling code
that'll use the autoreloader. Otherwise there's some kind of race
condition in the autoreload code when SIGINT is sent, where
restart_with_reloader() (called only if the env variable isn't set)
has the subprocess module calling p.kill() on a process that's already
exited, raising ProcessLookupError and printing an ugly traceback. This
causes non-deterministic test-run-dev failures.
This adds update_user to python_examples.py in zerver/openapi.
This also adds update-user.md to templates/zerver/api and adds
update-user to rest-endpoints.md (templates/zerver/help/include).
This adds deactivate_user to python_examples.py in zerver/openapi.
This also adds delete-user.md to templates/zerver/api and adds
delete-user to rest-endpoints.md (templates/zerver/help/include).
This adds get_single_user to python_examples.py in zerver/openapi.
This also adds get-single-user.md to templates/zerver/api and adds
get-single-user to rest-endpoints.md (templates/zerver/help/include).
This adds the OpenAPI format data for /users/{user_id} endpoint
and also removes 'users/{user_id}' from 'pending_endpoints' in
zerver/tests/test_openapi.py .
We are gonna phase out util.get_message_topic()
in our entire codebase eventually, but we
certainly don't need it here, since the local
echo codepath is using brand new objects that
we construct inside the compose code, and
there's no danger of legacy "subject" data.
My goal for the markdown code is to keep it
free of any accidental dependencies that we
can easily avoid, as I think there's some
possible future where we split out the code
as its own library for people who want to
render Zulip markdown in non-core projects.
These functions were just shims that were
used in the somewhat painful migration from
subject_* to topic_*.
The commit 4572be8c27
fixed it so that the client never needs to
deal with "subject_links".
So now we just go back to simpler code:
message.topic_links = links
links = message.topic_links
I am not quite ready to declare victory on
the subject/topic migration, but we are super
close. In this commit I bump a blueslip
warning to a blueslip error, so that we'll
be notified of any codepath that is still
using the janky fall-back-to-subject defensive
code here.
If we go a couple days without any errors, then
we can remove the blueslip warning and the
defensive code immediately and then inline
the callers at our leisure. I wouldn't be
wildly against keeping these wrappers in some
parts of the code, but that debate is out of
the scope of this immediate fix, and I haven't
thought hard about it yet.
We can basically sweep set_message_topic() now,
if we wanted to, since it's truly just a one-liner.
(At one point it was encapsulating something
like `message.subject = foo`).
This required a tiny change to compose_fade
test setup.
We now handle the all/everyone/stream case at
the top of userMentionHandler.
Previously the code would do strange things
in the case that some user had the name "all"
or "everyone" or "stream". It would only
affect local echo, and maybe we prevent users
from having those names, so I doubt there
were any real user-facing issues here.
But the new code is clearly more simple and
more correct.
Most of this logic is specific to markdown
message processing, so we move the code to
markdown.js.
The only responsibility that we leave with
`emoji.js` is to provide us with a list
of translations (regex and replacement text).
But now `markdown.js` actually (directly) executes
those translations against Zulip messages
as part of its preprocessing.
This should simplify the upcoming mobile conversion.
Instead of mobile needing to duplicate this fairly
complex function, they will just need to pass
us in a list similar to `emoji_translations` inside
of `emoji.js`. That code has a comment that shows
what the data structure looks like.
There are six emoticon regexes that allow us
make translations such as ":)" to ":slight_smile".
We now build these as soon as we read in the
JSON data, instead of rebuilding them every time
we convert a message to markdown.
It's possible that we should just hardcode this
data:
[
{ regex: /(\:\))/g, replacement_text: ':slight_smile:' },
{ regex: /(\(\:)/g, replacement_text: ':slight_smile:' },
{ regex: /(\:\/)/g, replacement_text: '😕' },
{ regex: /(<3)/g, replacement_text: '❤️' },
{ regex: /(\:\()/g, replacement_text: ':frown:' },
{ regex: /(\:\|)/g, replacement_text: '😑' }
]
OTOH I suppose it's possible that some server
admins will want to modify emoji_codes.json to
have custom emoticons.
I am 99% sure we can rely on trimRight() and
trim() being available in all browsers that
we support. I verified in FF.
This removes the util dependency from both
modules touched here.
We now treat util like a leaf module and
use "require" to import it everywhere it's used.
An earlier version of this commit moved
util into our "shared" library, but we
decided to wait on that. Once we're ready
to do that, we should only need to do a
simple search/replace on various
require/zrequire statements plus a small
tweak to one of the custom linter checks.
It turns out we don't really need util.js
for our most immediate code-sharing goal,
which is to reuse our markdown code on
mobile. There's a little bit of cleanup
still remaining to break the dependency,
but it's minor.
The util module still calls the global
blueslip module in one place, but that
code is about to be removed in the next
few commits.
I am pretty confident that once we start
sharing things like the typeahead code
more aggressively, we'll start having
dependencies on util. The module is barely
more than 300 lines long, so we'll probably
just move the whole thing into shared
rather than break it apart. Also, we
can continue to nibble away at the
cruftier parts of the module.
‘req_var in request.GET’ was previously believed to be slow from
profiling results. However, the real explanation for those profiling
results is that WSGIRequest.GET is a lazy cached property, so there’s
no reason to avoid it if we’re accessing request.GET anyway.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In https://github.com/zulip/zulip/pull/12823 some changes to the realms
structure have been made, so now both in production and development
cross-realm bots live in the realm with string_id "zulipinternal".
There was a TODO in retention code to eliminate a conditional in a query
that became redundant with this change, and also the zulipinternal realm
should be omitted from the archiving process in archive_messages().
Profiling suggests this saves about 600us in the runtime of every GET
/events request attempting to resolve URLs to determine whether we
need to do the APPEND_SLASH behavior.
It's possible that we end up doing the same URL resolution work later
and we're just moving around some runtime, but I think even if we do,
Django probably doesn't do any fancy caching that would mean doing
this query twice doesn't just do twice the work.
In any case, we probably want to extend this behavior to our whole API
because the APPEND_SLASH redirect behavior is essentially a bug there.
That is a more involved refactor, however.
We use this single regular expression for processing essentially every
request, so it's definitely worth hinting to Python that we're going
to do so by compiling it. Saves about 40us per request.
A sloppy implementation of the main has_request_variables wrapper
function meant that it did two very inefficient things:
* To combine together the GET and POST parameters, it would make a
copy of the request.GET QueryDict object, which combined with the
fact that these objects are slow to access, consumed about 90us per
argument.
* Doing this in a loop (one time per argument), rather than once,
which resulted in us doing this 11 times for a `GET /events` query.
Fixing this to just make a dictionary and combine things with some
small loops saved about 1 millisecond from the total runtime of GET
/events (for comparison, the total actual work of that view function
is about 700ms).
We need to fix at least one test that used a bad mock HttpRequest
object that didn't have a .GET property.
The comment explains this issue, but effectively, the upgrade to
Django 2.x means that Django's built-in django.request logger was
writing to our errors logs WARNING-level data for every 404 and 400
error. We don't consider user errors to be a problem worth
highlighting in that log file.
This specifically fixes an issue where a customer chose monthly payment
plan instead of annual, getting stuck with the higher monthly payments.
With this change, the Customer plan can be set to ENDED and they can go
through /upgrade/ again and choose annual payments.
A proper follow-up should be to make it possible to change between
monthly/annual on the /billing/ page.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.
Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:
* urls: django.urls.resolvers.RegexURLPattern has been replaced by
django.urls.resolvers.URLPattern; affects OpenAPI code and related
features which re-parse Django's internals.
https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.
There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.
Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.
Fixes#10835.
Since essentially the first use of Tornado in Zulip, we've been
maintaining our Tornado+Django system, AsyncDjangoHandler, with
several hundred lines of Django code copied into it.
The goal for that code was simple: We wanted a way to use our Django
middleware (for code sharing reasons) inside a Tornado process (since
we wanted to use Tornado for our async events system).
As part of the Django 2.2.x upgrade, I looked at upgrading this
implementation to be based off modern Django, and it's definitely
possible to do that:
* Continue forking load_middleware to save response middleware.
* Continue manually running the Django response middleware.
* Continue working out a hack involving copying all of _get_response
to change a couple lines allowing us our Tornado code to not
actually return the Django HttpResponse so we can long-poll. The
previous hack of returning None stopped being viable with the Django 2.2
MiddlewareMixin.__call__ implementation.
But I decided to take this opportunity to look at trying to avoid
copying material Django code, and there is a way to do it:
* Replace RespondAsynchronously with a response.asynchronous attribute
on the HttpResponse; this allows Django to run its normal plumbing
happily in a way that should be stable over time, and then we
proceed to discard the response inside the Tornado `get()` method to
implement long-polling. (Better yet might be raising an
exception?). This lets us eliminate maintaining a patched copy of
_get_response.
* Removing the @asynchronous decorator, which didn't add anything now
that we only have one API endpoint backend (with two frontend call
points) that could call into this. Combined with the last bullet,
this lets us remove a significant hack from our
never_cache_responses function.
* Calling the normal Django `get_response` method from zulip_finish
after creating a duplicate request to process, rather than writing
totally custom code to do that. This lets us eliminate maintaining
a patched copy of Django's load_middleware.
* Adding detailed comments explaining how this is supposed to work,
what problems we encounter, and how we solve various problems, which
is critical to being able to modify this code in the future.
A key advantage of these changes is that the exact same code should
work on Django 1.11, Django 2.2, and Django 3.x, because we're no
longer copying large blocks of core Django code and thus should be
much less vulnerable to refactors.
There may be a modest performance downside, in that we now run both
request and response middleware twice when longpolling (once for the
request we discard). We may be able to avoid the expensive part of
it, Zulip's own request/response middleware, with a bit of additional
custom code to save work for requests where we're planning to discard
the response. Profiling will be important to understanding what's
worth doing here.
This generalizes existing code for the presence code path that is
generically useful for avoiding useless work that will be discarded.
We make an exception for the one type of request that needs to happen
while reloading, namely the one to clean up our event queue.
We used to have a block of code doing this just in the presence
endpoint because that's where we'd had error-handling problems with it
not being present, but it seems more correct for it to run
unconditionally on all HTTP requests.
This requires adding a dependency of channel on reload_state, which we
record in the webpack configuration for now.
The actual goal we have is that suspect_offline is correct so that we
can rely on that field when determining how to do error handling in
the presence system.
This should return us to a situation where we won't get blueslip
browser error reporting for users created while a device was offline
just before it reloads.
This avoids risk of logging blueslip errors for user IDs seen in the
presence response that we haven't heard about from the server_events
system because we're offline and in the process of reloading.
The issue only affected large realms; see
02bc630881 and `git log
-Ssuspect_offline` for details.
Hellosign now posts their callback as form/multipart, which Django only
permits to be read once. Attempts to access request.body after the
initial read throw "django.http.request.RawPostDataException: You
cannot access body after reading from request's data stream".
Fixes#13847.
This makes it possible to create a Zulip account from the mobile or
desktop apps and have the end result be that the user is logged in on
their mobile device.
We may need small changes in the desktop and/or mobile apps to support
this.
Closes#10859.
1) Created a new class `DatabaseType` and access its objects inside
`template_database_status()` instead of sending five arguments with
default values.
2) Made `check_files` and `setting_name` local variables instead of
function parameters since they had same value(None) for every call.
Fixes#13845.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.
Update the backend to use emoji_codes.json too instead of the three
separate JSON files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I believe we can remove these and rely on
other parts of our testing/code-review
to ensure template quality.
These tests never really exercised our
app code, as evidenced by us not regressing
any of the 100%-line-coverage files.
We have a couple other ways that we verify
the correct format of the templates:
- webpack (can they compile?)
- check-templates (are they nicely indented?)
For deep testing, we have Casper, which
exercises most of our most important templates
in some meaningful way.
I think it's pretty rare that we get bugs
now that are directly caused by bad templates,
and an even smaller subset of them would
have been caught by the node tests.
If that trend changes in the future, I would prefer to
just do something "greenfield" to address
any common problems rather than resurrect
this code, but we could always resurrect it
from git.
The template node tests did check a little bit of
detail about which fields are there, but not
in an integrated way, so that aspect of the tests
wasn't very useful either.
This adds Ubuntu 19.10 as a valid provisioning target.
The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
This code was very useful when first implemented to help catch errors
where our backend templates didn't render, but has been superceded by
the success of our URL coverage testing (which ensures every URL
supported by Zulip's urls.py is accessed by our tests, with a few
exceptions) and other tests covering all of the emails Zulip sends.
It has a significant maintenance cost because it's a bit hacky and
involves generating fake context, so it makes sense to remove these.
Any future coverage issues with templates should be addressed with a
direct test that just accessing the relevant URL or sends the relevant
email.
In python 3.5-3.6, generic types had an __origin__ attribute which
indicated which generic they originated from; the code was reflecting on
that value to check types against the openapi spec. In python3.7, this
changed, and there's no longer an immediately simple way to get this
information in all cases. __origin__ appears to be the implementing
class now, returning `list` or `collections.abc.Iterator` rather than
`typing.List` and `typing.Iterator`. This adds a sloppy-but-effective
mechanism for inferring if a type maps to the List/Dict/Iterator/Mapping
types and gets the test suite passing again.
Every CLI program should have a usage message.
Also add a mention in the `push-to-pull-request` usage message of
its participation in the `refs/remotes/pr/` pseudo-remote feature.
This gives us the right behavior when using the `url.*.insteadOf`
mechanism for aliases in Git remote URLs. For example, if
one's ~/.gitconfig has:
[url "git@github.com:"]
insteadOf = gh:
then `git remote add upstream gh:zulip/zulip` will work great, as
the nice, short, mnemonic `gh:` prefix gets expanded to the more
finicky `git@github.com:`. I use just such a prefix routinely.
But the feature does require that scripts go through the right
abstractions. In particular `git remote get-url`, since Git 2.7
(from 2016), exists for exactly this reason. A plain `git config`
command bypasses the expansion, getting the verbatim `gh:...`
version, which doesn't work.
So, switch to that.
As a bonus, we get to behave correctly if for some reason the user
has configured a push URL distinct from the fetch URL for this
remote, just by adding `--push`. With `git config`, we'd have had
to manually implement the fallback from `remote.upstream.pushUrl` to
`remote.upstream.url` in order to properly handle that case.
The URL used earlier no longer consists of authentication guide for
github and google. So, two different permalinks to google and github
in authentication.html are added to config_error.html to direct the
user to proper authentication setup guide.
We now use realm_id for querying UserPresence
instead of building a big WHERE clause from the
list of user_ids.
This commit may be a bit hard to measure, since
we still get the list of user_ids for the PushToken
query in the same method.
It adds this index:
"zerver_userpresence_realm_id_timestamp_25f410da_idx" btree (realm_id, "timestamp")
We expect this index to provide a major performance improvement when
fetching presence data for the whole realm from the database on
servers like zulipchat.com hosting several realms.
We now validate streams with a separate
function from PM recipients.
It's confusing enough all the ways you can
encode a stream or encode the PM recipients,
but trying to do it all in one function was
hard to reason about and led to at least one
bug.
In particular, there was a bug where streams
with commas in them would get split. Now
we just don't ever split on commas inside
of `extract_stream_indicator`.
Fixes#13836
After removing internal_send_message() in a recent
commit, we now have only two callers for
extract_recipients, and they are both related
to our REQ mechanism that always passes strings
to converters. (If there are default values,
REQ does not call the converters.)
We therefore make two changes:
- use the more strict annotation of "str"
for the `s` parameter
- don't bother with the isinstance check
Note that while the test mocks the actual message
send, we now have a `get_stream` call in the queue
worker, so we have to set up a real stream for
testing (or we could have mocked that as well, but
it didn't seem necessary). The setup queries add
to the amount of queries reported by the test,
plus the `get_stream` call. I just made the
query count a digits regex, which is a little bit
lame, but I don't think it's worth risking test
flakes for this.
This effectively reverts the following
commit from May 2019:
be527905ca
The implementation of closest() was a bit
buggy and complex. It's easy enough
to just stub the method yourself. We may
want to eventually re-implement it, but we
should follow the template of parent/set_parent.
If you fail to stub `closest` zjquery gives
a fairly helpful error message:
Error: You must create a stub for $("link-stub").closest
We stub out jquery elements rather than giving
the illusion of having real DOM.
Also, we make it so that the message_store
interaction has an assertion attached to it.
This index is intended to optimize the performance of the very
frequently run query of "what is the presence status of all users in a
realm?".
Main changes:
- add realm_id to UserPresence
- add index for realm_id
- backfill realm_id for old rows
- change all writes to UserPresence to include
realm_id
The index is of this form:
"zerver_userpresence_realm_id_5c4ef5a9" btree (realm_id)
We will create an index on (realm_id, timestamp) in a
future commit, but I think it's a bit faster if you do
the backfill before the index.
There's also a minor tweak to the populate_db script.
This is just a refactoring to the more modern API
for sending internal messages.
To make this work we now plumb the email_gateway
flag through `internal_send_stream_message` instead
of `internal_send_message`.
We also change `send_zulip` to have its callers
pass in a full UserProfile object (which one of
them already had).
We prefer this to internal_send_message().
We are trying to deprecate `internal_send_message`,
which has extra moving parts related to
`extract_recipients` and `Addressee.legacy_build`.
There are two chunks of code that I touch here
that look pretty similar, but I'm not quite
sure they're worth de-duplicating, since they
use different topics and different message
content.
Instead of having `notify_new_user` delegate
all the heavy lifting to `send_signup_message`,
we just rename `send_signup_message` to be
`notify_new_user` and remove the one-line
wrapper.
We remove a lot of obsolete complexity:
- `internal` was no longer ever set to True
by real code, so we kill it off as well
as well as killing off the internal_blurb code
and the now-obsolete test
- the `sender` parameter was actually an
email, not a UserProfile, but I think
that got past mypy due to the caller
passing in something from settings.py
- we were only passing in NOTIFICATION_BOT
for the sender, so we just hard code
that now
- we eliminate the verbose
`admin_realm_signup_notifications_stream`
parameter and just hard code it to
"signups"
- we weren't using the optional realm
parameter
There's also a long ugly comment in
`get_recipient_info` related to this code
that I amended for now.
We should try to take action in a subsequent
commit.
In the next commit we're going to change what the
server sends for the following:
- page_params
- server responses to /json/users/me/presence
We will **not** yet be changing the format of the data
that we get in events when users update their presence.
It's also just a bit in flux what our final formats
will be for various presence payloads, and different
optimizations may lead us to use different data
structures in different payloads.
So for now we decouple these two things:
raw_info: this is intended to represent a
snapshot of the latest data from the
server, including some data like
timestamps that are only used
in downstream calculations and not
user-facing
exports.presence_info: this is calculated
info for modules like buddy_data that
just need to know active vs. idle and
last_active_date
Another change that happens here is we rename
set_info_for_user to update_info_for_event,
which just makes it clear that the function
expects data in the "event" format (as opposed
to the format for page_params or server
responses).
As of now keeping the intermediate raw_info data
around feels slightly awkward, because we just
immediately calculate presence_info for any kind
of update. This may be sorta surprising if you
just skim the code and see the various timeout
constants. You would think we might be automatically
expiring "active" statuses in the client due to
the simple passage of time, but in fact the precise
places we do this are all triggered by new data
from the server and we re-calculate statuses
immediately.
(There are indirect ways that clients
have timing logic, since they ask the
server for new data at various intervals, but a
smarter client could simply expire users on its
own, or at least with a more efficient transfer
of info between it and the server. One of
the thing that complicates client-side logic
is that server and client clocks may be out
of sync. Also, it's not inherently super expensive
to get updates from the server.)
The important details for the test setup here
are just the number of users who are active.
We don't need to simulate the currently awkward
way of populating this data.
The _.each calls with an inline function expression have already been
converted to for…of loops. We could do that here, but using .forEach
when we’re just reusing an existing function seems like a good
guideline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
predicate is expected to return a function, not a boolean. The
boolean true was causing _.filter to match items with a property named
"true", which is definitely not what was intended. Matching no items
is probably also not intended, but matching every item causes the test
to fail.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This should somewhat reduce the gravity of the failure mode for cases
where the message the user clicked cannot be found (which would be a
significant bug on its own merit in any case).
The keys for message_store are since the recent Map migration intended
to be integer message IDs, not strings (and likely were always
intended to be integers; the failure mode may simply have shifted).
This may just be a new bug, but this max also fix#9549; certainly
we'll want to redo any investigation with this fix in place.
Fixes#9549.
We just get the stream_name from the sub struct now.
This mostly affects node tests.
The only place in real code where we called add_sub()
was when we initialized data from the server.
We now have 100% line coverage on 71 JS files.
This is thanks to about 150 people who have
contributed code to frontend/node_tests.
And then 126 files are still short of 100% line
coverage.
We now enforce line coverage with a set called
EXEMPT_FILES, which are the files for which
we do NOT expect to have 100% coverage.
Using an exemption list makes it so that adding
a new JS file to the project without 100% line
coverage will cause the build to fail. This will
encourage folks to be intentional about their
lack of test coverage.
If a file that had 100% coverage somehow regressed
to 0% coverage, we would report an error to the
console, but we weren't treating it as an actual
failure.
We've probably always had this bug, but it probably
rarely was an issue, since devs might have seen
the error locally, or hopefully whatever crazy
thing you did to totally remove coverage would
have had other symptoms.
If this was intentional, I suspect it might have
had something to do with wanting to get coverage
reports when you just run individual tests. But
a while back we changed it so that when you run
individual tests, we don't do the line coverage
enforcement.
This avoids an unnecessary join to UserProfile.
To verify this, you can do `print(queries)` in the
`test_get_custom_profile_fields_from_api` test. It's
kinda noisy, so I excerpted them below...
Before:
SELECT ...
FROM "zerver_customprofilefieldvalue"
INNER JOIN "zerver_userprofile" ON ("zerver_customprofilefieldvalue"."user_profile_id" = "zerver_userprofile"."id")
INNER JOIN "zerver_customprofilefield" ON ("zerver_customprofilefieldvalue"."field_id" = "zerver_customprofilefield"."id")
WHERE "zerver_userprofile"."realm_id" = 2
After:
SELECT ...
FROM "zerver_customprofilefieldvalue"
INNER JOIN "zerver_customprofilefield" ON ("zerver_customprofilefieldvalue"."field_id" = "zerver_customprofilefield"."id")
WHERE "zerver_customprofilefield"."realm_id" = 2'
I don't have any way to measure the two queries with
realistic data, but I would assume the second
query is significantly faster on most of our instances,
since CustomProfileField should be tiny.
I am trying to optimize a query in this endpoint.
I don't think I'll actually reduce the number of
queries, but I wanted to capture the query and
this was the easiest way to do it, so might as
well check in the code! :)
The line removed here is a noop, as both sides of the
immediately following conditional reassign the
same variable.
This harmless cruft was the result of the recent commit
1ae5964ab8, which added
support for single-user GETs.
This fixes a bug where our asynchronous requests were only copying the
Content-Type header (i.e. the one case where we're noticed) from the
Django HttpResponse. I'm not sure what the impact of this would be;
the rate-limiting headers rarely come up when breaking a long-polled
request. But it seems clearly an improvement to do this in a
consistent fashion.
Only the headers piece is a change; in Tornado
self.finish(x)
is equivalent to:
self.write(x)
self.finish()
We now require all of our unit tests to handle
blueslip errors for warn/error/fatal. This
simplifies the zblueslip code to not have any
options passed in.
Most of the places changed here fell into two
categories:
- We were just missing a random piece of
setup data in a happy path test.
- We were testing error handling in just
a lazy way to ensure 100% coverage. Often
these error codepaths were fairly
contrived.
The one place where we especially lazy was
the stream_data tests, and those are now
more thorough.
This saves a tiny bit of bandwidth, but more
importantly, it protects us against races for
stream name changes. There's some argument that
if the user is thinking they're sending to
old_stream_name, and unbeknownst to them, the
stream has changed to new_stream_name, then we
should fail. But I think 99% of the time the
user just wants the message to go that stream
despite any renames.
In order to verify the blueslip error, we
had to turn on error checking, which required
a tiny fix to a place where we left out
a stream_id for add_sub.
Apparently, the arguments passed to template_database_status were
incorrect for the manual testing development database, in that we
didn't pass a status_dir when calling into that code from provision.
The result was that provisioning before running `test-backend` would
ignore changes to the list of check_files (etc.) made after rebasing,
and vice versa.
The cleanest fix is to compute status_dir from other values passed in;
I'm also going to open a follow-up issue for creating a better overall
interface here.
This adds a new API endpoint for querying basic data on a single other
user in the organization, reusing the existing infrastructure (and
view function!) for getting data on all users in an organization.
Fixes#12277.
This code is a bit flatter and just preps the data
for a single user. There is never any interaction
between the data for user A and user B, so we can
mostly avoid complicated nested data structures
and do most of the data-crunching on a per-user basis.
We also do an explicit sort of the data before
running it through groupby. The explicit sort
simplifies how we calculate `most_recent_info`
and also avoids needing to add `dt` to an intermediate
data structure.
Finally, when it comes to the individual client data,
the code has relied on the assumption that there is
only one row per client, which I believe to be true,
but now the code is more explicit about that.
The word "status" is vague, and this isn't
actually returning a list, so we now name it
get_presence_response.
I originally was gonna rename this to
get_presence_dict, but there's a function
called get_status_dict that returns a subset
of the response, so I think it's a bit more
clear that this is the bigger dict that
actually gets sent back.
We want to err on the side of server_timestamp being
old, since we may eventually use this to make responses
just include incremental changes, and we don't want a
time window (however small) when we miss presence rows.
The clients will be able to deal with duplicate data
to the extent that the time windows are overlapping.
Also, extracting the other local var here
(for `presences`) will set up a subsequent commit
where we re-format the data for clients with
slim_presence=True.
We avoid complicated code to update unread counts
by just using vdom.js.
One small change here is that if click on "more
topics", we replace it with the spinner instead
of putting the spinner after it. This saves us
a redraw under the new scheme.
In e3ad9baf1d, we introduced yet another
bug where we incorrectly shared event dictionaries between multiple
queues.
Fortunately, the logging that reports on "event was not in the queue"
issues worked and detected this on chat.zulip.org, but this is a clear
indication that the comments we have around this system were not
sufficient to produce correct behavior.
We fix this by changing event_queue.push, the code that mutates the
event dictionaries, to do the shallow copies itself. The only
downside here is process_message_event, a relatively low-traffic code
path, does an extra per-queue dictionary copy. Given that presence,
heartbeat, and message reading events are likely more traffic and
dealing with HTTP is likely much more expensive than a dictionary
copy, this probably doesn't matter performance-wise.
(And if profiling later finds it is, there are potential workarounds
like passing a skip_copy argument we can do).
django-phonenumber-field 2.4.0 adds tighter phone number validation
that rejects +12223334444 for having an invalid area code. This was
reverted in 4.0.0, but django-two-factor-auth still requires <3.99.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Due to try-catch deoptimization, Babel strict mode for…of loops run
about 5× slower in Firefox than Babel loose mode for…of, native
for…of, or forEach (which are all about the same speed). Chrome
doesn’t seem to care.
For some reason we need to explicitly add the core-js Symbol polyfill
near the beginning of the common bundle. Otherwise it gets loaded at
the wrong time and the Casper tests fail.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Babel strict generates more code for [...x] than you’d like, while
Babel loose mode assumes x is an array.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We had a plan at some point to use this to display a phone icon or
something for users who would receive push notifications if you
messaged them. IT's not clear that feature was a good idea in any
case, but it certainly shouldn't be synced as presence data; it would
change >100x less often than the rest of presence and so should likely
be synced differently, maybe as a property on user. So it's best to
delete this prototype.
The “Smileys & People” category has been split into “Smilys & Emotion”
and “People & Body”.
Also, fix generate_sha1sum_emoji to read the emoji-datasource-google
version from yarn.lock, since package.json only gives a version range.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
When quoting a message with fenced code blocks without a language,
we used to have ambiguity in which '```' fence terminates the quote.
This commit adds explicitly non-interfering fences, which fixes the
above issue as well as makes the raw message easier to quickly read.
Fixes#12446.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.
It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.
This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.
With many tweaks by tabbott to documentation under /help, etc.
Fixes#13616.
My upstream remote is named origin, so commit-message-lint was always
complaining at me. Detect the right remote name from the output of
`git remote -v`.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This fixes the buggy behavior for streams which inherits the notification
setting from UserProfile, and are actively opened in "Streams > Stream
settings", if a user has opened two browser windows, and changes the
notification setting from "Settings > Notifications", then the changes
don't reflect such "Streams > Stream settings" notification setting
checkboxes for such stream.
Partially fixes: #12304.
Here we have attached our handler to `.sub_setting_checkbox` so
`e.currentTarget` will return element with class `.sub_setting_checkbox`
but `e.target` will return exactly which element we have clicked, which
could be a child of `.sub_setting_checkbox`. So instead of,
```
$(e.target).closest(".sub_setting_checkbox")
```
we can use
```
$(e.currentTarget)
```
which is more clean and intuitive.
- `e.currentTarget` is less popular which could be the reason behind using
two step hack to get the targetted element.
Rather than defining two different jquery event-handlers for two different
events, we can use a single jquery handler as the function is the same for
both handlers.
https://github.com/Bouke/django-two-factor-auth/issues/297
This setting was added in 1.9 version of the app and can be used
harmleslly in our current Django 1.11-based code and will prevent an
error on Django 2.1+ when we move there.
This is required for our migration to Django 2.2. authenticate()
definitions need to have that starting with Django 2.1.
rate_limit_auth needs to be adjusted to expect the request in the first
positional argument instead of a kwarg.
Since it took a lot of effort to debug the original issue that caused
us to introduce suspect_offline, it seems worth writing a comment
explaining why we won't see that issue here.
We now use user_ids for presence, so we don't need
to worry about races related to unknown emails
being sent to us. Now we just update the data
structure based on user_id, and
it will be there when we render the presence
widget for that user_id, or else it will
simply be ignored.
It's not clear to me whether we still need
dont_block here, so I didn't touch that code.
Here is the commit that added the suspect_offline
flag, for easy reference:
f207450cdb
This flag affects page_params and the
payload you get back from POSTs to this
url:
users/me/presence
The flag does not yet affect the
presence events that get sent to a
client.
If you look at info_for, it clearly never returns
`undefined`, so this defensive code isn't preventing
any bugs.
Also, we are doing a better job now of filtering
user_ids in upstream code.
This is defensive code for the scenario that we
have a user_id in presence but not people. This is
unlikely to occur by the time that we actually render
the buddy list, which is the context for this code.
We have previously been reporting an error here via
the people code, but we add an additional warning.
Also, we filter the user_id from the result.
This reverts commit d84646f091 (which
incorrectly assumed in unread_topic_counter that the messages were
present in the message store), while fixing the type confusion problem
by using IntDict for stream_id keys.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
Fixes “TypeError: sourceContent.split is not a function” at
blueslip_stacktrace.ts:60 when there’s another error during page load.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In the future, any property which doesn't have any dependent setting can be
added to `simple_dropdown_properties` list, which automates setting the
value of dropdowns on saving.
This should ensure that folks rebasing past this commit from an older
database model get their database rebuilt in the way that will
match the test_subs.py query count of 40.
Earlier, on narrowing the window to some particular sizes,
long stream names used to overlap with the subscribe and view stream
buttons.
The issue was resolved by cutting the stream name short and putting
ellipses at the end. A title was provided to the stream name div so that
the entire stream name would be visible on hovering over it.
Fixes: #13139
stream_count and topic_count in the actual code have been IntDict
since commit 9ba1829243 (#13569).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Fixes type confusion in unread_topic_counter, which uses stream IDs as
keys.
Since unread_topic_counter calls message_store.get now, update the
mocks so that message_store.get knows about our mocked messages.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Add a simple compatibility function for AWX 9.x.x. Before AWX 9.x.x
a "friendly_name" key was sent by default. Afterwards it was removed
from being a default key but we can still more or less determine if
the triggering event was a job from the REST-style URL.
Note: It is also technically possible to add the key back by defining
a custom notification template in AWX/Tower.
Resolves#13295.
This makes the state cleaner for the tests. Tests that want to have rate
limiting set up their own desired rules anyway, and having some
pre-existing ones from the default settings can conflict with the
desired ones.
This applies rate limiting (through a decorator) of authenticate()
functions in the Email and LDAP backends - because those are the ones
where we check user's password.
The limiting is based on the username that the authentication is
attempted for - more than X attempts in Y minutes to a username is not
permitted.
If the limit is exceeded, RateLimited exception will be raised - this
can be either handled in a custom way by the code that calls
authenticate(), or it will be handled by RateLimitMiddleware and return
a json_error as the response.
We will want to raise RateLimited in authenticate() in rate limiting
code - Django's authenticate() mechanism catches PermissionDenied, which
we don't want for RateLimited. We want RateLimited to propagate to our
code that called the authenticate() function.
As more types of rate limiting of requests are added, one request may
end up having various limits applied to it - and the middleware needs to
be able to handle that. We implement that through a set_response_headers
function, which sets the X-RateLimit-* headers in a sensible way based
on all the limits that were applied to the request.
validate_otp_params needs to be moved to backends.py, because as of this
commit it'll be used both there and in views.auth - and import from
views.auth to backends.py causes circular import issue.
This makes get_raw_user_data, which was being imported indirectly
from zerver.lib.events inside zerver/views/users.py, get imported
from zerver.lib.users where it actually is.
While the result of this change doesn't completely do what we need, it
does remove a huge amount of duplicated lists of fields. With a bit
more similar work, we should be able to eliminate a broad category of
potential bugs involving Stream and Subscription objects being
represented inconsistently in the API.
Work towards #13787.
This has the side of effect of making new fields we add to Stream be
automatically included, which will help maintain this code as we
upgrade it.
This commit adds is_web_public, history_public_to_subscribers, and
email_notifications fields to the dictionary.
Tests require adjusting, because the class-based view has an additional
redirect - through /uid/set-password/ and the token is read from the
session. See Django code of PasswordResetConfirmView.
The `notification_settings_null` field of the `client_capabilities`
parameter is, apparently unintentionally, required.
This is mostly harmless. However, if any _future_ fields are made
required, all existing clients using this parameter will break, and it
will be needlessly difficult for new clients to specify new
capabilities in a backwards-compatible way.
Attempt to stave that possibility off with warnings.
(No functional changes.)
This modifies get_cross_realm_dicts in zerver.lib.users to call
format_user_row. This is done to remove current and prevent future
inconsistencies between in the dictionary formats for get_raw_user_data
and get_cross_realm_dicts.
Implementation substantially rewritten by tabbott.
Fixes#13638.
This moves get_cross_realm_dicts (from zerver.lib.actions),
get_raw_user_data and get_custom_profile_field_values (from
zerver.lib.events) to zerver.lib.users.
Only the getter of the is_new_member property is added,
to the UserProfile class. This is done to deduplicate
action of checking whether a user is a new member or not.
Previously the sender was not included in display_recipient when
a private message was locally echoed. This broke the copy conversation
link functionality, if the user try to copy the link immedeatly after
sending the message. This issue is present only during local echo.
This was fixed by including the recipient of the user during
local echo.
Fixes#13547.
Edited the warning to clearly state that most members/most stream members
will be notified on using wildcard mentions, along with the specific
mention (e.g. @ALL, @everyone and @stream).
Did a separate check for all wildcard mentions in util.js and stored the
corresponding mention in wildcard_mention inside compose.js.
Fixes: #13636
This change is in series of de-duplication of code in "Other permission"
section for various dropdowns.
Here rather than using "by_anyone" and "disabled" for the `value` attribute
of options, we use actual numeric values. As a result, we don't need to
manually handle to extract the data to be sent to the backend on saving.
This change is in series of de-duplication of code in "Other permission"
section for various dropdowns.
Here rather than using "by_admins_only" and "by_admins_only" for `value`
attribute of options, we use actual numeric values. This helps in
de-duplicating lot of code which is vulnerable to bugs.
For few settings like `waiting_period_threshold` it makes sense to have the
"value" attribute of option to have a value other than the actual setting
value because multiple settings are depending upon this dropdown, so
handling them in JS code makes more sense. But for many settings (which has
integer values), we have followed a wrong trend over the time of
representing every new dropdown with human-readable values and manually
handling them in JS Code, where it makes more sense to use actual setting
value. The result of which is code has become less concise, sensible and
less likely to be mistaken.
This is a preliminary commit for upcoming change where we will use
"bot_creation_policy_values" like approach for many other settings where
dropdown represents the only single setting of integer type.
This extracts the user_data inner function from get_raw_user_data as a
reusable function. We intend to reuse it for cross-realm user dicts.
A few changes were made while extracting it:
* Renaming the UserProfile argument to acting_user, so we can do loops
over a local user_profile variable.
* Moved it to zerver.lib.users, as that's a more appropriate home for
this function formatting data on users.
* Simplified the calling convention for passing custom profile fields
to reflect the fact that this function processes a single user (and
is expected to be called in a loop).
We now use vdom-ish techniques to track the
list items for the pm list. When we go to update
the list, we only re-render nodes whose data
has changed, with two exceptions:
- Obviously, the first time we do a full render.
- If the keys for the items have changed (i.e.
a new node has come in or the order has changed),
we just re-render the whole list.
If the keys are the same since the last re-render, we
only re-render individual items if their data has
changed.
Most of the new code is in these two modules:
- pm_list_dom.js
- vdom.js
We remove all of the code in pm_list.js that is
related to updating DOM with unread counts.
For presence updates, we are now *never*
re-rendering the whole list, since presence
updates only change individual line items and
don't affect the keys. Instead, we just update
any changed elements in place.
The main thing that makes this all work is the
`update` method in `vdom`, which is totally generic
and essentially does a few simple jobs:
- detect if keys are different
- just render the whole ul as needed
- for items that change, do the appropriate
jQuery to update the item in place
Note that this code seems to play nice with simplebar.
Also, this code continues to use templates to render
the individual list items.
FWIW this code isn't radically different than list_render,
but it's got some key differences:
- There are fewer bells and whistles in this code.
Some of the stuff that list_render does is overkill
for the PM list.
- This code detects data changes.
Note that the vdom scheme is agnostic about templates;
it simply requires the child nodes to provide a render
method. (This is similar to list_render, which is also
technically agnostic about rendering, but which also
does use templates in most cases.)
These fixes are somewhat related to #13605, but we
haven't gotten a solid repro on that issue, and
the scrolling issues there may be orthogonal to the
redraws. But having fewer moving parts here should
help, and we won't get the rug pulled out from under
us on every presence update.
There are two possible extensions to this that are
somewhat overlapping in nature, but can be done
one a time.
* We can do a deeper vdom approach here that
gets us away from templates, and just have
nodes write to an AST. I have this on another
branch, but it might be overkill.
* We can avoid some redraws by detecting where
keys are moving up and down. I'm not completely
sure we need it for the PM list.
If this gets merged, we may want to try similar
things for the stream list, which also does a fairly
complicated mixture of big-hammer re-renders and
surgical updates-in-place (with custom code).
BTW we have 100% line coverage for vdom.js.
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".
At the time, we just needed a name quickly, but it was never a good
name, just a placeholder. This removes that placeholder name from
much of the codebase. A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
These docstrings hadn't been properly updated in years, and bad an
awkward mix of a bad version of the user-facing documentation and
details that are no longer true (e.g. references to "Voyager").
(One important detail is that we have real documentation for this
system now).
Closes#13736.
zerver.lib.server_initialization.create_internal has precisely the same
code (you can copy-and-paste swap them, with one level of indentation
adjustment, without generating any diff) so they can be trivially
deduplicated.
zerver.lib.server_initialization.create_users has precisely the same
code (you can copy-and-paste swap them without generating any diff) so
they can be trivially deduplicated.
This doesn't change any behavior, the purpose of this is to make the
function identical to what we have in server_initialization.py so that
it can be deduplicated in follow-up commits.
We mostly needed this for Casper tests, and that
usage was eliminated in the prior commit.
There was also some strange defensive code from
ecc42bc9f8 that
is really ancient and which I am eliminating:
const email = row.attr("data-email");
if ($("#deactivation_user_modal .email").html() !== email) {
blueslip.error("User deactivation canceled due to non-matching fields.");
ui_report.message(i18n.t("Deactivation encountered an error. Please reload and try again."),
$("#home-error"), 'alert-error');
}
If the code was there to protect against live
updates for email changes, then we no longer
have to worry about that, since we use user_ids
now as keys.
Or it might have to do with some ancient bug
where you could pop open two modals at once
or something. You can actually change users while
the modal is open (which is kinda strange, but ok),
and it works fine.
When testing this, I ran into the glitch that we
don't open redraw the Deactivated Users panel after
going into the User panel and deactivating a user.
In addition to making our schema check stricter, it also makes it
possible for us to extend check_events_dict to do additional
validation that's only expected for the full event object.
Now that we have the type situation of having anchor support passing a
string, this is a much more natural way to implement
use_first_unread_anchor.
We still support the old interface to avoid breaking compatibility
with legacy versions of the mobile apps.
A wart that has long been present inin Zulip's get_messages API is how
to request "the latest messages" in the API. Previously, the
recommendation was basically to pass anchor=10000000000000000 (for an
appropriately huge number). An accident of the server's implementation
meant that specific number of 0s was actually important to avoid a
buggy (or at least wasteful) value of found_newest=False if the query
had specified num_after=0 (since we didn't check).
This was the cause of the mobile issue
https://github.com/zulip/zulip-mobile/issues/3654.
The solution is to allow passing a special value of anchor='newest',
basically a special string-type value that the server can interpret as
meaning the user precisely just wants the most recent messages. We
also add an analogous anchor='oldest' or similar to avoid folks
needing to write a somewhat ugly anchor=0 for fetching the very first
messages.
We may want to also replace the use_first_unread_anchor argument to be
a "first_unread" value for the anchor parameter.
While it's not always ideal to make a value have a variable type like
this, in this case it seems like a really clean way to express the
idea of what the user is asking for in the API.
This flag allows rendering as a single isolated page, without the
navigation in header and footer that otherwise provides links to the
rest of the site.
The portico layout, including the styling of the "hero" area at top,
all remains the same.
We don't yet ever set this flag; that'll come next.
This makes the code more readable, by just passing the anchor through
without changing its field name back and forth.
There's no reason for this parameter to involve parsing and integer --
it should be a number in all incoming code paths.
This fixes a bug where that clients using the legacy approach of a
"very large anchor" value with the intent to only get the most recent
messages would only get found_newest=True if they used the specific
value LARGER_THAN_MAX_MESSAGE_ID. Now any value at least that large
will work.
In upcoming commits, we plan to replace this with passing the string
"last", but it seems worth removing the buggy "special value" behavior
while we're touching this code.
This is required for the upcoming type behavior of the "anchor"
parameter.
This change is the minimal work required to have our OpenAPI code not
fail when checking a union-type value of this form. We'll likely want
to, in the future, do something nicer, but it'd require more extensive
infrastructure for parsing of OpenAPI data that it's worth with our
current approach (we may want to switch to using a library).
The proximal issue here is that in upcoming commits, we're going to
change the type of the `anchor` field in `get_messages_backend` to
support passing either an integer or a string.
Many of our tests using POSTRequestMock currently define a query
object that uses integer values for the integer fields we're going to
pass into it, e.g. {'num_after': 0}. That is the correct type for
that field in the Zulip API, before HTTP encoding turns it into a
string. However, because POSTRequestMock didn't use HTTP encoding at
all (which will convert the 0 into a '0'), it ended up passing an
integer to a function that can't possible receive one as an argument.
Ideally, we'd just get rid of POSTRequestMock, since it's a hack, and
just do real HTTP requests instead.
But since it's used in a lot of places making doing so somewhat
impractical, we can get past this issue by just making POSTRequestMock
convert integers to strings.
The feature is used for editing stream descriptions as well, and in
any case, what's important is that it's a content-editable widget (aka
a form of input box).
This fix recently went on master, although it
hasn't actually been deployed yet (not even to czo),
so user impact should be zero:
0fa67c84d8
The fix mostly improved things, but it broke the
logic for pill containers. The symptom was that
if you tried to autocomplete "Cordelia" in the
pill box we'd instead invoke the "c" hotkey and
try to compose to a stream.
Using logging.info() rather than logger.info() meant that our
zulip.soft_deactivation logger configuration (which, in particular,
included not logging to the console) was not active on this log line,
resulting in the `manage.py soft_deactivate_users` cron job sending
emails every time it ran.
Fixes#13750.
A few comments are added to explain more clearly the changes made in
b23a5431cd namely about not using realm
arguments in LoggingCount Stats and the need to pass realm argument in
pull function.
The comments were tweaked by tabbott for readability.
In templates we determine checkboxes are disabled by using the following
`if` clause,
```
{{#if (or (and is_muted notification_setting) realm_setting_disabled)}}
disabled="disabled"
{{/if}}
```
and it is more intuitive to do such calculation in javascript code, so we
added an `if_disabled` attribute in `settings` context which replaces
logical operations from `if` statement.
So for non-notification settings, it is
```
is_disabled: check_realm_setting[setting]
```
where check_realm_setting[setting] is same as realm_setting_disabled.
and for notifiaction settings it is,
```
ret.is_disabled = check_realm_setting[setting] || sub.is_muted;
```
Profiles of typing in the Zulip webapp's compose box after opening the
stream creation widget showed that hotkey.processing_text was a
significant expense. There's no good reason for this -- the function
just needs to inspect the focused element; it just was written with a
sloppy selector.
While there's a secondary issue that, there's no good reason for this
extremely latency-sensitive code path (typing an additional character)
to be doing something extremely inefficient.
We now give "slight smile" precedence over
"small airplane" if you type "sm".
More generally, we favor popularity over prefix
matches for emoji matches, as long as the popular
emoji matches on any of its pieces.
I removed a slightly confusing code comment, which I
will address in a follow up commit. Basically,
"slight smile" still doesn't win over "small airplane"
when you search for "sm", which kind of defeats the
purpose of having popular_emojis for the typeahead
use case. This is a problem with sort_emojis, though,
so when the comment was next to the list of popular
emojis, it wasn't really actionable.
We are sweeping the codebase to use startsWith
when possible. I kept this on a separate
commit due to us vendoring the library, just
to reduce some noise.
Using startsWith is faster than indexOf, especially for long strings
and short prefixes. It's also a lot more readable. The only reason
we weren't using it was when a lot of the code was originally written,
it wasn't available.
We only convert the query to lowercase outside the
loop for an Nx speedup, where N = number of items.
And then we use startsWith instead of indexOf, which
means we don't senselessly search entire strings
for matches.
(We've had startsWith polyfills for a while now.)
Unfortunately, unless a string start with the
exact casing of the query, we still create an
entire lowercase copy of the string for the case
insensitive match. For the English use case
(and many other languages), we could further
optimize this by slicing the string before
converting it to lowercase.
Unfortunately, you have languages like German
with the straße/STRASSE problem. It's not clear
to me how we handle them with the current code,
but I don't want to break that yet.
We use the nice es6 syntax to create the get_item
helpers (in the callers and for the default
value in the function).
Also we use better es6 style for the looping.
In Django 2.0, request.user.is_authenticated stops supporting
`.is_authenticated()` and becomes just a property. In 1.11, it's a
CallableProperty (i.e. can be used either way), and we already use it
as a property in several other places, so we should just switch to
using it consistently now to get it off of our Django 2.x migration
checklist.
Previously, we didn't track opening and closing fences separately,
with led to bugs like not parsing a list that was immediately after
a quoted fence; we treated each ``` as a new fence.
This commit rewrites the function to maintain a stack of currently
open fences. If any of the parent fences is a code fence, we do not
insert a new line before a list.
We also add some test cases specifically to test this behavior with
complexly nested lists.
Fixes#13745.
This extracts get_emoji_matcher and all the
functions it depended on, most of which were
in composebox_typeahead.js.
We also move remove_diacritics out of the people
module.
This is the first major step for #13728.
This change allows the stream names to fit in one line when
we have a long string particularly for notifications stream
with a message 'recieves notifications for new streams'.
Adding invited users to the notifications stream unconditionally isn't
a correct behaviour for guest users, where the previous behavior of
including the notifications stream no longer makes sense. Therefore,
while inviting a new user, the notifications stream is listed along
with other streams with a message "recieves notifications for new
streams" in order to distinguish it from other streams.
Fixes#13645.
We used to put the user's email in a value, which was
redundant (we could find the value from
our parent's label) and brittle (would break
on email changes).
Now the DOM's a bit slimmer and more robust.
Also note that we now deal with user_ids, not emails,
in the call stack until we hit the "edge" and convert
to emails for the server.
This fixes some harmless type errors from the
following commit:
6ec5a1f306
The IntDict code automatically converts strings to
integers, so this was not a user-facing problem, but
we want to have our callers do the conversions
explicitly.
The desktop otp flow (to be added in next commits) will want to generate
one-time tokens for the app that will allow it to obtain an
authenticated session. log_into_subdomain will be the endpoint to pass
the one-time token to. Currently it uses signed data as its input
"tokens", which is not compatible with the otp flow, which requires
simpler (and fixed-length) token. Thus the correct scheme to use is to
store the authenticated data in redis and return a token tied to the
data, which should be passed to the log_into_subdomain endpoint.
In this commit, we replace the "pass signed data around" scheme with the
redis scheme, because there's no point having both.
This field wasn't used for anything, and I think it has very limited
use for debugging, since fundamentally, it'll almost always have a
value within the hour of the actual timestamp in FillState, and any
more fine-grained logging we might want would be available in the
analytics job's own logs.
The proximal reason to remove it is that apparently Django's
model_to_dict doesn't support auto_now fields, and that caused some
trouble when working on adding more complete import/export support for
analytics data.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.
Fixes#11685.
This extracts a function for computing show_invites and
show_add_streams, for better readability and testability.
This commit was substantially cleaned up by tabbott.
This legacy cross-realm bot hasn't been used in several years, as far
as I know. If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.
Fixes#13533.
The alt text of the leading images were displayed as preview
content in inbox by email clients like gmail. Since the leading
images were used mostly for decoration this made the preview
content gibberish. It's fine to set the alt attributes to empty
from accessibility point of view since the old alt attributes
did't added any meaningful information.
This changeset is prepartory work for doing something reasonable with
analytics data during the zulip -> zulip data import process (and
potentially e.g. slack -> Zulip as well).
To support that, we need to make it possible to do our analytics
calculations for a single realm.
We do this while maintaining backwards compatibility and avoiding
massive duplicated code by adding an optional `realm` argument to the
entrypoints to the analytics system, especially process_count_stat.
More work involving restructuring FillState will be required for this
to be actually usable for its intented purpose, but this commit is a
nice checkpoint along the way.
Tweaked by tabbott to adjust comments and disable InstallationCount
updates when a realm argument is specified.
authenticate_remote_user already takes care of calling the authenticate
with the dummy backend. Also, return_data is not used and catching
DoesNotExist exception is not needed, as the dummy backend just returns
None if user isn't found.
Because of how login_or_register_remote_user code is structured, this
doesn't change how the flow will go, but it's not a clean use of
login_or_register_remote_user to call it with is_signup=True if sign up
shouldn't actually happen - and may be fragile when refactoring
login_or_register_remote_user.
In other places where we set request._email, we set it to the
delivery_email, as that's more informative in orgs with hidden email
settings, where user.email will be useless.
When a user clicked the current emoji format in "display settings",
we'd show an infinite loading spinner (basically as a side effect of
trying to tell the server to change the emoji format to what it
already was).
Fix this by aborting early if the emoji format is already the option
that the user clicked.
Fixes#13684.
We now only go the server if both of these
conditions are true:
- our message data seems incomplete for
the stream
- we haven't already fetched history
This function will make more sense when we start
tracking api calls that retrieve topic history.
The unit tests here are kinda duplicating what we
have in the stream_data tests. If we move the
function out of stream_data, we can kill off the
tests there, but for now I think a bit of duplicate
testing is fine here.
All the callers seem to have integer stream_ids
already, either from the message object or
some sub object.
We also use clear() inside the test-only reset()
method.
Previously, links to deleted streams would be incorrectly rendered as
stream's name).
Fixes an issue that was reported where after deleting the "general"
stream, the welcome turtle messages might appear as links to
This updates our error handling of invalid Slack API tokens (and other
networking error handling) to mostly make sense:
* A token that doesn't start with `xoxp-` gives an extended error early.
* An AssertionError for the codebase is correctly declared as such.
* We check for token shape errors before querying the Slack API.
We could still do useful work to raise custom exception classes here.
Thanks to @stavrospat for raising this issue.
This is mostly for tactical reasons. It's hard to
get 100% test coverage on topic_list.js, but it
should be easy to get 100% test coverage on this
very important function.
I considered just moving this code into topic_data.js,
but it just didn't feel quite right. I feel like
this is a pretty core piece of code that's nice
to be by itself and not be near other complicated
code that does stuff like build widgets or talk
to servers. (And, again, it's not just the actual
code here, which is pretty small, it's the unit
tests, which are inherently verbose to exercise
all the edge cases.)
This test mostly tests how we glue everything
together, but I want to change that in an upcoming
commit.
Also, the data stuff that it tests is now better
covered by the test recent tests I added.
There was an edge case with the old
code when you had exactly between 6 and 8
topics and all in cache, with a couple of
the topics being unread.
We would show "more topics" when you were
actually seeing all your possible topics.
To test this:
- create 7 topics on Venice
- as Iago, narrow to any of the Venice
topics
- as Aaron, send unreads to 3 or 4
of the other topics
Eventually Iago will have all possible
topics in the sidebar. On master we'll
show "more topics", whereas after this commit
we correctly avoid that.
It's a pretty harmless bug, since it just
leads to a useless zoom-in.
I have always felt we should zoom-in
regardless of how many topics you have,
just for consistency sake, but I also
understand the rationale behind our
current intentions.
This is basically trying to confine the
rendering logic to a smaller function,
since I want to work toward a better
approach for redrawing the topic list.
Also, since the new function is now
purely data-oriented, it will be a
bit easier to test various edge cases.
If you clicked for no more topics and then the server didn't find any,
we once had code that would say "No more topics" in light gray at the
bottom of the topic list.
The feature appears to have been broken by some detail in the
`self.dom` refactoring. More importantly, it's not clear it's useful
as opposed to clutter.
Since we added the `stream.first_message_id` feature, it's now very
rare for the `more topics` option to appear when there aren't in fact
older topics that could be fetched. In cases where there are not, the
UI is still clear about what's happening -- it shows a loading
indicator and then displays a list of topics that doesn't have
anything new.
So we're removing this feature; we can re-add it without too much
difficulty if user feedback in the future suggests it would be useful
after all.
The only place we ever set active-sub-filter is
right after we build the template, so there is
no reason to have it be a separate step.
(I made a similar fix to pm_list recently, and
this helps set the stage for doing vdom-like
stuff.)
The previous logic was a bit byzantine, making a lot of inferences
based on which conditionals had already been processed that made it
hard to read. This simple function approach promises to be more
readable.
This is for consistency with how we show unreads in muted topics at
the stream level, avoiding distracting users with the appearance of
unread messages in muted topics that they've made clear they are not
interested in.
Arguably, we should show a faded count if there are unreads on muted
topics (but none on unmuted topics), but that seems somewhat complex
to maintain, and we'd benefit from user feedback to make an effective
decision on whether it'd be an improvement.
Fixes#13676.
I think this probably matches users' expected behavior that muted
streams shouldn't get in their way unless the user is actively looking
for them. If a user has a lot of muted topics with active traffic
(e.g. because topics corresponding to channels in a mirrored Slack
instance), they would previously find their 5 slots cluttered with
those muted topics even if there were unmuted topics with unread
messages.
Fixes#13677.
responses is an module analogous to httpretty for mocking external
URLs, with a very similar interface (potentially cleaner in that it
makes use of context managers).
The most important (in the moment) problem with httpretty is that it
breaks the ability to use redis in parts of code where httpretty is
enabled. From more research, the module in general has tendency to
have various troublesome bugs with breaking URLs that it shouldn't be
affecting, caused by it working at the socket interface layer. While
those issues could be fixed, responses seems to be less buggy (based
on both third-party reports like ckan/ckan#4755 and our own experience
in removing workarounds for bugs in httpretty) and is more actively
maintained.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.
I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.
Fixes#13655.
If an email is sent with the .prefer-html option, but it has no html
body, it's better to fall back to plaintext content instead of treating
it as a user error.
Closes#13484.
These options tell zulip whether to prefer the plaintext or html version
of the email message. prefer-text is the default behavior, so including
the option doesn't change anything as of now, but we're adding it to
prepare to potentially change the default behavior in the future.
As we add more address options, which will have different behavior than
simply setting option_name=True, we need to migrate this subsystem to
something that better supports more complex logic and will allow
encapsulating it, instead of needing to be put all over the
decode_email_address function.
This essentially unused legacy variable was causing Zulip to query the
database at import time, which is generally not something we aim to
do.
Combined with the issue fixed in the previous commit, this variable
resulted in test-backend providing an unhelpful crash when provision
hadn't updated the unit testing database.
Since the intent of our testing code was clearly to clear this cache
for every test, there's no reason for it to be a module-level global.
This allows us to remove an unnecessary import from test_runner.py,
which in combination with DEFAULT_REALM's definition was causing us to
run models code before running migrations inside test-backend.
(That bug, in turn, caused test-backend's check for whether migrations
needs to be run to happen sadly after trying to access a Realm,
trigger a test-backend crash if the Realm model had changed since the
last provision).
In this commit, we basically match any kinda of jinja2 start tag,
no matter its special kind (eg. jinja2_whitespace_stripped_start)
to any kinda jinja2 end tag (eg. jinja2_whitespace_stripped_end)
Idea is special operators like `-` do not change the meaning of
inline tag and thus matching shouldn't depend upon this.
Due to a known but unfixed bug in the Python standard library’s
urllib.parse module (CVE-2015-2104), a crafted URL could bypass the
validation in the previous patch and still achieve an open redirect.
https://bugs.python.org/issue23505
Switch to using django.utils.http.is_safe_url, which already contains
a workaround for this bug.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We may revisit this in the future, but similar to is:private, the
current Zulip user experience makes users expect that in the
is:mentioned view, they should really be able to mark messages as
read.
Further, the practice use case for not marking them as read is very
low, since it's rare for someone to have so many mentions that
revisiting the mentions view isn't sufficient to see everything that
needs their attention.
Previously, is_exactly() had already been repalced with can_bucket_by().
This commit removes is_exactly() and replaces its usage in our tests
with can_bucket_by().
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
For Manage Streams, when we render the subscriptions
template, a significant amount of time is taken
by the "t" helper.
Obviously for the first call, we expect "t" to be
somewhat expensive, but subsuquent calls should be
fast, but i18next seems to have some overhead.
Also, we can save a tiny bit of overhead (marking it
as a safe string) that comes from our helper.
As an aside, are we sure it's ok to mark translations
as safe strings?
To test before and after, use blueslip.timings before
and after this commit. When I tested with about 300
streams, the difference is pretty striking:
without cache: 100ms
with cache: 20ms
This is particularly interesting, since the subscriptions
templates have long strings for things like the SVG-based
checkmarks, but they're not really the bottleneck.
Unfortunately, this doesn't seem to be a huge win
elsewhere. In some places we don't call "t", but of
course those might change in the future and benefit from
the cache. And in other places we have smart widgets
that avoid rendering all N objects at one (e.g. buddy
list and list_render).
So this might be too big a hammer to speed up one
screen (albeit a really slow one). It's possible
that we should simply move the i18n.t step **outside**
of certain templates to avoid doing them in a loop.
This new function Casper testing function improperly used
`casper.then` in a nested fashion rather than in series, which doesn't
work how one expects. This likely caused the test flakes we've
started seeing with this code path since adding
submit_notifications_stream_settings (though it's hard to prove).
It's theoretically possible to have configured a Zulip server where
the system bots live in the same realm as normal users (and may have
in fact been the default in early Zulip releases? Unclear.). We
should handle these without the migration intended to clean up naming
for the system bot realm crashing.
Fixes#13660.
We now incorporate people.get_message_people() in our
logic for compose/PM typeaheads. This not only gives
users better results in some cases, but it will also
improve performance for large realms in some cases.
We'll use this in two places coming up, so it's
worth extracting, plus I wanted to add the
fairly lengthy comment here. (Tim, feel free
to edit down the comment as you see fit).
A recent commit removed test coverage for the
actual filtering/sorting of mention typeaheads
when you did a non-silent method. This commit
now tests that important step again.
Note that we also had (and still have) tests
that make sure the is_silent flag is set
correctly by get_candidates.
We don't have a true full-stack test, but those
can be quite tricky to set up and maintain.
This is relatively unobtrusive, and we don't send
anything to the server.
But any user can now enter blueslip.timings in the
console to see a map of how long things take in
milliseconds. We only record one timing per
event label (i.e. the most recent).
It's pretty easy to test this by just clicking
around. For 300 users/streams most things are
fast except for:
- initialize_everything
- manage streams (render_subscriptions)
Both do lots of nontrivial work, although
"manage streams" is a bit surprising, since
we're only measuring how long to build the
HTML from the templates (whereas the real
time is probably browser rendering costs).
This change sets us up to optimize how we
filter users in the admin user settings.
See #13554 for more context on the user
facing issues.
This fix is basically three related things:
- Add filterer options to list_render.
- Add helper method to people.js.
- Use filterer in settings_users.js.
The filter "callback" was only a "callback" in the
most general sense of the word.
It's just a filter predicate that returns a bool.
This is to prepare for another filtering option,
where the caller can filter the whole list
themselves. I haven't figured out what I will name
the new option yet, but I know I want to make the
two options have specific names.
We are already providing callbacks everywhere, so
it would be nice to eliminate some dead code.
This also speeds things up ever so slightly (no
longer type-checking the option every time through
the loop).
We also split out exports.filter to make unit testing
easier. The function seems kinda silly now, being so
small, but I hope to add another filtering option soon.
It's a bit confusing when you read this code to know
where the original list was created. I'm not a huge
fan of the cache scheme here, but it does seem to
work for live updates.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Currently, if we change stream we see the immediate saving of stream, but
it is more convenient to have "Save" and "Discard" buttons as we use
everywhere else in the organization setting subsystem.
This is a preliminary commit for further commits where we will be using the
newly created function `save_discard_widget_status_handler` in click
handler for changing the notification stream.
This refactors `discard_property_element_changes` and
`check_property_changed` function to move conditional statements of
properties that need to be handled separately. It's a preliminary commit in
the series of using save/discard widget for notification stream setting.
As the part of making notification stream settings to change using
"save/discard" widget instead of immediate saving, we need to access the
stream id which is being selected at the moment.
(This is another preliminary commit in the direction of having
"save/discard" widget show up rather than saving immediately.)
The code for selecting and processing the stream for both types of
notifications is almost the same, so de-duplicated.
For "New stream notifications" and "New user notifications" it is more
intuitive to just use the new system for showing success/saving status
feedback.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.
At this point the tool is just a little too noisy
to be helpful. An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS. I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.
I just don't think it's worth maintaining the tool
at this point.
Fixes#13416
We used to search only one level in depth through the MIME structure,
and thus would miss attachments that were nested deeper (which can
happen with some email clients). We can take advantage of message.walk()
to iterate through each MIME part.
return in that loop was a bug, which would lead to the To: header not
being set even though data['recipient'] = str(message['To']) is being
run next, thus requiring the header. We can remove the return
statement and now the loop will overwrite all the potentially
troublesome headers.
With LDAP authentication, we don't currently have a good way to
support the default stream groups feature.
The old behavior was just to assume a user select every default stream
group, which seems wrong; since we didn't prompt the user about these,
we should just ignore the feature.
This fixes the error where we pass `user_id` of 'string' type as the
argument instead of 'integer' to `exports.get_person_from_user_id` which
further passes `user_id` to InDict.has() function which accepts integer
argument only.
Extracting the function makes it a bit easier to
test and use in a generic way.
Also, I wanted this to live in stream_data, so that
it's easier to find if we change how we model
subscriber data.
Finally, I use _.every to do the subset check
instead of `_.difference`, since _.difference
is actually N-squared:
_.difference = restArguments(function(array, rest) {
rest = flatten(rest, true, true);
return _.filter(array, function(value){
return !_.contains(rest, value);
});
});
And we don't actually want to build a list only
to check that it's zero vs. nonzero length.
We now do this, which short circuits as soon
as it finds any key that is only in sub1:
return _.every(sub1.subscribers.keys(), (key) => {
return sub2_set.has(key);
});
First, there are no more convoluted signals.
We also simplify the parameter to just the "mentioned"
object corresponding to either a user or a broadcast
mention.
For the user group scenario, this has always been dead
code, which you only realized when you got to the comment
at the bottom. Now we actually do nothing.
And I moved the relevant commment to the
the typeahead code (with new wording).
I also moved the is_silent check to the caller. I don't
feel too strongly about that either way. It's kind of silly
to call a function only to give that function an additional
responsibility to worry about. On the other hand, I see
the logic of that function enforcing everything. I went
with the former for now.
Arguably we should have a warning for silent mentions,
since doing a silent mention of somebody not on a stream
is a good indication of a typo. I do understand the use
case, but the user can always ignore the warning. Anyway,
we have decent test coverage on this.
This isn't really an extraction; it's more giving
a name to an anonymous function and moving it to
higher module scope.
We convert this to an ordinary function call, which
allows us to move it out of intialize().
Since there's just one simple parameter now (linked_stream),
we can avoid some error checking.
We also avoid the comment that describes the function,
since it now has a name.
And then one minor tweak is to do the inexpensive
`invite_only` higher in the function. This will be
a nice speedup when you link to really large public
streams.
The unit tests are also a bit easier to read now--less
setup and more explicit names.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.
These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
This is easier to read and faster, because it
avoids some unnecessary encoding on the pm-with
part, plus just a lot of extra logic that amounts
to just appending the slug.
Performance for this function is relevant because it is used
for every user every time we rerender the right sidebar.
Previously, if you tried to invite a user whose account had been
deactivated, we didn't provide a clear path forward for reactivating
the users, which was confusing.
We fix this by plumbing through to the frontend the information that
there is an existing user account with that email address in this
organization, but that it's deactivated. For administrators, we
provide a link for how to reactivate the user.
Fixes#8144.
This fixes a similar problem to the last commit; we don't use
memcached with the test database, so we don't need to flush memcached
when rebuilding it.
(And if we try, we'll get exceptions trying to access the relevant
settings).
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.
We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.
We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.
Fixes#6617.
The sort_recipients helper is used for many different
typeaheads, such as compose PMs, compose mentions,
and some settings-related code.
We now avoid unnecessary sorting steps in cases
where we have plenty of results in the top buckets
(such as users who match on prefix).
This change should not have any user-facing
implications.
This method is a bit complex, but I think it's
worthwhile to force PM autocompletes and mention
autocompletes through the same code path.
We also kill off this method:
typeahead_helper.sort_people_and_user_groups
This fixes some regressions from a recent
commit that might not have been deployed.
9f7be51ce8
Even if that change had been deployed, it
should not have been user-facing, but it
would have spammed us with blueslip errors
every time somebody used the stream/topic
popovers in the left sidebar.
There's no reason any more to have a single function
filter both persons and groups. Instead, we just
filter each cohort with a more direct function.
This is a minor performance speedup (avoiding the
conditional in the loop), but I mostly wanted to
simplify the code.
The sort_people_and_user_groups function's only
value-add over sort_recipients is to split out
groups and users, but the caller already had
them split out, so it was kinda silly to concatenate
them back together.
I doubt this was a dumb decision at the time; I think
it was probably a consequence of how bootstrap's
normal approach is kinda inflexible when you're
using typeahead to pull data from multiple sources.
I wanted to kill off sort_people_and_user_groups
completely, but the mention/silent_mention autocompletes
are still a bit awkward to refactor.
For historical reasons pm_list was handling just
one possible edge case of where is:private was
combined with other search terms, namely the
pm-with operator.
The code was correct in realizing the is:private
was redundant there, but now we handle that
upstream in Filter.fix_operators (see previous
commit).
Now we just look for any is:private term.
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.
(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
We should take adventage of the recipient field being denormalized into
the Stream model. We don't need to make queries to figure out a stream's
recipient id, so we take advantage of that to eliminate some of
those redundant queries and simplify StreamRecipientMap.
This change makes these two functions more alike:
- get_search_result
- get_search_result_legacy
To test the UI modify zerver/views/home.py by
replacing `settings.SEARCH_PILLS_ENABLED` with
`True`. I only did a quick sanity check, since
any bugs with the new system are more likely due
to bitrot than any changes I have made here.
The history is this:
Tim cloned the code (before the smaller
helpers were extracted):
db4f6e278f
In 8b153f6452
Shubham removed get_operator_subset_suggestions but
accidentally left a `concat` statement in that got
misapplied to the previous suggestions:
- suggestions = get_operator_subset_suggestions(operators);
result = result.concat(suggestions);
The error there was carried over in some recent changes,
but this commit fixes that strangeness.
In 73e4f3b3fa
Shubham made this change, which makes sense only for
pills, and this code remains intact.
- if (operators.length > 0) {
- last = operators.slice(-1)[0];
+ if (query_operators.length > 0) {
+ last = query_operators.slice(-1)[0];
+ } else {
+ // If query_operators = [] then last will remain
+ // {operator: '', operand: '', negated: false}; from above.
+ // `last` has not yet been added to operators/query_operators.
+ // The code below adds last to operators/query_operators
+ operators.push(last);
+ query_operators.push(last);
}
Mohit made a couple changes to both old and new.
Anders made a couple non-substantive changes related to
the ES6 migration.
Steve (me) made several structural changes to the code. For
some of them I only changed the legacy code, not the pills
code. I didn't fix Shubham's mistake until this change.
Now the two functions should look similar except in the places
where they are intentionally different. I also added a comment
explaining the get_operator_subset_suggestions difference.
Fixes#13609
There may be a deeper issue that various JavaScript logic expects
every message to have a `.message_content` element, but we definitely
should have the `.rendered_markdown` class on any markdown content.
Fixes#13634.
The reason we use functions here will be clear
in the next change.
This is a "prefactor" commit that doesn't change
any user-facing behavior nor significantly change
performance.
A few reasons to extract it:
- we can shorten lines (and not repeat query
every time)
- we can scope the big block comment explaining
why util.prefix_sort is a strange name
- the name is better (it's an O(N) operation that
mostly partitions)
- we may want to swap it out with a true partition
function that's truly a partition (since the
case checks done by prefix_sort are possibly
either a non-feature or mostly overridden by
the other sorts)
This removes zerver/webhooks/trello/view/exceptions.py, which
contained legacy Trello webhook exception related classes. We replace
them with UnexpectedWebhookEventType, which results in our standard
exception handling for unknown event types running (avoiding too-high
priority error logging).
Fixes#13467.
Resolves these warnings from puppet-lint.
puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 14
puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 19
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This improves the approach of creating multiple parallel processes by
using subprocess.Popen() instead of run_parallel() and
subprocess.call() while exporting an organization's message
history. This prevents forking twice for individual subprocess.
While this has some performance benefit, the main reason to fix this
is that it fixes an issue with the data export web UI introduced in
run_parallel forks exited).
Fixes#12904.
process_missed_message did nothing other than calling
send_to_missed_message_address with the same arguments, so there's no
reason to have these as separate functions.
Addresses point 1 of #13533.
MissedMessageEmailAddress objects get tied to the specific that was
missed by the user. A useful benefit of that is that email message sent
to that address will handle topic changes - if the message that was
missed gets its topic changed, the email response will get posted under
the new topic, while in the old model it would get posted under the
old topic, which could potentially be confusing.
Migrating redis data to this new model is a bit tricky, so the migration
code has comments explaining some of the compromises made there, and
test_migrations.py tests handling of the various possible cases that
could arise.
Preparatory commit for making the email mirror use the database instead
of redis for missed message addresses.
This model will represent missed message email addresses, which
currently have their data stored in redis.
The redis data will be converted and migrated into these models and
the email mirror will start using them in the main commit.
For cross realm bots, explicitly set bot_owner_id
to None. This makes it clear that the cross realm
bots have no owner, whereas before it could be
misdiagnosed as the server forgetting to set the
field.
The recip.id || recip.user_id idiom has only been
needed for some old unit tests.
It was previously required as a bad workaround for the
local echo issue fixed in dd1a6a97bd
where we would get `display_recipient` values added in an invalid format.
I want to be able to easily test this without
having to simulate all the jQuery side effects.
This simply preserves the old logic, which seems
to handle one edge case without handling every
possible edge case. The edge cases aren't super
important here, though, since the only thing it affects
is bolding "Private Messages", and when to do that
is somewhat up to personal tastes.
Having said that, we could definitely improve
this code and possibly should move some of this
logic to either narrow_state.js or filter.js.
Instead of doing various ad-hoc calculations of
which PM is "active" and plumbing it through various
functions and then updating it via jQuery instead of
just the template, we now just calculate `is_active`
in `_build_private_messages_list` with a little
helper function.
This test mostly tests logic that I'm about
to remove in subsequent commits, and it's a bit
messy.
This commit removes 100% line coverage, but I
will restore that a few commits later.
In 3cfc3ca24b I removed
the feature that limited PM conversations to five or
less (including the active conversation), but I
didn't clean up this parameter. I think lint was
confused by the fact that we did mutate it.
I am wondering if this started out as an experiment
and was never fully polished before the push? Or
maybe I was just careless. Anyway, I don't
think were any symptoms here--it was just dead code
that we didn't need.
This fixes a rebase issue between the int_dict introduction and use
for people.js with the introduce of filter_values on dict.js and use
inside people.js.
Note that we haven't fully swept this for Dict,
since some dicts are keyed by strings. For
example PM counts can have a huddle like
"101,102,103" as a key.
This should be slightly more performant, and we
often call this function N times, such as when
rendering the buddy list.
There's a minor change to pm_list to avoid
an unnecessary computation on huddles that would
otherwise trigger a blueslip warning for the
huddles case.
Once we get past the special check for fake
person objects already having `pm_recipient_count`,
we can rely on the object being a `person`
object with `user_id` set.
When we are pulling data from message.display_recipient
for private messages, the user_id field is always
called 'id', not 'user_id', so we can simplify
some defensive code.
This required lots of manual testing:
- search/navigate user presence
- send PM and mention user
- pay attention to compose fade
- send stream msg and mention user
- open Private Messages in top-left and click
- test unread counts
- invite user who already has account
- search for users in search bar
- check user settings
- User Groups
- Users
- Deactivated Users
- Bots
- create a bot
- mention user groups
- send group PM then click on lower right
- view/edit/create streams
If there are still pieces of code that don't convert
ids to ints, the code should still work but report
blueslip errors.
I try to mostly convert user_ids to ints in the callers,
since often the callers are dealing with small amounts
of data, like user ids from huddles.
We only ever show 3 or 4 people in search suggestions
(possibly w/a couple variations, like pm-with/sender/etc.),
so we can try to search a smaller subset of people
before going through the entire realm.
We use message_store.user_ids() for this, since you
typically want to search messages for people that
have sent messages recently, and we already sort
based on PM conversations.
This should avoid some memory allocations.
We also use build_person_matcher to avoid
repeating the same logic over and over
again to process the query into termlets.
We also remove people.get_all_persons() and
people.person_matches_query().
This may actually be a slowdown for the worst case
scenario, but it sets us up to be able to easily
short circuit the removal of diacritic characters
for users that have pure ascii names.
For example, czo has lots of names like this:
- Tim Abbott
- Steve Howell
Since they're pure ascii, we can do a one-time
check. A subsequent commit will show how we use
this.
This looks like simple code cleanup, but it's more
than that.
The code cleanup here is that we don't have three
callbacks to get a list of typeaheads for bootstrap.
Instead, we just have one function that does all the
main work.
And then the speedup comes from the fact we no longer
need to remove diacritics from the query for every
time through our loop of seeing if a person matches
the query.
It's a bit subtle to see in the diff, but these are
the relevant lines:
const matcher = exports.get_person_or_user_group_matcher(query);
const filtered_results = _.filter(people_and_groups, matcher);
Before this, bootstrap was doing $.grep, and we'd have
to reinitialize the matcher for every person.
If you profile this before and after, you'll see that
remove_diacritics gets called fewer times.
To profile this, you want to loads lots of users into
your DB and try to autocomplete "Extra", as in "Extra1 User".
If you try to autocomplete something else, then my patch
won't really help, and `remove_diacritics` will still
show up as expensive. Because it is that expensive a function.
These had to be done in tandem, since they were
both kinda coupled to the function that is now
called query_matches_name_description.
(This commit slightly negatively impacts PM
lookups, but this is addressed in the subsequent
commit, which makes PMs much faster. The impact
is super minimal--it's just an extra function
dispatch.)
This may seem silly now, since we are returning a function
that still dispatches over all flavors of search for
every item, but subsequent commits will make it obvious
why I'm doing this.
We want to do our own matching of items, rather than
just giving a callback to bootstrap, which does $.grep
on all the items.
Doing our own matching gives us flexibility for future
improvements like custom data structures for searching
through big amounts of data. Even in the short term
we can speed up searches by pulling expensive operations
outside the grep/filter call.
This architecture has been in place for our search
bar since ~2014.
The benchmark is commented out. It takes only a few
milliseconds to run, so there may be no reason not
to always run it. It doesn't test correctness, so
it would arguably inflate line coverage, but set/get
are obviously covered elsewhere.
We now require the actual tests to explicitly
to zrequire Dict, rather than magically adding this.
In one case, the use of Dict was clearly just for
the test (not the app), so I converted that an ordinary
JS object (see timerender.js).
We have ~5 years of proof that we'll probably never
extend Dict with more options.
Breaking the classes into makes both a little faster
(no options to check), and we remove some options
in FoldDict that are never used (from/from_array).
A possible next step is to fine-tune the Dict to use
Map internally.
Note that the TypeScript types for FoldDict are now
more specific (requiring string keys). Of course,
this isn't really enforced until we convert other
modules to TS.
Model classes fetched through apps.get_model don't get methods or class
attributes. It's not feasible to add them to all these objects in
use_db_models, but Recipient.PERSONAL etc. are worth setting, since
doing that increases the range of functions that can successfully be
imported and called in test_migrations.py.
These tests had a lot of very repetetive, identical mocking, in some
tests without even doing anything with the mocks. It's cleaner to put
the mock in the one relevant, common place for all the tests that need
it, and remove it from tests who had no use for the mocking.
Fixes#13504.
This commit is purely an improvement in error handling.
We used to not do any validation on keys before passing them to
memcached, which meant for invalid keys, memcached's own key
validation would throw an exception. Unfortunately, the resulting
error messages are super hard to read; the traceback structure doesn't
even show where the call into memcached happened.
In this commit we add validation to all the basic cache_* functions, and
appropriate handling in their callers.
We also add a lot of tests for the new behavior, which has the nice
effect of giving us decent coverage of all these core caching
functions which previously had been primarily tested manually.
These are leftovers from where we had default settings in the
settings.py file. Now that the files are separate those references to
"below" are not correct.
If ldap sync is run while ldap is misconfigured, it can end up causing
troublesome deactivations due to not finding users in ldap -
deactivating all users, or deactivating all administrators of a realm,
which then will require manual intervention to reactivate at least one
admin in django shell.
This change prevents such potential troublesome situations which are
overwhelmingly likely to be unintentional. If intentional, --force
option can be used to remove the protection.
We had a potentially nasty bug where we
weren't guaranteeing that all/stream/everyone
collated in consistent ways inside of
`compare_people_for_relevance`, which can
send certain types of sort algorithms into
an infinite loop. I doubt this ever happened
in practice, but it's obviously worth fixing.
Now we also have a clear tiebreaker between
any two all/everyone/stream mentions, which
is the idx field.
Finally, this should be a bit more efficient.
We don't have people named "all". Instead, we
create pseudo person objects with email/full_name
of "all" (along with some other fields). The tests
now reflect this.
This name was misleading, since this code is used
in sort_recipients, which happens when you, for
example, autocomplete persons in the "To:" box
when composing (and has nothing to do with
mentioning).
This makes it a bit easier to find common patterns,
plus it sets us up to pull the calls even further
up the stack.
The first rule of dealing with user data is sanitize
at the edges, not deep down in some function that
has many callers. Putting this code so deep down
in the stack means it's more likely to be called in
a loop.
This moves clean_query into all the callers
of query_matches_source_attrs.
This doesn't change anything performance-wise,
but it sets up future commits.
This change is easy--we only had one caller.
This change means any query going against a
target with multiple `match_attrs`, such as
user names (first name, last) only has to
clean the query once per person.
We want to mostly deprecate this function (see
the comment I added), so I gave it a more specific
name.
Ideally I'd just fix `stream_create`, but it does
use this function in a couple places, and it's helpful
to reuse the same sort here. In one place stream_create
actually unshifts the "me" user back to the top of the
list, which makes sense for its use case.
This change should prevent test flakes, plus
it's more deterministic behavior for clients,
who will generally comma-join the ids into
a key for their internal data structures.
I was able to verify test coverage on this
by making the sort reversed, which would
cause test_huddle_send_message_events to
fail.
If two user_ids in a recent huddle have ids
that sort lexically differently than numerically,
such as 7 and 66, then we were creating two
different buckets in pm_conversations.
This regression was introduced in
263ac0eb45 on
November 21, 2019.
Instead of having our callers pass in a possibly
non-canonical version of a user_ids_string, just
have them pass in a list.
The next commit will canonicalize the sort.
The server may send us ids in the order
[11, 2], instead of [2, 11]. We don't want
to rely on server behavior, regardless, for
the sort.
Our tests now show we process that data.
The current code is is still buggy and causes
us to show the same huddle two different times
for situations where the lexical sort doesn't
match the numerical sort.
This happens on czo often, where Tim is user
7, and his id sorts lexically after ids like
58, 622, 4444, etc.
This should be about 4 times faster, saving something like half a
millisecond on each stream of 10000 subscribers.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The only thing get_color() does is look
up a sub:
exports.get_color = function (stream_name) {
const sub = exports.get_sub(stream_name);
if (sub === undefined) {
return stream_color.default_color;
}
return sub.color;
};
So if we have a sub already, there's no point
calling the helper.
Obviously, this isn't a huge deal, but it happens
N times during page load.
This should make any operation on subscribed
streams faster (we won't need to filter out
unsubscribed streams every time).
I started writing this before I realized we
had a bug where we call `subscribed_streams`
in a nested loop.
After fixing the bugs, this is not as much of
a bottleneck, but it's still a speedup in many
important places:
* build left sidebar
* every keystroke in search bar
* first keystroke in making #stream_links
* every keystroke in compose stream box
The streams settings code is kinda complicated.
It does a non-deterministic sort of the "others"
bucket when you add elements to the left panel.
They get hidden, anyway. Our values() call now
puts subscribed streams first. It never guaranteed
order, but putting subscribed streams first is
probably a good behavior for most situations.
This defers O(N*S) operations, where
N = number of streams
S = number of subscribers per stream
In many cases we never do an O(N) operation on
a stream. Exceptions include:
- checking stream links from the compose box
- editing a stream
- adding members to a newly added stream
An operation that used to be O(N)--computing
the number of subscribers--is now O(1), and we
don't even pay O(N) on a one-time basis to
compute it (not counting the cost to build the
array from JSON, but we have to do that).
Calling `set_filter_out_inactives` is expensive, since we
count up the number of subscribed streams, which iterates
through all your streams, creates a new list of subscribed
streams, then counts them.
In my dev setup, I created 700 streams, and this shaved
about 700ms off of the initial call to `build_stream_list`.
If we aren't showing users emails, then we don't
want to use emails in the search.
And if we are showing users emails, we want to
search on the email that's displayed to them.
For admins this will be delivery_email.
For regular users we arguably shouldn't search
on emails either, since it mostly causes confusion,
but this commit just preserves the current
behavior for those users (unless `show_email` is
false).
We want to be able to unit test this value,
since it's conditional on several factors:
- am I an admin?
- can non-admins view emails?
- do we have delivery_email for the user?
I'm mocking show_email in the tests, since the
show_email code is in `settings_org` and
kind of hard to unit test. It's not impossible,
but it's too much for this commit. (Either
we need to extract it out to a nice file or
deal with mocking jQuery. That module is
mostly data-oriented, so it would be nice
to have something like `settings_config` that
is actually pure data.)
This was duplicate code. I'm moving it to people
for pragmatic reasons--it's hard to unit test stuff
in settings_users.js due to all the jQuery.
It's also nice to have all people-related search
code in one place, just for auditing purposes.
It appears c28c3015 caused a regression where we
set `email` to undefined if a user does not have
`delivery_email` set, and this causes filtering
of users to fail for admins doing user settings.
This fixes only one of the issues reported in
issue #13554.
There's probably no easy fix to scrolling taking
long, but I think fixing search will mostly
address that complaint.
The Rust folks seem to agree with me that the
search results are too noisy. If I search for
"s" I get:
* names like Steve (good)
* names like Jesse (noisy)
* anybody with s in their email (super noisy)
Here is the relevant code:
return (
item.full_name.toLowerCase().indexOf(value) >= 0 ||
email.toLowerCase().indexOf(value) >= 0
);
We now can call is_ascii only once per search termlet
when we are filtering multiple persons on the same
query. (This requires the caller to use
`build_person_matcher` outside a loop or before
a `_.filter` call.)
This is not a major speedup, but we do a couple
simple things here:
- trim the query outside the function we
build (that might be called multiple times)
- don't split names before we possibly
early-exit with an email match
This will allow use to change some O(N) behavior
to O(1) where we are performing the same query
on a bunch of people. (Subsequent commits will
actually take advantage of this prefactoring.)
Once we have max_items results, stop trying
to get more items.
This should really help large realms when
you do a search on streams that turns up
more than N streams (where N is about 12).
We won't even bother to find people.
This class gives us more control over attaching
suggestions to our eventual result. The main
thing we do now is remove duplicates as they're
encountered.
This will make sense in the follow up commit,
where we can short circuit actions as soon as
we get enough results.
This has a few benefits:
- we remove some duplicate code
- we can see finalize_results in profiles
It turns out finalize_results is expensive
for some searches. If the search itself doesn't
do a ton of work but returns a lot of results,
we see it in finalize_results. It brings to
attention that we should be truncating items
earlier instead of doing lots of unnecessary
work.
This isn't a huge speedup, but it's an easy
code change.
We remove the two-liner highlight_with_escaping,
which was only called in one place, and when
we inline it into the caller, we can pull the
first line, which builds the regex, out of the
loop.
The code we removed in highlight_with_escaping
is exactly the same code as in
highlight_with_escaping_and_regex.
I actually copy/pasted this code five years
ago and am now removing the duplication. :)
When we're highlighting all the people that show
up in a search from the search bar, we need
to fairly expensively build a regex from the
query:
query = query.toLowerCase();
query = query.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g, '\\$&');
const regex = new RegExp('(^' + query + ')', 'ig');
Even though the final regex is presumably cached, we
still needed to do that `query.replace` for every person.
Even for relatively small numbers of persons, this would
show up in profiles as expensive.
Now we just build the query once by using a pattern
where you call a function outside the loop to build
an inner function that's used in the loop that closes
on the `query` above. The diff probably shows this
better than I explained it here.
Extracting this calculation makes it easier to hack
it when you're trying to load lots of users.
We probably want a slightly more realistic calculation
here for stress testing. And also fewer rows. But
at least now it's a little more clear what it's doing.
QueueProcessingWorker and LoopQueueProcessingWorker are abstract classes
meant to be subclassed by a class that will define its own consume()
or consume_batch() method. ABCs are suited for that and we can tag
consume/consume_batch with the @abstractmethod wrapper which will
prevent subclasses that don't define these methods properly to be
impossible to even instantiate (as opposed to only crashing once
consume() is called). It's also nicely detected by mypy, which will
throw errors such as this on invalid use:
error: Only concrete class can be given where "Type[TestWorker]" is
expected
error: Cannot instantiate abstract class 'TestWorker' with abstract
attribute 'consume'
Due to it being detected by mypy, we can remove the test
test_worker_noconsume which just tested the old version of this -
raising an exception when the unimplemented consume() gets called. Now
it can be handled already on the linter level.
LoopQueueProcessingWorker can handle exceptions inside consume_batch in
a similar manner to how QueueProcessingWorker handles exceptions inside
consume.
Our ldap integration is quite sensitive to misconfigurations, so more
logging is better than less to help debug those issues.
Despite the following docstring on ZulipLDAPException:
"Since this inherits from _LDAPUser.AuthenticationFailed, these will
be caught and logged at debug level inside django-auth-ldap's
authenticate()"
We weren't actually logging anything, because debug level messages were
ignored due to our general logging settings. It is however desirable to
log these errors, as they can prove useful in debugging configuration
problems. The django_auth_ldap logger can get fairly spammy on debug
level, so we delegate ldap logging to a separate file
/var/log/zulip/ldap.log to avoid spamming server.log too much.
Fixes this error after rebooting the host:
$ sudo ./destroy-all -f
zulip-install-bionic-41MM2
lxc-stop: zulip-install-bionic-41MM2: tools/lxc_stop.c: main: 191 zulip-install-bionic-41MM2 is not running
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The host environment variables (especially PATH) should not be allowed
to pollute the test and could interfere with it.
This allows test-install to run on a NixOS host.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
A block of LDAP integration code related to data synchronization did
not correctly handle EMAIL_ADDRESS_VISIBILITY_ADMINS, as it was
accessing .email, not .delivery_email, both for logging and doing the
mapping between email addresses and LDAP users.
Fixes#13539.
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.
It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.
Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions. Thanks to @alfonsrv for the suggestion.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.
Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
Fixes#13528.
The email_auth_enabled check caused all enabled backends to get
initialized, and thus if LDAP was enabled the check_ldap_config()
check would cause an error if LDAP was misconfigured
(for example missing the new settings).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
