When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.
The inconsistent style between these three buttons looked bad.
We have to take some care with the "Starred messages" and "Edit" ones,
to make sure they live-update properly.
Previously, when the user presses 'Enter' within a input
field while keyboard focus in is in the topic edit textbox
it incorrectly opened the dropdown list widget.
This is because the dropdown button had the default type attribute of
'submit' which triggers the click event binded to it. Fix it by
explicitly defining it's type attribute to be a button.
Fixes#18415
Previously, the focus event tends to skip the first
element within the dropdown list when the user presses
the down arrow.
Added a stopPropogation event for keydown handler within
dropdown_list_widget which propoerly binds the focus
event to the dropdown list items.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.
Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This ensures that the `git describe` queries that we run for caching
Zulip's Git version are guaranteed to include recent releases.
This change ensures that we have accurate output even if we're pointed
at a fork of Zulip that never updates its tags.
Additionally, it will make it possible to record the
`git merge-base upstream/master` in future commits.
Note that because we run this code before unpacking the new version,
the pre-upgrade version of this code runs.
As a result, we cannot assume that the upstream repository exists.
These logs were pretty spammy, and there have long been much better
ways to communicate to system administrators that the incoming email
gateway is great, including, most importantly, in the section of the
emails themselves that explains how replying works.
Previously, the "Move topic" option was only displayed to organization
administrators, despite the new setting making who can move topics
between streams configurable.
Making this migration requires us to remove the text on the previous
"Admin actions" divider, and just make it an hrule, to avoid confusing
normal users, while still providing a hint that moving topics is a
non-personal operation.
We also change the ordering for consistency.
This commit adds the dropdown in 'Stream settings' section of organization
permissions page to control who can move messages between streams and
also hides the stream-edit UI in message-edit form accordingly.
Fixes#14499.
This commit extracts the logic for testing user_can_invite_others_to_realm
user_can_create_streams, and user_can_subscribe_other_users to a single
function test_policy and this function test_policy is called passing
different policies and validation functions as arguments.
This helps in removing a lot of duplicated code.
Previously only admins were allowed to move messages between streams
and admins are allowed to post in any stream irresepctive of stream
post policy, so there was no need to check for stream post policy.
But as we now allow other members to also move messages, we need
to check whether the user who is moving the message is allowed
to post to the target stream (i.e. stream to which the messages
are being moved) and thus we allow moving messages only if the
user is allowed to post in target stream.
b7b1ec0aeb made our checks of the response
format stronger, to enforce that the json translates to a valid dict.
However, old client code (zulip_botserver) was using "" as equivalent to
response_not_required - so we need to keep backward-compatibility to not
break things built on it.
This also handles a few other places missed earlier like
narrow headings, beacuse they use the same function.
We already rerender the PM list for events, so there's no
need to do anything special when someone is muted/unmuted.
`people.get_full_names` is now only used in the settings pages
while creating ListWidgets etc, so we add a new test for
it to ensure coverage.
Currently, moving messages between streams is an action limited to
organization administrators. A big part of the motivation for that
restriction was to prevent users from moving messages from a private
stream without shared history as a way to access messages they should
not have access to.
Organization administrators can already just make the stream have
shared history if they want to access its messages, but allowing
non-administrators to move messages between would have
introduced a security bug without this change.
This completes the effort to make it possible to use
bulk_access_message in contexts where there are more than a handful of
messages without creating performance issues.
Directly rendering the default view on pressing `escape` key will lead
to default_view being rendered on the current window hash. So, we set
empty hash to load default view and let hashchange handle it.
This fixes a bug introduced in
59a45d3521.
Long ago, we changed Zulip to inspect server-provided topic history in
our compose and search typeahead modules, but did not trigger a query
to the server to fetch data.
This could result in confusing experiences where someone thought that
a topic doesn't exist that just isn't present in recent history.
Once we merge this, we may want to adjust the messaging in typeahead
to advertise that the first option will create a new topic, since this
change may make it feel more like creating topics is a heavyweight
operation.
Fixes#9857.
We’re getting unexplained exceptions from URL.createObjectURL in
Firefox 88 (#18374); contain them to this function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, the hook to save drafts when closing the compose box was a
focusout on #compose-textarea. This mostly worked, and was essentially
invisible to users (because the animation to notify you about saved
drafts was in the close_compose_ui widget that is hidden), but you'd
end up seeing the tooltip appear (in the wrong place) if you closed
the compose box immediately after sending a message with Tab+Enter.
The correct fix is for the drafts hook to be just before we start
clearing compose box state, inside hide_box.
This was difficult to catch in a development environment, since one
doesn't use that flow with "Enter sends" enabled.
Cleaning up test_realm_domains.RealmDomainTest.test_list_realm_domains,
test_subs.StreamAdminTest.test_private_stream_live_updates,
test_subs.StreamAdminTest.test_realm_admin_can_update_unsub_private_stream
and test_subs.StreamAdminTest.test_non_admin_cannot_access_unsub_private_stream.
We were apparently not using the ListWidget API for replacing the
content of a widget without removing the whole thing from the DOM and
replacing it.
Unless ListWidget has unexpected bugs, this should have the exact same
result as the previous logic, with much a nicer user experience.
We use styling same as that of the number shown for overflow
(e.g "+3") which has a gray background.
To keep these avatars up-to-date, we need to rerender the
recent topics view after receiving a `muted_users` event.
Also update the user documentation to mention this detail.
We deliberately avoid mentioning "recent topics" because
this applies to mobile too.
This seems more consistent with what users would expect; Recent topics
may be a better default view in general, but if a user has picked
another default view, we should use that where the default view is
desired.
This new function optimizes how we fetch subscriptions
for streams. Basically, it excludes most long-term-idle
users from the query.
With 8k users, of which all but 400 are long term idle,
this speeds up get_recipient_info from about 150ms
to 50ms.
Overall this change appears to save a factor of 2-3 in the backend
processing time for sending or editing a message in large, public
streams in chat.zulip.org (at 18K users today).
The default label for empty narrows depends on whether it's a
stream/topic narrow or a PMs narrow.
We leave some default text in compose.hbs for reply label
because it take some time for the js to display the
correct label.
We change the title to be more descriptive and also link to the
overall Zulip changelog, since one can reasonably get to this page
when googling for "Zulip changelog".
The browser seems to autofill the `Name` field of the add-playground
form. Most likely this behavior is a result of value of the `name`
input field being `name`, causing the browser do to something weird
here. This name is now changed to `playground_name`.
If the caller has already fetched the Stream or subscription details
for the user, those can be passed to has_message_access to avoid extra
database queries.
These thresholds are in relationship to the
`autovacuum_freeze_max_age`, *not* the XID wraparound, which happens
at 2^31-1. As such, it is *perfectly normal* that they hit 100%, and
then autovacuum kicks in and brings it back down. The unusual
condition is that PostgreSQL pushes past the point where an autovacuum
would be triggered -- therein lies the XID wraparound danger.
With the `autovacuum_freeze_max_age` set to 2000000000 in
`postgresql.conf`, XID wraparound happens at 107.3%. Set the warning
and error thresholds to below this, but above 100% so this does not
trigger constantly.
The old logic, inline in the compose area, has produced a very weird
effect where the buttons would move to fit the notification, ever
since design changes to use the full bottom row space.
We address this by just using a Tippy tooltip instead.
When the format of the response received from the outgoing webhook
server is invalid (unparsable json, or just wrong format that doesn't
translate into a dictionary etc.), a message with the error is sent to
the bot owner. We should include the actual payload to make reasonable
debugging possible.
In notify_bot_owner we have to move the `if response_content` block to
append the payload to the message whenever it was specified as an
argument to the function. It shouldn't be nested inside
`elif status_code` as before.
This removes a possible window where an installer error could leave
`nvm` in a state where it had prepended the full path to the
newly-installed `npm` to `$PATH`; we would like to avoid `nvm`
fiddling with path whenever possible (ref ebe930ab2c).
Share code between `safe_full_names` and `get_display_full_names`
functions, and rename `safe_full_names` to `get_full_names_for_poll`,
because that was the only place where this was used earlier.
This also has the nice side effect of showing "Muted user" instead
of the muted username in poll results.
This makes it parallel with deliver_scheduled_messages, and clarifies
that it is not used for simply sending outgoing emails (e.g. the
`email_senders` queue).
This also renames the supervisor job to match.
The reason for this flake is it missed clicking on the Set
All button (`.subs_set_all_users`) because it calculated
its position before/during it starts scrolling.
There are two scrolling events caused by typing `ot`,
* First is due to internal focus call before typing
which scrolled down the page to bring input in the view.
* Second is after it typed `ot` the user list is updated to
one value, which caused the modal height to decrease rapidly.
Note: It theory, there is three scrolling event,
First is of course internal focus then, after it typed `o`
the list is updated to two values and one is after `t` the
list is updated to a single value.
But as puppeteer is very quick it directly jumped to the
scrolling event after it typed `ot`.
For a more detailed explanation read,
https://chat.zulip.org/#narrow/stream/43-automated-testing/topic/master.20failing/near/1173996.
This commit just temporarily fixes this flake, it's not the
best approach to use time-based delays as they can't be
robust (e.g If the machine is super slow then time-based
delays can fail.).
Instead of changing the hash to the default_view hash, we render
the default_view directly in case of recent topics and all messages.
This also fixes the bug that user is unable to go back in
browser window history.
We don't want to rely on browser hash to check if RT is visible
because soon we want to display default_view without any hash
and RT may or may not be visible in that hash.
Active stream's hash look like this - `#streams/1/announce`
Since the stream_id is present where typically hash section
is for all other hashes, get_current_hash_section should return
the same result.
The screenshot generating mechanism doesn't work for newrelic and
causes error because its configuration file doesn't exist. This
commit fixes the configuration and re-generate the screenshots.
New bot avatars are generated with this tool. Having the avatars generated,
we can run generate-integration-docs-screenshot to generate the doc
screenshots.
Fixes: #17792
Non-webhook integrations should have fixtures containing mock messages
in json format with fields "subject" and "body" indicating the topic
and content respectively.
Added a commit to bind vim keys with the SettingsPanelMenu,
allowing users to control the settings and organization tabs
using 'h','j','k' and 'l' keys.
vim_left('h') -> Switch between tab by shifting focus to the left.
vim_right('l') -> Switch between tab by shifting focus to the right.
vim_up('k') -> Move up.
vim_down('j') -> Move down.
Also link to it from the API documentation page,
other help pages, and the confirmation dialog for
muting a user.
With substantial edits by tabbott and alya.
* Revert "frontend: Remove hover effect from small messagebox."
This reverts commit 27d9643274.
* Revert "frontend: Use placeholder style for small messagebox text."
This reverts commit 8453aac260.
* Revert "frontend: Make "Reply" button look more like a textbox."
This reverts commit 9fdd7184c6.
A message containing wildcard mention when quoted (which
is turned into a silent mention) or message with silent
wildcard mention notifies the users by sending desktop,
sound, and missed message email notifications. This
is clearly a bug which is fixed by this commit.
Fixes: #18354.
To calculate the max-height of buddy_list_wrapper, we use
invite-link-height. This works alright if inviting is enabled
for all. However for users who aren't permitted to invite,
the element does not render, hence the logic uses `0` as
default value for subtraction in `get_new_heights` function
under resize.js. This leads to the keyboard-shortcut icon
rendered below the browser window. Hence use the parent div
`right-sidebar-shortcuts` height for the logic.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameter.
Bump api feature level and highlight the fix for `emojiset`
parameter of `settings/display` endpoint in zulip.yaml file.
Fixes part of #18035.
* Remove unnecessary json validator for string validator.
* Update frontend to pass right validator.
* Update zulip.yaml to pass right parameter for curl request
in openapi.
* Update python_examples to pass right paramater.
Fixes part of #18035.
Commit ad76df25ac (#18405) uses Map
incorrectly. A JavaScript Map needs to be indexed with .get() and
.set(), not with [].
Also, clean up the API: ‘lookup_table’ is a meaningless variable name,
and there was no information in the array that wasn’t also in the Map.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous `top_offset` calculation didn't include the height
of the panels which led the calculations to be performed as if
a portion which was hidden behind the searchbox, was visible to
the user.
The new formula is corect as the frb_top calculation in
panel.resize_app also uses panels_height.
Note that the documentation cannot fully use our macros, because
Uptime Robot requires an & of the end of the URL, because of how it
passes its payload.
Fixes#13854. Fixes#13939.
A modal is added to edit the realm linkifier which
supports ui_report error.
Puppeteer tests to verify linkifier update and an
invalid test to verify that linkifier error messages
are reported on the modal are added as well.
This commit adds `title` attribute and removes
`aria-hidden` attribute in `Delete` button in
linkifiers table.
`aria-hidden` attribute is used only for icons on buttons
that have a plain-text label.
Implemented dropdown_list_widget in Move topic popover which enables,
the functionality to search for streams while moving a particular topic.
The aim is to reduce the user effort by having an input
typeahead instead of haivng a dropdown list of streams.
Closes#14860.
Previously, the focus event was triggering on a hidden
dropdown input field, which caused it to not gain focus when
clicked on dropdown button.
This is so because focus events cannot be triggered to
hidden elements or elements that aren't visible in DOM.
Added a fix by explicilty triggering the focus event to
dropdown input field only if the input field is visible in the DOM.
Followup of zulip#17926.
Basically, the aim is to concise modal texts
to use 'Confirm' instead of long labels since they can
tricky for translators and can also create bad strings
having long words.
Link to zulip-terminal's README which contains installation
instructions.
It is marked alpha to indicate that user may not find all
the features that zulip support in the client and may
run into unexpected errors.
This is good to do in general for these type of event handlers
and this also fixes a bug where pressing on reply button in
the recent topics will open compose with a new line since
browser assumed enter was pressed on compose_textarea since
process_enter_key returned false.
In 215320bc72, we added the typeahead for
the `pygments_name` field which helped lookup the human-readable
`pretty_name`. However, we forgot to remove the workaround method
mentioned in the docs to lookup the name before the typeahead was
introduced.
Since we support hooking up code playgrounds for custom languages
outside of the list of languages supported by Pygments, we display
an option to select a "Custom language" in the typeahead.
The `pygments_name` field typeahead shows a list of human-readable
`pretty_name`to pick from. However, the Pygment aliases which are
used for code highlighting within a code block are different from
these. This mismatch might be confusing for folks unfamiliar with
technical details of the "code playgrounds" feature.
To bridge the gap, we now include the Pygment aliases within
parenthesis along with its human-readable name, in the typeaheads.
E.g: Python 2.x (py2, python2).
Added a fix to bind vim_up(k) and vim_down(j) keys
with the draft section, allowing users to scroll
between the drafts using 'k' (scroll up) and
'j'(scroll down) keys.
Fixes#18397
Previously, we relied on the browser placeholder text style, but this
makes it impossible to style text to look like a placeholder.
Chrome uses `color` to set the placeholder, while Firefox uses
`opacity`. This commit sets both, since setting one without the other
will lead to strange behaviour.
We pick the color for the light and day themes to ensure that we meet
WCAG standards for accessibility.
Update docs on how to build an incoming webhook integration to be more
accurate on exactly what is needed at this point in time as they were
quite out of date.
Make it easier to locate docs for how to build an integration by adding
a link to the docs at the top of the docs on how to document an
integration.
Tweaked by tabbott to adjust the whole first paragraph and make the
link more contextual.
Added Help link widget within account settings for
deactivate account UI modal which redirects to
/help/deactivate-your-account within the user documentation.
Previously, we used to hide 1:1 PMs with muted users everywhere
in the UI. This commit makes it so that such messages will now be
visible in `pm-with/<muted_user>` narrows, meaning these will not
be excluded from message lists, but will still be hidden under
the "This message was hidden." dialog implemented earlier.
* We hide 1:1 PMs from and to muted users throughout
the UI, because doing so will not lead to loss of
conversational context. The "to" part is also important,
because the last few messages sent to a user before
muting them would probably be asking them to stop
spamming.
* After this change, we will need to do filtering for either
user or topic muting in pretty much all narrows, so we need
to keep the `_all_items` list in MessageListData always
up-to-date.
* A further commit will relax this and make it possible to
view these messages only when in a `pm-with/muted_user`
narrow.
This basically reverts 4bd7ec7c36 and
3a9dfc02e6.
The plan earlier was to have compeletely different codepaths
for user and topic muting, so that we could call seperate
functions in the message list class on receiving the respective
events.
However, this cannot be done, because if we, for example, on
receiving a `muted_users` event, filter `_all_items` based on
just user mutes, and store the result in `_items`, then, that
result may still contain topic-muted messages, which is
undesirable. Hence whenever we filter messages, we must do so
based on both user as well as topic muting.
(The code for the former will be added in further commits.)
So, we will have a single function which will handle updating
the message lists for muting.
These mock objects did not buy any ease in testing, as
evident from the fact that this commit hardly contains
any changes to the tests themselves.
This commit also removes some unnecessary `filter: undefined`,
parameters sent to MessageList constructor.
We were losing scroll position after re-render because there
were not enough rows present in the table to focus back to
the same position. With this commit it is possible now since we
always render with enough rows to do so.
We save the scroll position of the user by keeping the topic
row at the center of the visible scroll container in focus. The
avoids focus being reset to first topic row when recent topics
renders. Thus, this improves the UX of recent topics.
Add a function which is called before every render to
get the number of items it can render. This can be used by
instance to load custom number of items as per its state.
We allow ListWidget instances to pass functions in opts that can
be called after scrolling to determine when to render.
Also, allow a callback function to be called pre render.
We move compose from being a part of message feed to
being a part of middle column which is a common parent of recent
topics and message feed. This allows us to use a common compose
box for both the views. Fortunately, compose actions were
independent of this change so there weren't any evident
side effects.
Fixes#17543
We combine the two loops into one, so that we
can check our flags before creating the
UserMessageList object.
And we lift a few calculations out of the loop.
For 8k users, with 95% long-term-idle, this was
about a 10x speedup for me. (~30ms -> 3ms)
This removes unnecessary json_validator for string parameters in the
BigBlueButton video calls endpoints. Note that this breaks links to
video meetings sent before the upgrade; there's not much we can do
about that.
Since this is the last commit in this series, we update the
ZULIP_FEATURE_LEVEL for this batch of changes.
Fixes part of #18035.
* Remove unnecessary json_validator for string parameters.
* Remove unnecessary JSON encoding in frontend calls. Structurally,
JavaScript does correct encoding without explicit JSON encoding.
Fixes part of #18035.
Remove unnecessary json_validator for string parameters. This change
does not modify JavaScript because we don't have a frontend for these
API endpoints yet.
Fixes part of #18035.
During the upgrade process of a postgresql-only Zulip installation,
(`puppet_classes = zulip::profile::postgresql` in
`/etc/zulip/zulip.conf`) either `scripts/start-server` or
`scripts/stop-server` fail because they try to handle supervisor
services that are not available (e.g. Tornado) since only
`/etc/supervisor/conf.d/zulip/zulip_db.conf` is present and not
`/etc/supervisor/conf.d/zulip/zulip.conf`.
While this wasn't previously supported, it's a pretty reasonable thing
to do, and can be readily supported by just adding a few conditionals.
The previous hashers mirrored the ones used in production, but that was
non-ideal because those are slow. Replacing them with quick hashers is a
performance improvement for those tests.
Raising jsonableError in the authentication form was non-ideal because
it took the user to an ugly page with the returned json.
We also add logging of this rare occurence of the scenario being
handled here.
user_profile.check_password(password) in authenticate of
EmailAuthBackend can raise PasswordTooWeakError; this happens when the
user's password is weaker than the current required policies and needs
to be rehashed (E.g. because, as in Django 3.2, the minimum salt
entropy increased).
This is a very rare case, but still needs a good user-facing error
message. We raise a json error to handle this with a user-facing error
message.
See this comment by Mateusz Mandera for a detailed explanation
about this case along with a traceback it generates.
https://github.com/zulip/zulip/pull/15449#discussion_r448308614
The authenticate function of EmailAuthBackend had request param
type set Optional[HttpRequest] had `None` as default. This
function is never called without a request. So this changes it to
require an HttpRequest parameter.
It was made `Optional` in bc062e1c4d,
because this parameter was new in Django at the time.
We're safe to make it a required argument as everything worked well
before that recent commit and Mateusz Mandera and I checked if it gets
`None` anywhere and found only authenticate of non EmailAuthBackend
gets `None` in some places like `dev_direct_login`.
All the places in tests where this function got `None` as request
were fixed in previous commit.
Matching the full process name (-x without -f) or full command
line (-xf) is less prone to mistakes like matching a random substring
of some other command line or pgrep matching itself.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The `.refresh-failed-message button` was registering clicks even
while the button was spinning (has already been clicked once).
Thus a network request was sent for every subsequent click which
raises an exception that the local id is not found in the message
store as it had already been reifyed by the first request.
Fixes#18375.
Support for the timeouts, and tests for them, was added in
53a8b2ac87 -- though no code could have set them after 31597cf33e.
Add a 10-second default timeout. Observationally, p99 is just about
5s, with everything else being previously being destined to meet the
30s worker timeout; 10s provides a sizable buffer between them.
Fixes#17742.
This corner case was reported in #17320, basically, the
issue was when two or more alert words were used
consecutively with a single space between them, it didn't
detect the even number word as `alert word`.
In this 009b7bca24 commit `before_punctuation`
regex was updated to use lookbehind feature of regex.
This caused a regex error in some browsers (reported in
Safari) because lookbehind feature is not yet supported
on all the browsers (https://caniuse.com/js-regexp-lookbehind).
This commit fixes that error by reverting to stable regex which
works on all the browsers.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Leave the Intel build as the prominent default, since it will run on
both platforms. (I would have liked to detect the appropriate
platform, but Apple seems to have put significant effort into making
that impossible for anti-fingerprinting reasons, which is probably an
overall good.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Due to spaghetti CSS that should be fixed but isn’t fixed here, the
<span> wrapper is still needed so the hover effect is applied.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This button will allow users to avoid a distracting red banner across
their screen, while they wait for their sysadmin to do the upgrade
work.
Fixes: #18359
Since we don't process private messages yet, we don't
need to re-render when we receive a new private message
as it doesn't change any data related to recent_topics.
As we change the icon for Group PMs in #18160, the docs related
to it is now outdated. This commit removes the documentation part
that is outdated from status-and-availability section.
Move `get_setup_webhook_message` to
`zerver/lib/webhooks/common.py` so multiple integrations can use this
rather than just those which import `zerver/lib/webhooks/git.py`. Also
added the documentation for this.
Since the invariant we're trying to protect is that every realm has an
active owner, we should check precisely that.
The root bug here, which the parent commit failed to fix properly, is
that we were doing a "greater than" check when we clearly originally
meant a "less than" check -- lower role numbers have more permissions.
Django's default SMTP implementation can raise various exceptions
when trying to send an email. In order to allow Zulip calling code
to catch fewer exceptions to handle any cause of "email not
sent", we translate most of them into EmailNotDeliveredException.
The non-translated exceptions concern the connection with the
SMTP server. They were not merged with the rest to keep some
details about the nature of these.
Tests are implemented in the test_send_email.py module.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
blocks. Instead of being called a syntax highlighting label, this is
now referred to as a "language tag", since it serves both syntax
highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
Currently users that try to deploy Zulip through docker has errors
because LDAP group search configuration can't be automated.
Reverts a hunk of f5197518a9.
This will allow us to avoid duplication of array filtering
logic of the form-
`Array.filter((user_id) => !muting.is_user_muted(user_id))` and
`Array.filter((person) => !muting.is_user_muted(person.user_id))`
We want to exclude 1:1 PMs from muted users everywhere
except in `pm-with/<muted_user>`.
This method will help us determine whether we are in such
a narrow.
* We show a "Click here to reveal." hyperlink in the hidden
message dialog for user to click on and read a hidden message.
* The "reveal" action is temporary, in the sense that a revealed
message will again be hidden once the broswer tab reloads or
if the user renarrows.
* When a message is revealed, we make sure to show the sender
of that message, even if it isn't the first message of it's group.
This is because the first message of that message group (which
would have otherwise shown the sender) can still be hidden.
* Reactions and background color after revealing a message are
the same as if the message hadn'e been hidden at all in the
first place.
* We hide the sender and reactions on messages sent by muted
users, and replace the content with a "This message was hidden"
dialog.
* Ideally, we should collapse a series of consequetive
messages sent by muted users into one such dialog, but
that could break the cursor behaviour and `near/<message_id`
links, so we as of now show one dialog per muted message.
* Because we hide the sender, there is a chance of the first
hidden message in a group looking like it was sent by the
author of the message above it. To tackle this, we intentionally
make the hidden message dialog float-left, so that it is clear
that this is a special type of message.
* For context, we still show the timestamp of the message.
* Starring, editing, deleting etc a message still work just like
before.
A further commit will add the ability to reveal a
hidden message.
This new pages accomplishes several interrelated things:
* Documents that Zulip Cloud runs master and how that works.
* Documents policies on how long client apps are expected to support
old releases in our compatibility matrix.
* Removes the 3-years-stale roadmap article.
* Provides a central place to talk about different versions in Zulip.
* Provides a better place to link to from our "you need to upgrade" nag.
This content is not intended to be final, but should be finalized in
the next week or so.
Fixes#18322.
We were not considering the owner role in `sort_role` function
which was leading to improper sorting when sorting the users
list according to role. This commit fixes this bug by considering
role in `sort_role` function.
As discussed in the comment, this is a critical scalability
optimization for organizations with thousands of users.
With substantial comment updates by tabbott.
We also add a check to see if `message_feed_container` is visible,
since it is possible that recent topics could be visible which
was not a possibility when this logic was written.
This page is documentation for our participation in GSoD, which was a
few years ago. We can resurrect it if we participate in the program
again, so there's no need to keep it as a live page.
Linked the Help Center document in places like
- zulip.yaml (/events, /register/, realm/playgrounds,
/realm/playgrounds/{playground_id})
- /help/format-your-message-using-markdown (Linked to make
users reading the markdown code block style, aware of this
feature)
- /templates/settings/playground_settings_admin.hbs (Linked
as a reference to read more about playgrounds before
configuring one)
Also showcase the feature on /features and /for/open-source.
The typeahead suggests a human-readable `pretty_name` for the
`language` field in the add-playgrounds form.
The suggestions are sorted based on the popularity of these
languages.
E.g: A `py` prefix would match with `Python` first before
matching with others like `Python 2.x` based on priority.
The UI makes use of an onclick on the trash icon to call
DELETE /realm/playground/{playground_id}. The id is
extracted from the data attribute `data-playground-id`
set on that element.
The design of the form is similar to the linkifiers page
and is styled similarly.
The introduction text for "Code playgrounds" is improved
with more details and examples.
Also, we can remove the hardcoded playground and the fix
we had previously done to prevent breaking the hardcoded
playground.
This was accidentally removed in 27e4f5da92
as the initial idea was to have another parameter `stream_id`
for stream type messages instead of sending it in this `to` parameter.
Also fixed a description of `to` for `stream` type.
This is a prep refactor, instead of creating Confirmation
object and using `confirmation_url` for generating confirmation
link/url, using `create_confirmation_link` would be a cleaner
approach, also this can help us avoid failing test in case
Confirmation model is changed.
Part of #16359.
This commit displays the loading animation and hides other
avatar options as delete_user_avatar is called and as we get
response from the server, we show the options back and hide
the loading animation.
These two blocks of code has incorrectly placed or implemented
conditionals for the rare corner case of Zepyhr mirroring
organizations.
Longer-term, we'll want to refactor this further to directly reference
can_render_subscribers, making the logic more general.
The previous commit remove an obsolete form laying within
subscription_table_body.hbs. However, doing so broke the functionality
of triggering the stream-creation modal using the 'Enter' hotkey.
This commit re-introduces the functionality of loading the stream
creation modal only if the user has the permission to create streams.
At the same time, we make sure it's still possible to subscribe to
streams within zephyr mirroring organizations.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.
Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.
Fixes#15819.
This will stop dropping events in the case that the background
`maybe_send_batched_email` thread takes longer than 30s. However, see
also #15280 and the TODO comment about how we lose events upon
restart; this worker is still lossy.
The 'reply' button shows the stream>topic or recipient(s) of the
selected message, for better UX. It also expands to fill the
remaining horizontal space in the button bar -- this should help make
it easier for new users to figure out how to reply.
Finally, it uses "Message" instead of "Reply", to better match the
compose box.
Fixes#17940.
In puppet, we use pgrep in the collection stage, to see if rabbitmq is
running. Sufficiently bare-bones systems will not have
`procps` (which provides `pgrep`) installed yet, which makes the
install abort when running `puppet` for the first time.
Just installing the `procps` package in Puppet is insufficient,
because the check in the `unless` block runs when Puppet is
determining which resources it needs to instantiate, and in what
order; any package installation has yet to happen. As
`erlang-base` (which provides `epmd`) happens to have a dependency of
`procps`, any system without `pgrep` will also not have `epmd`
installed or running. Regardless, it is safe to run `epmd -daemon`
even if one is already running, as the comment above notes.
Using `pgrep -f epmd` to determine if `empd` is running is a race
condition with itself, since the pgrep is attempting to match the
"full process name" and its own full process name contains "epmd".
This leads to epmd not being started when it should be, which in turn
leads to rabbitmq-server failing to start.
Use the standard trick for this, namely a one-character character
class, to prevent self-matching.
Followup of #17926.
Basically, the aim is to concise modal texts
to use 'Confirm' instead of long labels since they can
tricky for translators and can also create bad strings
having long words.
Earlier, a user can only mute a topic from its recipient bar but can't
unmute it from there (and in fact we displayed an option to mute even
if the topic was already muted!). This commit fixes that bug and
allows a user also to unmute the topic from its recipient bar.
There are two core issues here;
* We did not have code, an icon, etc. for the "already muted" case in
the recipient bar logic at all.
* We did not rerender messages in !excludes_muted_topics views when
muting state changed.
See: 660475bd0c for background on when
we started only rerendering the streams with excludes_muted_topics
after muting changes. Rerendering of newly muted topics are important
for live rendering if a user is narrowed to that topic itself, which
are essentially all excludes_muted_topics narrows anyway.
Hence, now, we rerender by calling the `rerender` function for muted
topics (which is done just before we update the items for muting via
the function: `update_items_for_topic_muting`).
Tweaked by tabbott to add comments explaining the reasoning and
long-term plans.
Fixes#15223.
We are restricting the feedback widget (confirmation pop-up)
to be visible to the users if they have muted the topic using
the hotkey: `M`, because if the users have muted the topic in
some other way then it is known to them about their activity.
In fact, the confirmation pop-up was intrusive to some users
(see #2367).
While using the hotkey they can unknowingly mute the topic. Hence, in
such a case, it is necessary to acknowledge the users about their
action.
This commit modifies the test_wildcard_mention_restrictions test
for checking that moderators are not allowed to send messages
with wildcard mention if wildcard_mention_policy is set to
WILDCARD_MENTION_POLICY_ADMINS. Previously, we were checking
for members, but it is better to check for moderators.
This test was wrong, in that it checked for the English status string
when the current language was expected to be German; this happened to
work until now because with i18next we were failing to include the
string in question in our translation data; and early today I did the
first translation data sync since our translators had translated that
string for German.
Apparently, after upgrading to Django 3.2, mutating is_staff and then
saving can result in a user's session being destroyed.
In any case, this test is probably better written using two different
users with the different roles, which we have in our initial database
anyway.
In Django 3.2 slugify strips trailing dashes and underscores:
0382ecfe02
sanitize_name doesn't so this difference should be documented like the
others.
Underscore character is already covered by \w, so _ in the regex is
redundant. Also the docstring is mildly incorrect - underscore already
is an allowed character by django's slugify (and always was) for the
aforementioned reason.
This is a straightforward upgrade in terms of changes needed.
Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys
- `The default_app_config application configuration variable is deprecated, due
to the now automatic AppConfig discovery.`
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery
To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py. When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."
An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
Since we don't have content for GIPHY popover at the time
popover is rendered, we need to render with some fake
content otherwise popover library doesn't allows for us
to show the popover.
This came into notice after 5adc6d7297
broke show popover behaviour for GIPHY but didn't break it for
emoji popover as emoji popover already renders with some content.
Prior to this commit, popover library used `title` as content for
GIPHY popover.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
Now that we are passing source realm's id instead of string_id in
source realm selector, it makes sense to rename the "source_realm" field
to "source_realm_id".
In the source realm selector, when we select a realm from which we want
to import the data, we pass the source realm's string_id. The problem
with this approach is that the string_id can be an empty string. This
commit makes the source_realm pass the realm's id instead of string_id.
Now, the source_realm's value will either be an integer or "" (empty
string) when we don't want to import settings from any realm.
This commit fixes the following help pages -
- configure-who-can-create-a-stream.md to use "Stream permissions"
subsection and not "Other permissions".
- configure-who-can-invite-to-streams.md to use "Stream permissions"
subsection and not "Other permissions".
- restrict-wildcard-mentions.md to use "Stream permissions" subsection
and not "Organization permissions".
- restrict-bot-creation.md to use "Other permissions" subsection and
not "Organization permissions".
- restrict-visibility-of-email-addresses.md to use "User identity"
subsection and not "Organization permissions".
This commit fixes invite-new-users.md to show all the options of who
can invite under "Are invitations required for joining the organization?"
and makes this page similar to help pages for other policies.
This commit updates restrict-permissions-of-new-members.md
to add "inviting users to organization" in the list of actions
that can be restricted for new members.
We also fix the subsection of waiting_period_threshold to be
"Joining the organization".
* Add mobile app instructions for interacting with them.
* Fix inconsistent headings.
* Document the new state that starred message counts are the deafult,
and mention the "come back to" workflow for them.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell. Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.
Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.
Fixes#18258.
In Debian, becoming root as `su` does not alter the `$PATH`; this can
lead to the root user not having `/usr/sbin` in its path, and thus
the `useradd zulip` step of the installer fails.
Fixes#17441.
Currently only enabled in development, since the exact details don't
seem right..
Co-Author-By: Signior-X <b19188@students.iitmandi.ac.in>
Co-Author-By: Aman Agrawal <amanagr@zulip.com>
Implements UI for #8005.
Instead of taking the "onion" approach, where all services are
stopped, and then started back up again, default to a rolling restart
across all processes. This draws out how long the overall "restart"
takes, but minimizes the time that any of the services are down. This
minimizes user-visible impact and queue buildup.
In cases where speed is more important than minimal impact (for
example, there is already a current outage), a --less-graceful flag is
provided, which brings the services down more suddenly, and back up in
a still-correct order.
Commit dc67870 introduced a user confirmation modal before
deleting profile picture, leaving the user documentation
unchanged.
Added a commit to reflect the newly added change within
the user documentation as well.
This avoids slightly glitchy looking behavior in certain scroll
positions where there just isn't enough space above link to make it
look like it's top of the other element.
Currently we show the PMs list align with the Private
messages header that make them no difference between
header and its elements list and it may confuse the user
to notice that PMs list is expanded or not. We follow
a trend of shifting the child elements slightly towards
right from the parent header element in stream-topic list.
Maintaining this trend we fix this issue similarly, the
PMs list is shifted by 10px towards the right.
Replacing the group PMs icon with "fa fa-group" icon
drops the color class "fraction_present" logic. As there
is no more use of fraction_present class this commit
cleanups its all existence from the codebase.
It is difficult to distinguish group PMs from 1:1 PMs, so to
improve the UI it is better to show different icon for group
PMs. Here we are using fa fa-group icon for group PMs.
Fixes#18069.
The comment mentioned the values in this dict to be consistent
wth values in settings_config.invited_as_values.
But settings_config.invited_as_values was replaced by
settings_config.user_role_values in 136c005f3f, so
updated the comment accordingly.
This commit adds both frontend and backend code to invite a user as
moderator. We allow only existing owners and admins to invite a user
as a moderator.
The function get_role_for_new_user was added to get role from the
invited_as value, as invited_as values were one of (1,2,3,4)
previously, but it was then changed to be the actual role value,
i.e. one of (100, 200, 400, 600), in 1f8f227444.
So, we can safely remove this function now and use invited_as value
directly and handle realm_creation case by using an if condition.
This is a feature of GNU readlink that isn't in the BSD readlink
found on macOS.
For using this and other GNU coreutils features in our scripts in
general, we could use a solution like mobile's tools/lib/ensure-coreutils.sh
to get GNU coreutils on the PATH -- check if it's there already,
if not then try to find a Homebrew install of it and use that, if not
then print a helpful message.
But even then there'd be a bootstrapping problem of how to find
ensure-coreutils.sh . That involves exactly the same problem as we
have for finding git-tools.sh in these lines. So in fact in mobile
for the task of finding ensure-coreutils.sh in the first place, we
do without `readlink -f` anyway.
The one consequence of this behavior-wise is that if you make a
symlink somewhere that points directly at that script (say in your
`~/bin/`), and try to run it using that symlink, it won't work.
(It'll still work just fine if there are symlinks somewhere higher
up in the paths involved -- just not for the script itself.)
An ideal CLI program really should support that, I think, but
lacking a better idea, this seems an acceptable compromise.
Requesting external images is a privacy risk, so route all external
images through Camo.
Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
As of now, editing a widget doesn't update the rendered content.
It's important to ensure that existing votes or options added later on
don't get deleted when rendered.
This seems more complex than it's worth.
For now, we just prevent edits to widgets.
This commit makes the UI clearer that editing widgets isn't allowed.
See also:
https://github.com/zulip/zulip/issues/14229https://github.com/zulip/zulip/issues/14799Fixes#17156
Introduce a new class "table-sticky-headers" in the settings
and organisation settings HTML table tags and it is used
to make the table headers fix at the top. This commit also
add the background-color and hover properties to the
settings and organisation settings table to make them look
similar to the recent_topics_table.
This commit fixes a rare flake which was most probably
caused when we clicked on the `Check All` button, and
we instantly cleared the filter when it was still marking
the user `checked`, which nullified the effect of clicking
on `Check All` button.
`ensure_basic_avatar_image` and `ensure_medium_avatar_image` are
essentially the same thing, except a size parameter.
So, refactor them into a single function.
This doesn't introduce any functional changes.
Currently the tools/build-docs was slow
because the clean option was rebulding everything.
But this is only required if one wants the left
sidebar to update.
So now set the default to exclude clean and
add clean option only if --clean is passed.
Also a warning is displayed if clean option is
not passed that the left sidebar won't update.
Fixes#17961.
Change the script to python. This is done
for the following reasons.
* It enables us to use the sanity_check.
* Later when we add warning to include
--clean flag we can use the pre-existing
WARNING from zulip_tools rather than using
terminal color codes.
TODO: Currently this script is slow as the
clean option is expensive so instead use only
html by default and clean only if --clean
option is passed.
Part of #17961.
This avoids calling parse_user_agent twice when dealing with official
Zulip clients, and also makes the logical flow hopefully easier to read.
We move get_client_name out of decorator.py, since it no longer
belongs there, and give it a nicer name.
This ensures it is present for all requests; while that was already
essentially true via process_client being called from every standard
decorator, this allows middleware and other code to rely on this
having been set.
Currently, when there are some old starred messages
in a topic, it is possible that some of them won't be
unstarred on doing "Unstar all messages in topic".
This happens when those messages haven't been fetched
yet from the server, and we have no way to verify if
they actually belong to the topic.
This commit fixes that by changing this mechanism to
first fetch all starred messages in the topic from
the server, and then send their IDs back to the backend
to unstar them.
While doing this, we assume that the user does not
have more than 1000 starred messages in that topic.
Note that, we still depend on the local data to
decide whether or not the "Unstar all messages in
topic" option should be shown in the topic popover.
A method similar to the above cannot be used here, because
making server requests before opening the popover
could visually slow down the popover opening.
Using local data for the topic popover would probably
not be a big problem, because users would want to
unstar all messages in a topic probably after noticing
that there are a lot of them, meaning there was at least
one starred message from that topic which was already
fetched, which is sufficient for us to conclude that
we need to show the option in the topic popover.
Fixes#17790
The dialog now shows `stream > topic` instead of
just the topic.
This creates a new tiny widget for this purpose,
which can be reused in other places too.
The `undefined` case is probably very unlikely, but
if the `stream_id` is bad, after the user confirms,
the request will fail on the backend, which could
confuse the user, since there will be no changes to
the starred messages in view.
So, we don't open the confirmation dialog at all in
such cases.
This logic is already thoroughly tested in node tests for
the `MessageListData` class, and the `unmuted_messages`
method of `MessageList` only directly call the corresponding
method of the data class.
There is no need to store this field (and make sure
that it's the same in both the message_list and
message_list_data objects) on message_list objects,
because we can anyways access it easily with
`message_list.data.excludes_muted_topics`, and storing
it just on the data object seems more intuitive.
This also simplifies the constructor for the `MessageList`
class.
We hide a conversation from the top-left corner PM list if
it is-
* A 1:1 PM where the other user has been muted, or
* Huddle where all users have been muted.
* Extract out data as common variable so that it can
be reused in other tests in the future.
* Use consistent dummy user_ids across tests.
* Expand the data set to cover all possible cases-
- 1:1 PMs
- Huddles
- PMs with self
We'll extend this in the future with details on what is configurable,
but this at least ensures we don't have a state where they don't
appear in our documentation at all.
This commit adds code for live updating the role field of person
object on receiving role change event from server and thus enables
live update of role in users list of settings overlay.
This commit adds code to live update page_params.is_moderator and
person.is_moderator on receiving role change event.
The code for changing role to moderator and from moderator to any
other role was added in previous commit.
This commit modifies the code to show "Moderator" correctly for
realm moderators.
As, we show role using settings_config.user_role_values object and
the same is used for showing options in changing role dropdown, this
commit also adds the moderator option in that dropdown and thus allows
the user to change role to moderator and from moderator to any other
role from frontend.
But the code for live updating page_params.is_moderator and
person.is_moderator will be in further commit.
This commit adds 'admins and moderators' option in frontend for
invite_to_realm_policy.
The logic for moderator for checking whether user is allowed to invite
others or not was added in previous commit as we use common helper
user_has_permission for all policies.
But the test user_can_invite_others_to_relam is updated for including
moderator role in this commit.
Since, we now have role value in the person objects sent from server
we can directly use that to set the value of dropdown used for
changing user role, instead of using multiple if-else conditions
for checking is_admin, is_owner, etc.
Since, we now get role value in person objects sent from server, we
can simply user user_role_map to display role in different places
instead of having multiple if-else conditions to check flags like
is_admin, is_guest, etc.
This commit modifies the user objects returned by 'GET /users',
'GET /users/me', 'GET /users/{user_id}' and 'GET /users/{email}'
endpoints to include role field.
We also include role field in the page_params['realm_users'] dict
and in the person object sent in (type="realm_user", op="add")
event.
This will help determine potentail timeout lengths, as well as serve
as a generally-useful log for locations which do not have Smokescreen
enabled.
In service of #17742.
This help mobile and terminal clients understand whether a server
restart changed API feature levels or not, which in turn determines
whether they will need to resynchronize their data.
Also add tests and documentation for this previously undocumented
event type.
Fixes: #18205.
Event of type restart could not be handled properly, because of
its special behavior. For handling this event in most natural way
we recursively call `do_events_register` when restart event is
recieved, based on custom error created for this event.
Testing: Second call to get_user_events due to recursive calling
of do_event_register, is expected to not contain the restart event.
So new test added in test_event_system.py are based on above behavior
of get_user_events.
Fixes: #15541.
There were some changes that were lost/added by mistake
during a rebase of #17707 after #18154 was merged.
Fixes the GIF icon being hidden / displayed incorrectly
with respect to the settings.
These changes were originally part of
67527a2517 but
were lost during the rebase.
These details clarify the roles of `message_id` and `message_ids`, and
should help a great deal in helping clients correctly implement this
critical API endpoint.
The order of the fields of the update_message event were previous
seemingly-arbitrary. This tries to more coherently order them:
1. Metadata
2. Stream
3. Topic
4. Content
Commit 9afde790c6 introduced a bug
concerning outgoing emails inside the development environment. These
emails are not supposed to use a real connection with a mail
server as the send_messages function is overwritten inside the
EmailLogBackEnd class.
The bug was happening inside the initialize_connection function that
was introduced in the above-mentioned commit. This function is used
to refresh the connection with an SMTP server that would have closed
it. As the socket used to communicate with the server is not
initialized inside the development environment this function was
wrongly trying to send no-op commands.
The fix just checks that the connection argument of the function is
an EmailLogBackEnd object before trying the no-op command.
Additionally as it is sometimes useful to be able to send outgoing
emails inside the development environment the get_forward_address
function is used to check if a real connection exists between Zulip
and the server. If it is the case, as EmailLogBackEnd is a subclass
of smtp.EmailBackend, the connection will be nicely refreshed.
This commit was tested manually by checking that the console prints
correctly that an email is sent to the user when it signs in inside
the development environment. It was also tested when a mail provider
is specified and the mails were correctly received.
Fixes#17795
In PR #17014, we added support for deactivate-own-user.
And while doing so, we first deactivated the client and
then reactivated it. But this implementation is a bit
hacky.
So, to fix this, we're now deactivating a test_user so that
we don't have to reactivate it. We did so by changing the value
of authentication_line.
As we want to keep endpoint code out of the
“test_curl_examples”, we changed the value of
authentication_line in `curl_param_value_generators.py`.
To work this out, we create a new global variable named
AUTHENTICATION_LINE in “curl_param_value_generators.py”
and change its value in function “deactivate_own_user” and
to use this change in “test_curl_examples,” we import
AUTHENTICATION_LINE.
AUTHENTICATION_LINE is of list data type because we want a
pointer to original mutable object so that changes made during
run time show across the module. Another way to do this is to change
the way we import variable, but that will be inconsistent to
the way we had in all other files.
To remove confusion between AUTHENTICATION_LINE and
authentication_line we renamed authentication_line
to default_authentication_line.
This commit adds 'user_can_create_streams' helper which is
used to check whether user can create streams or not and
replaces all the instances of 'page_params.can_create_streams'.
This change helps us to remove the complex logic of updating
'page_params.can_create_streams' for 'realm_update' event in
'server_events_dispatch.js', as we will always get the updated
values from the added helper for checking whether the users can
create streams or not.
This commit adds 'user_can_subscribe_other_users' helper in
settings_data.js amd this helper will replaced all the
instances of page_params.can_subscribe_other_users.
We also remove the incorrect code in server_events_dispatch.js
where we were updating page_params.can_invite_to_stream which
is actually not used in other parts of code and instead of it
page_params.can_subscribe_other_users is used to check whether
user is allowed to subscribe others or not. This code was
added in 272ed9068.
Though this could have been done in a different commit, but as
we are adding the code to use the correct updated value in this
commit only, this has been fixed here.
There is also no need of adding that complex logic to update the
correct 'page_params.can_subscribe_other_users' field on
'realm_update' event, as we are using user_can_subscribe_other_users
helper and thus we always use the updated values to check whether
the user is allowed to subscribe others or not.
This commit adds 'user_can_invite_others_to_realm' function
in settings_data.js which replaces all the instances of
'page_params.can_invite_others_to_realm'.
This change makes it possible to remove the complex logic of
updating 'page_params.can_invite_others_to_realm' on
'realm_update' event as we are using above added helper instead
of using page_params object and thus we always use the updated
value to check whether user can invite others or not.
The 'user_has_permission' helper will be used by functions added
for create_stream_policy and invite_to_stream_policy in further
commits and will help in avoiding code duplication.
This commit replaces different objects - create_stream_policy_values,
invite_to_realm_policy_values and invite_to_stream_policy_values,
with a single object common_policy_values.
Though invite_to_realm_policy do not use other fields of objects like
description, order, etc. but we can keep it as it is for now as we
would separate this setting from "Are invitations required...."
dropdown and these fields will be used then.
This is a prep commit for commits that will add helper functions in
settings_data for these policies replacing the usage of page_params
object.
The reason we didn't have this before is that a GET to one of these
URLs would suffice to unsubscribe the user; but with the
List-Unsubscribe system, we need to allow POST from a third-party page
as well.
This allows access to be more configurable than just setting one
attribute. This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
3f4d0f72fd adjusted the types in
preparation for extending the functionality, but only in later commits
that are not merged yet did it make these updates to the types.
When notifications panel is open at the top, the buddy list and
streams-filter container gets truncated due to max-height exceeding
the necessary value. The commit fixes the issue by subtracting the
panels height when it is visible. The logic to do so is already
present, but we need to ensure that we trigger the full resize code
when we get into this situation.
Fixes: #18221Fixes: #17823
This change is made to comply with the corresponding views for
the API. The incrementor implementation in zulip_bots won't work
otherwise if send_message and send_reply return None as it needs
the message id.
This commit create a directory to store the mock message for nagios and
more will be added.
The json files in this directory will be used to config the screenshot
generating script for the documentations of non-webhook integrations.
As the user can select multiple stories and edit multiple
properties at the same time, this can generate requests
without a "primary_id" containing multiple actions, while
each action contains multiple changes.
Fixes: #18022
The fixture "story_update_add_github_pull_request" is changed here as it
doesn't make sense to link a story to a PR without having "pull_request_ids"
changed. The previous example is likely a mistake which occurs when you try
to add a PR that has already been added to a story. This commit also allows
comments under the PR that link it to a story to be sent to the stream.
Fixes: #18022
Incorrectly patching zerver.lib.webhooks.common.check_send_webhook
_message does not create a mock for the webhook as desired, causing
us to do tests with mock that has never been called.
Clubhouse has a feature for the user to select multiple stories and
update them at once. This will generate a request without primary_id.
Fixes: #18022
Instead of considering only the action with the primary id, this
refactors the helper functions for generating the topic and body
for the stream messages to accept an arbitrary action and generate
the corresponding message for each of the events.
Fixes: #18022
This prevents the regex from requiring multiple spaces between
adjacent alert words by using lookahead and lookbehind (rather than
the before/after checks each needing to eat a whitespace character) so
that consecutive alert words (if any) can be highlighted.
With a frontend test covering adjacent corner cases by tabbott.
Fixes#17320
setTimeout doesn't work all the time especially when trying to
hide and display the popover at once, like when you press
to open giphy popover in message edit form while another giphy
popover is open from the compose box.
MutationObserver works reliably, hence we choose to use it
instead.
We use an icon which is more clear for what it stands for.
Increase allowed size of message-control-buttons slightly so
that they are clearly visible. This is more important for
GIF icon to be visible properly than any other icon here.
In fe66aef0ad, this call to
blueslip.fatal was ported incorrectly to `throw new Error` despite the
latter not supporting the extra_data parameter.
We address this by just doing blueslip logging with the extra data first.
Fixes#18273.
This function requires two arguments; checking for the event being
undefined is code leftover from a previous implementation.
Fixes an issue reported in #18273.
For unknown reasons, 6f7b973d3b
introduced an invalid semi-duplicate call to ui_report.error that
passed the wrong number of parameters.
We should just call `ui_report.error`, which already had logic to
handle the distinction between 40x and 50x errors.
Fixes an issue reported in #18273.
Fixes#18052
When we select a draft with no stream, topic of
drafts gets disappeared. This commit fixes this bug.
In the restore_draft function of drafts.js, we were
explicitly removing the topic of the draft when the stream is
not set.
This was bit reluctant to me, like why we're removing
the topic when the stream is not set. After discussing this on
the Zulip organization chat, I came to know that all code for
drafts is added in commit 1929cc5190.
And as this is a big commit, this microcode would not have
gotten much scrutiny and might get missed.
So removing this feature is not an issue. Also, it may be a bad
design decision to delete topic when we restore drafts. So we should
remove this feature.
When we select a draft with no topic, we typically got navigated to an
empty narrow, because the topic field was empty. Rather than
navigating to a view that will show "Nothing's been sent here yet!",
we should not navigate at all.
Initially, in the restore_draft function of the drafts.js
we were only checking that stream is empty or not,
now we also check for the topic, and if both are set,
we navigate to related context; otherwise, we don't navigate.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour. This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
Long-term, we probably want to make the filtering options more
generic, but there's little harm in adding an option for a specific
group we're likely to email multiple times.
This was apparently one of our few portico pages that never got the
white-box migration.
I did a bit of copyediting while looking at this page as well.
This extends the /json/typing endpoint to also accept
stream_id and topic. With this change, the requests
sent to /json/typing should have these:
* `to`: a list set to
- recipients for a PM
- stream_id for a stream message
* `topic`, in case of stream message
along with `op`(start or stop).
On receiving a request with stream_id and topic, we send
typing events to clients with stream_typing_notifications set
to True for all users subscribed to that stream.
Update `docs/production/install.md` and
`docs/production/deployment.md` to document the install flags that can
be used as part of the installer more clearly.
Fixes#18122.
Previously, we used to do a kind of "local echo"
whenever the user muted/unmuted a topic. Meaning,
we used to do most of the UI update work before making
the API call to mute the topic, instead of after
receiving the `muted_topics` event. This behavior
has been so since the beginning of time
(b4b6fa14d3) and isn't ideal because:
1. If the request fails on the backend, the UI
could end up in an incorrect state.
2. Adds code complexity.
3. Makes it difficult to catch bugs related to
live-update (like the one fixed by
f725711ff2).
4. Isn't consistent with other parts of the
codebase, which do the UI update while handling
events.
This commit makes it so that all the UI update
work is done only after recieving the `muted_topics`
event.
The only possible issue with this strategy could
be users sending another duplicate request in the small
time span before receiving the event. But that isn't
a big problem, because all requests involved here are
idempotent, and the worst that can happen is a HTTP 400.
The comment explains in more detail, but this should help avoid cases
where a Zulip server accidentally avoids the nag by having upgraded to
a 2-year old Zulip version from a 3-year-old version 2 months ago.
This commit adds support for a `None` option in the dropdown menu
of `Notification sound`. When this option is selected, no audible
notification is sent to the user.
`None` will appear as the first option in the dropdown menu, since
this is not categorized as a playable audio.
This new option is added so that folks can disable audio notifications
without losing their other notification configuration (like for PMs, mentions).
Necessary test case is added for this new option.
Fixes#16090.
Item field is added to the pill items, this is done to utilize
it wherever possible, to distinguish between diffrent types
of pill items(stream|user_groups|users).
It is also a preparatory commit to support expanding of stream
and user group pills.
Since it was introduced in 5ce0db9f43,
we've had alt text for the hotspots illustration with an invalid
translation tag.
Fix this by removing the alt text, since the image has no functional
effect and so the right answer should it fail to load or the user is
visually impaired is to ignore it.
Deduplicate control buttons by re-using the
compose_control_buttons.
A link to `help` overlay was added to `message_edit_form`
as a part of this process.
This fixes a bug that when video provider is set to `Jitsi`
from `none` in organization settings while message_edit_form
is open, the video icon is not displayed since
it was not present in the message_edit_form DOM even if
compose.update_video_chat_button_display tries to display it.
It is fixed since the `.video_link` element is always present
in DOM of `message_edit_form` now.
We convert the following elements to use a class instead of
id for accessing them across the codebase:
* markdown_preview
* undo_markdown_preview
* markdown_preview_spinner
* message_edit_content
* preview_content
Converted them together since changes to one impacted the other in
some modules like click_handlers.
Also, added a function in rows to get `message_row`.
We use `.compose_upload_file` across compose and message_edit_form
for file upload icon. This will help us share common code between
`compose` and `message_edit_form`.
In both compose and `message_edit_form` we use `file_input`
class to identify the file `input` element. This will help
to more easily share common elements between compose and message_edit.
Since we can have multiple instances of `message_edit_form`, it
makes sense to have it as a class.
We track the message_edit_form by setting an id to
`form` element dependent on message_id.
This commit makes it so that muted users never appear
in the right sidebar buddy list, filter text or not.
The hiding is done in the frontend only, and we still
recieve presence data from the server as before, so
no extra work is required on unmuting someone, other
than to rerender the user list.
Long term if we find that there are too many muted users,
we may want to optimize how we send presence data, but
that is unlikely to happen.
The other less extreme option is to gray out muted users,
but that cannot be done because it would conflict
with the graying out we do for non-recipients when the
compose box is open.
This makes the naming more intuitive and gives us a
single place to add more conditions on filtering
user_ids, with the current motivation being filtering
muted users, the logic for which will be added in further
commits.
This is a direct code move which puts all the functions
related to `user_id` lists near the bottom of the file.
This will make the file slightly easier to read.
Add a `--dry-run` flag to send_custom_email management command
in order to provide a mechanism to verify the emails of the recipients
and the text of the email being sent before actually sending them.
Add tests to:
- Check that no emails are actually sent when we are in the dry-run mode.
- Check if the emails are printed correctly when we are in the dry-run mode.
Fixes#17767
Previously the outgoing emails were sent over several SMTP
connections through the EmailSendingWorker; establishing a new
connection each time adds notable overhead.
Redefine EmailSendingWorker worker to be a LoopQueueProcessingWorker,
which allows it to handle batches of events. At the same time, persist
the connection across email sending, if possible.
The connection is initialized in the constructor of the worker
in order to keep the same connection throughout the whole process.
The concrete implementation of the consume_batch function is simply
processing each email one at a time until they have all been sent.
In order to reuse the previously implemented decorator to retry
sending failures a new method that meets the decorator's required
arguments is declared inside the EmailSendingWorker class. This
allows to retry the sending process of a particular email inside
the batch if the caught exception leaves this process retriable.
A second retry mechanism is used inside the initialize_connection
function to redo the opening of the connection until it works or
until three attempts failed. For this purpose the backoff module
has been added to the dependencies and a test has been added to
ensure that this retry mechanism works well.
The connection is closed when the stop method is called.
Fixes: #17672.
This was introduced in 8321bd3f92 to serve as a sort of drop-in
replacement for zerver.lib.queue.queue_json_publish, but its use has
been subsequently cut out (e.g. `9fcdb6c83ac5`).
Remote its last callsite.
Long labels like "Yes, Unsubscribe this stream" can
be confusing for translators and it can also create bad
strings that can end with like 4 long words in German.
It is better to have the simple options like "Confirm"
and "Cancel". This commit fixes this issue by changing
the text to "Confirm".
Fixes#17926.
This commit removes redundant yarn cache by removing the old
version directories, i.e. All the directory under `~/.cache/yarn`
except `~/.cache/yarn/v6` (current version directory).
Fixes#15964.
Previously, realm emojis can override default emojis in emoji-picker,
if the user sets an exisitng emoji name to his/her custom emoji.
For clear understanding--
If a user sets an realm emoji with name `smile`, this leads to the
newly realm emoji override the existing default emoji `smile`.
To address such behaviour, Added a warning modal which requires
the user confirmation before overriding the default emoji.
Fixes#16913.
Moved `admin_user_list` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
rather than the `/templates/settings` folder.
Fixes#18227
Moved `admin_user_group_list` template to `/templates/settings/` folder
as earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder.
Also modified the node tests to reflect the new changes.
Moved `admin_tab` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder instead.
Moved `admin_settings_modals` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder.
Moved `admin_profile_field_list` template to `/templates/settings/` folder
as earlier, it was inaccurately placed within the `/templates` folder
rather than the `/templates/settings` folder.
Also modified the node tests to reflect the new changes.
Moved `admin_linkifier_list` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
rather than the `/templates/settings` folder.
Moved `admin_invites_list` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder instead.
Moved `admin_human_form` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder.
Moved `admin_export_list` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
rather than the `/templates/settings` folder.
Moved `admin_emoji_list` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and should have been within the `/templates/settings` folder.
Moved `admin_default_stream_list` template to `/templates/settings/`
folder as earlier, it was inaccurately placed within the `/templates`
folder and should have been within the `/templates/settings` folder
instead.
Moved `admin_bot_form` template to `/templates/settings/` folder as
earlier, it was inaccurately placed within the `/templates` folder
and `settings` folder is specifically for storing administrative
UI templates.
It's better to just raise JsonableError here, as that makes this error
processed in the central place for this kind of thing in do_rest_call:
---------
except JsonableError as e:
response_message = e.msg
logging.info("Outhook trigger failed:", stack_info=True)
fail_with_message(event, response_message)
response_message = f"The outgoing webhook server attempted to send a message in Zulip, but that request resulted in the following error:\n> {e}"
notify_bot_owner(event, failure_message=response_message)
return None
----------
which does all the things that are supposed to happen -
fail_with_message, appropriate logging and notifying the bot owner.
These aren't good mocks of a good reponse - a good response is supposed
to contain valid json that doesn't trigger error-handling in the
codepath. Without this change, all these actually trip up on
json.loads(response.text) in process_success_response.
This involves in two changes for styling.
1. The alert class is moved from alert.css to app_components.css as this
class serves nothing but to default .alert elements to be hidden. This
is only required in the webapp but not portico pages (where .alert
elements are preferred to be shown by default).
2. The import statement for alert.css is moved from app.js to common.js,
so that both the webapp and the portico pages can share the styles. This
will be fine to share the styles as .alert-display, .alert-animations,
.alert-box are more specific then .alert and they use nested class to
define styles for inner elements.
Undoes #17936 properly.
This improves the UX of creating a stream for atleast 1000+ users
realm by showing the the stream creation form much faster than
before.
Search, user addition, scrolling worked smoothly on 15k+
users realm as tested on dev setup.
Also, simplebar is used to replace the default scrollbar.
Fixes#16805
When moving a topic within a stream that is deactivated, the stream
may not be present in stream_data. Avoid throwing an exception in
this situation by leaving stream_name as undefined; existing logic
seems to anticipate that as a possibility, so we don't need a broader
change.
Longer term, we may want to just send to clients basic data about
archived streams that the user has access to.
Fixes: #17271
Remove content edit keys if present in edit_history_event
when passing to update_messages_for_topic_edit.
Since content edit is only applied to the edited_message,
this shouldn't be part of the rest of the messages for which
topic was edited. This was a bug identified by
editing topic and content of a message at the same time
when more than 1 message is affected.
This commit adds a helper function: `mute_or_unmute_topic`
in `click_handlers.js` to mute and unmute a topic. This function
is called when a user mutes/unmutes a topic from its recipient bar
or `recent topics`.
This is a prep commit for `Allow unmuting of a topic from its recipient bar`.
Related issue: #15223.
The bug happens in the case when the topic name is not set and the
user clicks on always_visible_topic_edit button results into unusual
behaviour of the always_visible_topic_edit button. To fix this, this
commit fix the behaviour by hiding and showing the
always_visible_topic_edit button in the appropiate situations, at the
same time we hide/show similar buttons.
Fixes#17813.
This allows us to use different "Show password" and "Hide password"
for these labels, which is more consistent with how other products
implement this.
It also lets us delete N duplicate copies of these strings in the HTML.
The "Save failed" standard text is appropriate for many of our
settings, but for changing one's password, we can go with just the
"Wrong password" part provided by the server.
The show password feature is a functionality to
toggle the visibility of the password fields in forms
so that one can check if they have entered the correct
password or not. We implement this using an eye icon
toggling which converts input field type from password
to text and vice-versa.
Fixes part of #17301.
This allows us to hide tooltips automatically when the
parent container is hidden while tooltip is active.
In an overlay, when a tooltip is active and `esc` is pressed,
the tooltip will remain active without this commit.
This has side effects of some properties of parent applying to
tooltips if property is directly set to `div`. Through manual testing,
only area where this was found was fixed.
We destroy the tooltips for which `reference` was either removed
from DOM or is hidden.
We only need to do this for tooltips
contained in simplebar containers for which tooltips can
overflow the boundary of the simplebar container.
There are 4 approaches we could have done this:
1. Asked tippy.js maintainers to do this for us.
In https://github.com/atomiks/tippyjs/issues/938 the
maintainer said that it is the responsibility of
the user to do so.
2. Tracked whenever we update the DOM for such elements and hide
tooltips when we were hiding the `reference` elements. This had
various problems like it is hard trigger events when certain elements have
been removed from DOM when `html()` method is used to render
new content.
3. Run an `optimized` periodic job to destroy tooltips when
`reference` elements are hidden. This isn't a good method to
do this since it sucks power and adds latency.
4. Use a `MutationObserver` on the parent element and watch
for changes. This methods seems to work well with no bad
side effects. We use this approach.
Instead of inserting tooltip inside `body`, we directly insert
it inside the `reference` element. This helps us to automatically
hide the tooltip when we hide the `reference` element.
This avoids the bug of tooltip persisting when the message reaction
is removed while the tooltip is active.
To reproduce:
* React 👍 to a message.
* Hover over the reaction.
* Press `+` from keyboard.
You will see reaction tooltip persisting while the reaction is hidden,
also "Add emoji" icon is displayed with tooltip.
Doing the same for elements which are inside a simplebar container
and for which tooltips can span outside the simplebar container,
makes the tooltips not visible or cut at the edges of simplebar
container since simplebar containers have overflow set to `hidden`.
This is something that cannot fixed as per
https://github.com/Grsmto/simplebar/issues/347
So, for simplebar contained elements we insert the tooltip to
`body`.
`reference` element: Element for which tooltip is displayed.
model__id syntax implies needing a JOIN on the model table to fetch the
id. That's usually redundant, because the first table in the query
simply has a 'model_id' column, so the id can be fetched directly.
Django is actually smart enough to not do those redundant joins, but we
should still avoid this misguided syntax.
The exceptions are ManytoMany fields and queries doing a backward
relationship lookup. If "streams" is a many-to-many relationship, then
streams_id is invalid - streams__id syntax is needed. If "y" is a
foreign fields from X to Y:
class X:
y = models.ForeignKey(Y)
then object x of class X has the field x.y_id, but y of class Y doesn't
have y.x_id. Thus Y queries need to be done like
Y.objects.filter(x__id__in=some_list)
There exists a logic bug (see #18236) which causes duplicate
usermessage rows to be inserted. Currently, this stops catch-up for
all users.
Catch and record the exception for each affected user, so we at least
make catch-up progress on other users.
In general, `./scripts/restart-server` will already work in any
circumstance where the server is already stopped and needs to be
started. However, it will output a couple minor warnings, and it is
not readily obvious that it *will* work correctly.
Add an alias for `restart-server` named `start-server`, for
parallelism with `stop-server`, which omits the steps of
`restart-server` which would stop the server first.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.
Add an explicit tool which only stops the relevant services. It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.
Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.
Fixes#14959.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files. However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.
Fix the supervisor configuration file paths.
Since the "mute users" feature isn't complete yet,
this UI is shown only in development setups.
Ideally we should have had this commit after the whole
feature was completed and merged, but doing so makes it
difficult to test and merge subparts of the feature one by
one (which is a better workflow, while we still decide what
exactly we want this feature to do).
This commit adds a new button in the user info popover
to mute or unmute the user, and uses a confirmation
dialog while muting, because muting a user accidently can lead
to the muter losing out on a lot of information.
TODOs when making this UI visible in production-
1. Make a /help page and link to it from the confirmation
dialog and the API docs.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian. We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.
Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.
Fixes#18179.
The stacktraces here are seldom useful -- for the calls to
upgrade-stage-2, we know precisely what was run. For the `run`
wrapper, the output contains the command that failed, which is
sufficient to identify where in the upgrade process it was. Showing
more stacktrace below the actual error merely confuses users and
scrolls the real error off of the screen.
This reverses the policy that was set, but incompletely enforced, by
commit 951514dd7d. The self-closing tag
syntax is clearer, more consistent, simpler to parse, compatible with
XML, preferred by Prettier, and (most importantly now) required by
FormatJS.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
For installs which use the `upgrade-zulip-from-git` process, the
deployment directory is a git checkout. This means that an
administrator can, as an emergency tool, run `git revert` and similar
commands -- assuming there is a `~/.gitconfig` set up for the zulip
user.
Add commands to `scripts/lib/install` to create a `~/.gitconfig` file
at installation time. The `user.name` and `user.email` fields are set
to the hostname and passed-in `--email` value, respectively.
Fixes#18039.
Previously, we would show the guest role oddly between organization
administrator and organization owner.
We preserve the property that the Member role is the default.
This was a mostly harmless bug, since those users cannot have active
clients, but fixing it will improve performance in any Zulip
organization where the vast majority of users are deactivated.
This commit adds an API to `zproject/urls.py` to edit/update
the realm linkifier. Its helper function to update the
database is added in `zerver/lib/actions.py`.
`zulip.yaml` is documented accordingly as well, clearly
stating that this API updates one linkifier at a time.
The tests are added for the API and helper function which
updates the realm linkifier.
Fixes#10830.
Use backend-endpoint function instead of helper function in
`test_realm_linkifiers.py` so that tests are more end-to-end.
The removed helper function: `do_add_linkifier` is tested in
`zerver/tests/test_events.py`.
Linkifier error message: `Filter not found` is
updated to `Linkifier not found.`.
Similarly, `filter_id` description is updated to:
`The ID of the linkifier that you want to remove.`,
renamed the term `filter` with `linkifier`, in `zulip.yaml`.
This extra commit adds support for creating user group pills
in cases that do not use typeahead like, pasting the group
name, copying it from the user group pill.
This completes the remaining work required to support
addition of all members of a user groups to stream.
Fixes#15186.
We update the pills typeahead logic to also include user group
results and pass the "user_group" key in `opts` to enable this
option for Add subscriber form.
The changes includes:
* Exporting the `query_matches_name_description` function, to
deduplicate the typeahead `matcher` logic.
* Creating a `people.is_known_user` function, to deduplicate
the typeahead `sorter` logic.
* Add a new `user_group_pill.js`, to allow adding user group
pills in input widgets that support pills.
This has been tested manually as well as by adding some new
node tests.
This commit removes`typeahead_helper.sort_recipientbox_typeahead`
which was introduced in 639ec9380a
for the private message recipient box when it used to accept comma
separated text input for recipients. All it doew was cleaning away
invalid recipients from string, like:
"a, , b" was cleaned to [a, b].
This can be now removed because it was now used only in
pill_typeahead.setup code path, and the pills code already handles
invalid cases by filtering out all falsy (invalid) recipients.
This is both more correct and also fixes this element having had a
name very similar to message_control_button, which refers to an
element in the message_controls section of a rendered message.
We move compose.html to compose.hbs file while keeping
`#compose` still in `home.html` as a hanger
where append rest of the elements.
This will provide us with two benefits:
* We could share common elements between message_edit_form and
compose.
* We can insert compose directly in any element. We may decide to
do it for recent topics.
Adds a setting UI to list all configured playgrounds
in a realm. The filter functionality can be used to
search playgrounds by its name or language.
Default sort is provided on the 'pygments_language'
field.
Front tests added to maintain server_event_dispatch
coverage. The `settings_playgrounds.js` file is added
to coverage exclusion list since it is majorly UI
based and will be tested using puppeteer tests (in
following commits).
To prevent breaking of the hardcoded playgrounds, we resort
to checking if realm_playgrounds is empty and falling back
to the hard-coded list if so. This logic is removed in the
followup commit which introduces the UI to add a playground.
This prevents us from having to json encode every field in the POST
request to /realm/playgrounds, and keeps the client logic simpler
when adding a playground.
I have added support for generating integration screenshots remotely by
adding a `realm_uri` parameter to `tools/message-screenshot.js` which we
then pass `realm.uri` to from within
`tools/generate-integration-docs-screenshot`.
I have made `tools/setup/optimize-svg` do the SVG optimization
automatically rather than just telling you the command to run if they
need optimizing. This included adding a `--check` parameter to use in
CI to only check as we previously did rather than actually running the
optimization.
I have also made `tools/setup/optimize-svg` execute
`tools/setup/generate_integration_bots_avatars.py` once it has run the
optimization to ensure it is always ran.
This makes it one less command to run when creating an integration,
but also means that we catch instances where a PNG has just been
copied into the `static/images/integrations/bot_avatars` folder as the
only instance where this won't be run is if `optimize-svg` has not
been run which would be caught in CI.
Fixes#18183. Fixes#18184.
The caller is supposed validate the stream and user realm match, but
since this is a security-sensitive function, we should have this
defensive code to protect against some validation bugs in the caller
leading to this being called incorrectly and returning True.
Fixes#17922.
These two places fetch subscriptions for the sake of getting user ids to
send events to. Clearly deactivated users should be excluded from that.
get_active_subscriptions_for_stream_id should allow specifying whether
subscriptions of deactivated users should be included in the result.
Active subs of deactivated users are a subtlety that's easy to miss
when writing relevant code, so we make include_deactivated_users a
mandatory kwarg - this will force callers to definitely give thought to
whether such subs should be included or not.
This commit is just a refactoring, we keep original behavior everywhere
- there are places where subs of deactivates users should probably be
excluded but aren't - we don't fix that here, it'll be addressed in
follow-up commits.
Earlier, the email label tag was dislocated with respect to
it's input field, causing the UI to be disorganized.
Rectified by moving the label tag above the email input field
and added placeholder value to the field.
This is in preparation for moving this code to @zulip/shared for use in
the mobile app, where we will want to use Sentry for reporting errors,
rather than blueslip.
The way I've done this only allows for reporting one type of error
(currently, blueslip.warn), but seeing as we only have one place we
report an error, that seems like something we can fix if we want more
error levels at a later date.
The unicode horizontal ellipsis is visually nicer.
b40e50f295 removed many of these,
by changing the text itself.
This commit handles the remaining few.
This commit adds new helper can_move_messages_between_streams
which will be used to check whether a user is allowed to move
messages from one stream to another according to value of
'move_messages_between_streams_policy'.
I have updated `tools/run-dev.py` to output the correct subdomain such as
`http://zulip.username.zulipdev.org` so that the user knows the correct
subdomain to access the Zulip Dev realm on.
I have updated the remote development documentations to be more accurate
when it comes to developing on a Zulip Development Droplet to ensure
the user knows to access at `zulip.username.zulipdev.org`.
Since all the message reactions are inserted before the
add reaction button, if it is the first child, we can safely
remove it.
We changed this from `only-child` to be `first-child` because
we append tooltips as siblings of `reaction_button` but since
they are appended, they are always appended after the `reaction_button`.
Thus, if there were tooltips present the reaction_button won't hide.
Current production code uses client_id in the event dict and this test
should be updated to reflect that. Old format event can still be
consumed by the worker, but that is already tested by
WorkerTest.test_UserActivityWorker.
Since c3a8a15bae removed the last
instance of code using the dictionary code path, we actually need to
wait until one can no longer upgrade directly from 4.x to master in
order to avoid breakage should we remove this compatibility code,
since only today did we stop generating the old event format.
The bulk deletion codepath was using dicts instead of user ids in the
event, as opposed to the other codepath which was adjusted to pass just
user ids before. We make the bulk codepath consistent with the other
one. Due to the dict-type events happening in 3.*, we move the goal for
deleting the compat code in process_notification to 5.0.
We use the snakeoil TLS certificate for PostgreSQL and Postfix; some
VMs install the `ssl-cert` package but (reasonably) don't build the
snakeoil certs into the image.
Build them as needed.
Fixes#14955.
django.conf.urls.url is actually a deprecated alias of
django.urls.re_path, but we want path instead of re_path.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
django.utils.encoding.smart_text is a deprecated alias of
django.utils.encoding.smart_str as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
django.utils.http.is_safe_url is a deprecated alias of
django.utils.http.url_has_allowed_host_and_scheme as of Django 3.0,
and will be removed in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reduces the complexity of our dependency graph.
It also makes sub_store.get parallel to message_store.get.
For both you pass in the relevant id to get the
full validated object.
This breaks an indirect dependency of stream_data
on the channel module.
It's a verbatim code move, apart from the one-line
helper `has_history_for`. It's not totally clear
to me why the original code doesn't call into
`is_complete_for_stream_id` to early-exit, but
figuring that out is outside the scope of my
change.
It's possible that we will eventually just subsume
this tiny module into topic_list once we finish
breaking all dependencies, but we may want to
reuse this for something like Recent Topics
or other similar UIs.
It's also possible that we'll want to rename
stream_topic_history -> stream_topic_history_data
sometime soon, possibly after we clean up its
dependency on message_util soon.
I have added a documentation page for the GitHub Actions integration to
`/integrations/doc/github-actions` with a link to the Zulip GitHub
Actions repository.
Tweaked by tabbott to add cross-links with the main GitHub integration.
I have updated any missing bot avatars after running the tool whilst
creating integrations. These needed adding otherwise the tool creates
them whenever it is ran.
This widget only filters the user's subscription -- it's only suggest
public streams that the user is not subscribed to. "Filter" is the
correct label for a widget with this use case.
We only update the `.private_messages_header` here since
unread_counts of `.expanded_private_message` are updated
via `pm_list.update_private_messages`.
This fixes the bug of PMs in `.expanded_private_message` having
the same unread count as `private_messages_header`.
Since we rerender the DOM of `.expanded_private_message` every
time we update unread count of PMs, we don't need to manually
update them here. Also, we always keep them on display since
there is no real need to toggle them. They are not visible
when they have 0 unread counts via `.zero_count`.
While rest of the app has ported to the new system of updating
unread_counts `activity` was not ported. This resulted in
unread count in buddy list not being updated when new
PMs arrive.
The series of commits to consolidate CSS classes
for the various unread-count spans across our app
created a bug where the stream_list.js code's selector
starting capturing the unread spans in topic list items.
Suppose you had a stream with these topics:
Foo 10
a 3
b 3
c 4
If another unread came in, you would briefly see:
Foo 11
a 11
b 11
c 11
Now we just use subscription_block to find the
element that we want to tweak.
I remove a convoluted node test here. Part of the
reason the node test was convoluted was that the
original implementation was overly complex. I will
try to re-introduce a simpler test soon, but this
is a bit of an emergency fix.
This reduces our dependency on message_list code (via
message_util), and it makes moving streams/topics and
deleting messages more performant.
For every single message that was being updated or
deleted, the previous code was basically re-computing
lots of things, including having to iterate through
every message in memory to find the messages matching
your topic.
Now everything basically happens in O(1) time.
The only O(N) computation is that we now lazily
re-compute the max message id every time you need it
for typeahead logic, and then we cache it for
subsequent use. The N here is the number of messages
that the particular sender has sent to the particular
stream/topic combination, so it should always be quite
small, except for certain spammy bots.
Once the max has been calculated, the common operation
of adding a message doesn't invalidate our cached
value. We only invalidate the cache on deletes.
The main change that we make here from a data
standpoint is that we just keep track of all
message_ids for all senders. The storage overhead here
should be negligible. By keeping track of our own
messages, we don't have to punt to other code for
update/delete situations.
There is similar code in recent_topics that I think can
be improved in similar ways, and it would allow us to
eliminate functions like this one:
export function get_messages_in_topic(stream_id, topic) {
return message_list.all
.all_messages()
.filter(
(x) =>
x.type === "stream" &&
x.stream_id === stream_id &&
x.topic.toLowerCase() === topic.toLowerCase(),
);
}
This wasn't being validated before. There wasn't any possibility to
actually succeed in moving a private message, because the codepath would
fail at assert message.is_stream_message() in do_update_message - but we
should have proper error handling for that case instead of internal
server errors.
Otherwise an admin can move a topic from a private stream they're no
longer a part of - including the newest messages in the topic, that
they're not supposed to have access to.
A bug in the implementation of the topic moving API resulted in
organization administrators being able to move messages to streams they
shouldn't be allowed to - private streams they weren't subscribed to and
streams in other organization hosted by the same Zulip installation.
In our current model realm admins can't send messages to private streams
they're not subscribed to - and being able move messages to a
stream effectively allows to send messages to that stream and thus the
two need to be consistent.
A bug in the implementation of the all_public_streams API feature
resulted in guest users being able to receive message traffic to public
streams that should have been only accessible to members of the
organization.
A bug in the implementation of the can_forge_sender permission
(previously is_api_super_user) resulted in users with this permission
being able to send messages appearing as if sent by a system bots,
including to other organizations hosted by the same Zulip installation.
- The send message API had a bug allowing an api super user to
use forging to send messages to other realms' streams, as a
cross-realm bot. We fix this most directly by eliminating the
realm_str parameter - it is not necessary for any valid current use
case. The email gateway doesn't use this API despite the comment in
that block suggesting otherwise.
- The conditionals inside access_stream_for_send_message are changed up
to improve security. They were generally not ordered very well,
allowing the function to successfully return due to very weak
acceptance conditions - skipping the higher importance checks that
should lead to raising an error.
- The query count in test_subs is decreased because
access_stream_for_send_message returns earlier when doing its check
for a cross-realm bot sender - some subscription checking queries are
skipped.
- A linkifier test in test_message_dict needs to be changed. It didn't
make much sense in the first place, because it was creating a message
by a normal user, to a stream outside of the user's realm. That
shouldn't even be allowed.
A bug in the implementation of replies to messages sent by outgoing
webhooks to private streams meant that an outgoing webhook bot could be
used to send messages to private streams that the user was not intended
to be able to send messages to.
Completely skipping stream access check in check_message whenever the
sender is an outgoing webhook bot is insecure, as it might allow someone
with access to the bot's API key to send arbitrary messages to all
streams in the organization. The check is only meant to be bypassed in
send_response_message, where the stream message is only being sent
because someone mentioned the bot in that stream (and thus the bot
posting there is the desired outcome). We get much better control over
what's going by passing an explicit argument to check_message when
skipping the access check is desirable.
We use an inverted color scheme to what we use for unread messages, so
that one's eyes scan these as different from unreads.
We also need to introduce a 1px offset because the border takes up space.
Fixes#17938.
In an effort to use a common class to display unread counts across
the app, we simplify the elements used to show unreads and use a
single `span` with `unread_count` class to do so.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
This commit replaces the "echo_data.orig_raw_content" with
"echo_data.raw_content" in "message_edit.js". If the user is editing
a message and it fails, all the edited data used to get lost. Now, the
new edited content appears so that the user does not lose his edited
content.
This bug has been present since the original #12145, where a
refactoring error resulted in using the pre-edit data.
Fixes#18142.
If you have two emojis of the same name, we were
trying to render the non-realm emoji without an
emoji_code, since the condition that I changed
here was erroneously just seeing if the name was
among the realm emojis.
This fix prevents a funny symptom that is reported
in #18135. The symptom is that the non-realm
emoji causes us to render the top-left emoji in
our sprite sheet, since we didn't put emoji_code
into the args.
This doesn't fix the fundamental problem that
having two emojis of the same name is extemely
confusing. Our markdown parser obviously can't
distinguish ":thank_you:" from ":thank_you:". For
the the typeahead case I suppose we could add some
kind of suffix.
So, this fix only simplifies debugging; it doesn't
get to the root of the problem.
We had a mix of the two names, and "video call provider" both feels
more professional and more clear about precisely what it does.
We don't change the API fields, since it doesn't seem worth an API
migration.
Documentation about using and configuring different video
call providers was present at /help/start-a-call. This
documentation was later moved to /integrations as asked in
issue #17588. So to avoid redundancy most of the documentation
in /help/start-a-call is removed, by providing appropriate
links, pointing to where new documentation lives.
Fixes: #17588.
Moves documentation about using zoom as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
Moves documentation about using Big Blue Button as video call
provider to /integrations. This documentation was earlier
present at /help/start-a-call and is moved as asked in issue #17588.
Moves documentation about using jitsi as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
This is a preparatory commit that documents
steps for setting a video chat provider for an
organization. It is added as a separate section
in /help/start-a-call. Permalinks to this section
are utilized in succeeding commits which move
documentation of our video call providers to
`/integrations`, as asked in issue #17588.
Tweaked by tabbott for clearer/more consist language.
KaTeX makes use of a "span.overlay" element for the little vector
arrow symbol on top of a `\vec` object. This conflicts with Zulip's
CSS for our overlays, which are divs with the `overlay` CSS class.
While KaTeX may rename their class
(https://github.com/KaTeX/KaTeX/issues/1456), we can work around this
issue by scoping our own overlay CSS and click handlers to
"div.overlay" rather than ".overlay".
Fixes#18068.
In zulip.yaml, `pattern` and `url_format_string` are extracted
as a shared parameters.
This is done as a preparatory commit for `Add settings UI to edit
linkifiers`.
Related issue: #10830.
This commit adds a helper function named `handle_linkifier_api_error`
to `static/js/settings_linkifiers.js`. As the name suggests, this
function handles the error returned from the API, specifically
for operations on linkifiers (like adding linkifier).
This is a prep commit for `Add setting to edit linkifiers`.
Related issue: #10830.
The main motive to add this helper function is to avoid copying
substantial blocks of code, as it tends to result in someone later
fixing bugs in only one of the two places.
This change attempts to highlight WSL 2 as the default installation
method for Windows; that is currently much more reliable than Vagrant.
Further work is probably needed to complete this transition.
This documentation does not work and has not been used for years.
At this point, `provision` is sufficiently flexible in terms of
supporting different platforms that any future work will be to extend
it, rather than maintaining awkward manual installation documentation.
This commits adds instructions to bring up the
vagrant development server using the Hyper-V provider.
Additionally, this commits also removes the indication
that this guide for `non vagrant use` from the top of
the document. Also fixes a little grammatical error
under the `Newer versions of supported distributions`
heading.
Fixes#16994.
HyperV is native to Windows and should give a
significant performance boost over other vagrant
providers. Running the Zulip development environment
using HyperV requires a few custom changes. This
commit aims to add HyperV support to Zulip's Vagrant
configuration.
Fixes part of zulip/zulip#16994
This commit adds a "Create your own" button on the integrations page. It
redirects to "api/integrations-overview" page and is placed by the side
of "Request an Integration" button.
Fixes#7935
There was no proper documentation to guide user to request an integration.
The following changes documents the whole process and links it from the
`/integrations/` page making it visible to the end-user.
Fixes#7935
In "test_curl_examples.py" we find the functions that registered but
never called. To improve readablity, now we have the full
implementation in curl_param_value_generators, rather than inspecting
its fields from another module.
Currently, there are separate tests for testing change of one role
to other, precisely 8, with most of them having similar structure
of code. This commit adds a helper function check_user_role_change
which contains all the code for testing and the tests for different
role just use this helper function to avoid duplication of code.
This refactor is helpful considering we would want to add tests
for moderators also, which would contain multiple tests for
testing changing different user roles to moderator and vice versa.
Tweaked by timabbott to make the code more readable by checking for
every user role flag instead of just checking the certain flags and
using conditionals.
Co-authored-by: Tim Abbott
This commit removes can_access_all_realm_members function as
it is not used anywhere in code other than tests.
This function was originally added in 4483e33102 and was
only used in digest.py other than the tests, but its use
in diget.py was removed in 735b6cb761 and the function
itself was not removed from models.py.
We refactor check_has_permission_policies to check for all user roles for
each value of policy. This will help in handle a case where a guest is
allowed to do something but moderator isn't.
We need to do user_profile.refresh_from_db() in validation_func because
the realm object from user_profile is used in has_permission and we need
updated realm instance after changing the policy.
This is a follow-up commit to 9a4c58cb.
Plurals are handled natively by the ICU MessageFormat syntax, so I
think we don’t have to do anything here.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reverts commit f81cc16a0f (#17999).
The {{#tr}} helper now includes the functionality that we wanted from
this.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This modal is rendered on demand and inserted into
revoke_invite_modal_holder. We don’t need another hidden copy here,
and having one just causes id conflicts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We don’t extract either of these strings correctly at present, but I’m
about to replace the entire frontend extraction system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, when viewing the source of a locally-echoed (pending)
message, if one tried to copy-paste using the copy button, the
"Copied!" status was displayed but the message was not copied.
This was because as we use message ids with a . in then for locally
echoed messages, which ClipboardJS considered an error (invalid
selector).
To solve this
* Remove the data-clipboard-target attribute of copy button.
* Dynamically set target in ClipboardJS config
the based on message id using querySelector
with the message id written with CSS.escape.
Fixes#18053.
* This introduces a new event type `realm_linkifiers` and
a new key for the initial data fetch of the same name.
Newer clients will be expected to use these.
* Backwards compatibility is ensured by changing neither
the current event nor the /register key. The data which
these hold is the same as before, but internally, it is
generated by processing the `realm_linkifiers` data.
We send both the old and the new event types to clients
whenever the linkifiers are changed.
Older clients will simply ignore the new event type, and
vice versa.
* The `realm/filters:GET` endpoint (which returns tuples)
is currently used by none of the official Zulip clients.
This commit replaces it with `realm/linkifiers:GET` which
returns data in the new dictionary format.
TODO: Update the `get_realm_filters` method in the API
bindings, to hit this new URL instead of the old one.
* This also updates the webapp frontend to use the newer
events and keys.
This commit fixes the issue of error message not getting
displayed when the `Full name` field, in bots settings, is given
a duplicate name of an already created bot with the same name.
We were closing the modal each time whether the request is
successful or not. Hence, we now close the modal only
when the request is successful and error is displayed on
the modal otherwise.
Fixes#18091.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.
This change will enable us to test users with escape characters in
their names.
I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
This logic likely never ran due to a combination of bugs.
* Running `maybe_update_markdown_engines` unconditionally meant that
`if md_engine_key in md_engines` was likely always true.
* Introduced in 65838bb: DEFAULT_MARKDOWN_KEY could never be in
md_engines, so should we have ever reached that code path, we'd have
tried to rebuild all markdown engines every time.
And it also wasn't clearly helpful -- because we fetch all linkifiers
for a realm on every request anyway, we don't really save database
queries by doing a bulk fetch on startup, and doing so would likely
result in a material regression to Zulip's overall startup time that
we were creating markdown engines for large numbers of realms in bulk
during process startup.
This removes strict_optional = True and warn_no_return = True (which
are already the default), and adds disallow_incomplete_defs = True and
warn_unused_configs = True, as well as several options that need to
remain False for now.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When a user is muted, in the same request,
we mark any existing unreads from that user
as read.
This is done for all types of messages
(PM/huddle/stream) and regardless of whether
the user was mentioned in them.
This will not break the unread count logic
of the web frontend, because that algorithm
decides which messages to mark as read based
only on the pointer location and the whitespace
at the bottom, not on what messages have already
been marked as read.
Messages sent by muted users are marked as read
as soon as they are sent (or, more accurately,
while creating the database entries itself), regardless
of type (stream/huddle/PM).
ede73ee4cd, makes it easy to
pass a list to `do_send_messages` containing user-ids for
whom the message should be marked as read.
We add the contents of this list to the set of muter IDs,
and then pass it on to `create_user_messages`.
This benefits from the caching behaviour of `get_muting_users`
and should not cause performance issues long term.
The consequence is that messages sent by muted users will
not contribute to unread counts and notifications.
This commit does not affect the unread messages
(if any) present just before muting, but only handles
subsequent messages. Old unreads will be handled in
further commits.
This commit defines a new function `get_muting_users`
which will return a list of IDs of users who have muted
a given user.
Whenever someone mutes/unmutes a user, the cache will be
flushed, and subsequently when that user sends a message,
the cache will be populated with the list of people who
have muted them (maybe empty).
This data is a good candidate for caching because-
1. The function will later be called from the message send
codepath, and we try to minimize database queries there.
2. The entries will be pretty tiny.
3. The entries won't churn too much. An average user will
send messages much more frequently than get muted/unmuted,
and the first time penalty of hitting the db and populating
the cache should ideally get amortized by avoiding several
DB lookups on subsequent message sends.
The actual code to call this function will be written in
further commits.
This makes it so that RealmAuditLog entries are
created when a user mutes/unmutes someone.
We don't really need to store the time, but we
do so anyways, because the `event_time` field
is currently a non-nullable one in the `RealmAuditLog`
model, and making it nullable would risk allowing
not specifying the time in other more important
code which also creates `RealmAuditLog` entries.
This also fixes an incorrect test of successfully
unmuting with the API. Earlier it did not mock
the time in the `views/muting.py` code to return
`mute_time`.
The possible errors had already been documented in
`zulip.yaml` in 3bfcaa3968.
I had missed adding some of them to the markdown files
in the API docs. This commit fixes that.
Commit 4a3ad0d introduced some extra stream-level parameters
to the `realm` object. This commit extends that to add a
max_message_length paramter too in the same server_level.
Previously, you had to request the `stream` event type in order to get
the stream-level parameters; this was a bad design in part because the
`subscription` event type has similar data and is preferred by most
clients.
So we move these to the `realm` object. We also add the maximum topic
length, as an adjacent parameter.
While changing this, we also fix these to better match the names of
similar API parameters.
Previously, clicking the codeblock copy button also
triggered the compose box to open.
This is because of the fact that ClipBoard.js lacks the
ability to stopPropogate the event when clicked.
Rectified by explicitly defining the copy click event
within `is_clickable_message_element`, which disallows
the triggering of reply box.
Effectively a workaround for
https://github.com/zenorocha/clipboard.js/pull/475.
Fixes#17861.
We'll get the same result by letting jQuery stringify these; so
explicit JSON encoding here is likely to just encourage contributors
to cargo-cult JSON-encode strings in the future.
* Don't require strings to be unnecessarily JSON-encoded.
* Use check_capped_string rather than custom code for length checks.
* Update frontend to pass the right parameters.
With a much simplified populate_data_for_request design suggested by
Anders; we only support a handful of data types, all of which are
correctly encoded automatically by jQuery.
Fixes part of #18035.
The comment explains the reasoning, but this should make it a lot
easier for users to visually focus on the menu items in the sidebar.
Changed by tabbott to use the same width as the right sidebar itself.
Inspired by #17169.
I have suppressed errors for github.com by adding an function to
exclude domains as well as urls; this is necessary because GitHub has
marked this tool's User-Agent as a blocker crawler.
I have also suppressed reoccurring url errors that do definitely exist.
Fixes#17928.
This website has been defunct for years, and seems unlikely to return.
I have removed the dead link to citizencodeofconduct.org that
consistently breaks the `tools/test-documentation` test tool.
Ideally, we'd print line numbers etc., but that's complicated because
the `translation.json` format doesn't include them, and usually it's
easy to find strings because you just added them anyway.
But adding a bit more of a `git grep` hint should help.
Fixes#14321.
All streams in left-side bar cause discrepancy
as it is unable to express if it is heading for
the current column as "Streams" is in previous
column or is it text description for the back icon.
Renaming "All streams" to "Back to streams" removes
discrepancy.
`uploads-route.noserve` and `uploads-route.internal` contained
identical location blocks for `/upload`, since differentiation was
necessary for Trusty until 33c941407b72; move the now-common sections
into `app`.
This the only differences between internal and S3 serving as a single
block which should be included or not based on config; move it to a
file which may or may not be placed in `app.d/`.
07779ea879 added an additional `proxy_set_header` of `X-Real-IP` to
`puppet/zulip/files/nginx/zulip-include-common/proxy`; as noted in
that commit, Tornado longpoll proxies already included such a line.
Unfortunately, this equates to setting that header _twice_ for Tornado
ports, like so:
```
X-Real-Ip: 198.199.116.58
X-Real-Ip: 198.199.116.58
```
...which is represented, once parsed by Django, as an IP of
`198.199.116.58, 198.199.116.58`. For IPv4, this odd "IP address" has
no problems, and appears in the access logs accordingly; for IPv6
addresses, however, its length is such that it overflows a call to
`getaddrinfo` when attempting to determine the validity of the IP.
Remove the now-duplicated inclusion of the header.
As zulip is tranfering its tooltip to tippy the
tooltips for subs sort options are tranfered to
use tippy instead of title. Placement is bottom.
Refer https://github.com/zulip/zulip/pull/17434.
If we are changing_hash, it means `window.location.hash` is the
new_hash, so we don't want to change hash further now.
This used to create a loop of changing hashes as we used to close
and open overlays if `hash_before_overlay` was an overlay.
Fixes#18011
This gives us information that browser hash has already changed
and now we are just showing the correct overlay. This can help us
make informative decision that if we want to change the hash back
to the last hash after closing the overlay or not.
Previously, when unmuting a user, we used to make
two database fetches - one to verify that the user
is has been muted before, and one while actually
unmuting the user.
This reduces that to one, by passing around the
`MutedUser` object fetched in the first round.
Since the new function returns `Optional[MutedUser]`,
we need to use a hack for events tests, because
mypy does not yet use the type inferred from
`assert foo is not None` in nested functions like lambdas.
See python/mypy@8780d45507.
Instead of using internal functions for data setup,
we use the API so that these tests are more
end-to-end.
This commit also removes a now unnecessary
`if date_muted is None` check.
This cleans up some code added in 3bfcaa3968.
Also fixes some indentation to be more readable:
- `mock.patch` is in a single line.
- Dictionaries are one field per line.
I have updated the path for the svgo module in the integrations
documentation as the other path would error out as `svgo` wasn't in
PATH, so have updated to use `yarn run svgo` instead.
With a tweak from tabbott to not use tippy on the entire message
content; this was a correct port of the apparent old bootstrap-tooltip
logic; however, that logic didn't actually work, so we just had
`title` behavior.
Given that showing a tooltip when hovering over the entire message
body feels wrong, we preserve the weird title behavior (which also
feels a bit off, but less so).
Since we need to use number values for these breakpoints directly
in some places, having `px` values exported for them is not
great as we have to write weird looking code to convert it into
number.
This was used by the old native Zulip Android app
(zulip/zulip-android). That app has been undeveloped for enough years
that we believe it no longer functions; as a result, there's no reason
to keep a prototype API endpoint for it (that we believe never worked).
This endpoint was needed by the ancient pre-electron desktop app
written in QT; we removed support for that in practice a long time
ago, and even the custom error messages for it in
5a22e73cc6.
So we can delete this endpoint as well.
Long labels like "Deactivate xyz" or "Delete xyz" can
be confusing for translators and it can also create bad
strings that can end with like 4 long words in German.
It is better to have the simple options like "Confirm"
and "Cancel". This commit fixes this issue by changing
the text to "Confirm" in their respective template files.
Fixes#17926.
Long labels like "Unstar messages" can be confusing for
translators and it can also create bad strings that can
end with like 4 long words in German. It is better to
have the simple options like "Confirm" and "Cancel".
This commit fixes this issue by changing such text
to "Confirm" as default in confirm_dialog_modal.
Part of #17926.
Tippyjs automatically places it to bottom.
NOTE: placement of tooltip is changed from 'bottom' to `auto`.
Custom css was set here to avoid tooltip being partially hidden
on small screens. This change automatically takes care of it
by showing the tooltip on right side of message_reaction on
small screens if it is partially hidden from the left or
vice versa.
We no longer need to add_tooltip_to_left_panel_row since
tippyjs.delegate will automatically do that for us.
Tippyjs automatically places it to left on small widths and bottom
for large widhts.
We don't want to import tippyjs module here
along with its dependencies, so we just copied
over tippyjs defaults here. They should be
work fine for /stats page even if we decide to change
defaults for the app in tippyjs and forget to do
it here.
Hiding and instantly showing a modal causes a race condition in
Bootstrap since `hide` and `show` calls are asynchronous.
See https://getbootstrap.com/docs/4.0/components/modal/#methods
Instead, use `overlays.open_modal` which prevents background
mouse events when a modal is present.
This prevents a user from maybe accidentally clicking on another deactivate
button while a modal is present. This also prevents a black screen
caused due to this race condition.
And since a user can't click on the button while the modal is present,
it doesn't make sense to hide it before showing it.
So, remove the `hide` call.
Fixes#17297
We keep the error message same for all cases when a user is not
allowed to subscribe others for all values of invite_to_stream_policy.
We raise error with different message for guest cases because it
is handled by decorators. We aim to change this behavior in future.
Explaining the details in error message isn't much important as
we do not show errors probably in API only, as we do not the show
the options itself in the frontend.
We keep the error message same for all cases when a user is not
allowed to create streams for all values of create_stream_policy.
We raise error with different message for guest cases because it
is handled by decorators. We aim to change this behavior in future.
Explaining the details in error message isn't much important as
we do not show errors probably in API only, as we do not the show
the options itself in the frontend.
We keep the error message same for all cases when a user is not
allowed to invite others for all values of invite_to_realm_policy.
We raise error with different message for guest cases because it
is handled by decorators. We aim to change this behavior in future.
Explaining the details in error message isn't much important as
we do not show errors probably in API only, as we do not the show
the options itself in the frontend.
This color picker did not hide even after exiting stream settings.
Fix by adding logic to auto-close any open color pickers when closing
stream settings.
Tweaked by tabbott to use the existing on-close handler, which is
important if one clicks outside the modal or otherwise navigates
another way.
Fixes#17334.
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit d84727ce7f (#17970) slightly
decreased the apparent size on some platforms depending on which font
was in use before, and some users complained that it was a bit hard to
read. Based on experiments with multiple platforms and monitors and
resolutions, this appears to be a good compromise that increases the
rendered height without increasing the width more than necessary.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes the API documentation tests following recent changes, but
isn't the right solution -- we probably want to change the API itself
to not require this strange JSON-encoding-of-a-string.
But this is necessary to have CI pass.
When the user's full name is long, the full name + `(you)`
in the buddy list starts to truncate, but when hover, the
tooltip displays the full name but not `(you)`.
This commit fixes this by adding `(you)` in the tooltip.
"Narrow to topic" option do exactly the same thing as
clicking the topic name does. So therefore decided to
remove that option from the popover. This commit removes
all the code related to .narrow_to_topic including it's
template code, on click event and a function used for it.
Fixes#18027.
We should disable the stream message retention dropdown for owners
of realm with limited plans. The behavior is correct in other
places, will also just explain the behavior in other places -
For limited plans-
1. Owners can see the dropdown in stream creation form but others
cannot, and in stream privacy modal, the modal is visible to all.
But the modal is disabled in all cases since the realm is on
limited plan.
For standard plans -
1. Owners can see and edit the dropdown in both stream creation
form and stream privacy modal. Non-owners cannot see the modal
in stream creation form but they can see the dropdown in stream
privacy modal and it will be disabled.
Thus, the only change in this commit is to disable the dropdown
in the stream creation form for owners and in limited plan realms.
The other behavior mentioned above was already there.
The comments explain in some detail, but basically we were displaying
the types for booleans incorrectly, and the types for strings in a
somewhat confusing fashion. Fix this with comments explaining the logic.
Using JSON dumping also results in our showing strings inside
quotation marks in our examples, which seems net helpful.
Thanks to ArunSankarKs for finding where we needed to change the
codebase.
Fixes#18021.
There is a bug where invite button is not enabled again after sending
invite is failed, but it is enabled on successful completion of invite.
I am not able to figure out why it is behaving differently in success and
error cases, even when the code to enable button is in complete() function
and it is called in both the cases.
But, I can confirm that adding .button('reset') fixes this bug as button
is disabled using the .button('loading') call and this is according to the
bootstrap docs only. But the bootstrap docs mention that the .button(string)
method has been removed in v4.0 as mentioned here
https://getbootstrap.com/docs/3.4/javascript/.
So, as we would want to avoid using depereceated or removed methods, this
commit simply changes the code to disable the button and set the loading
text using simple jquery and this also solves the above mentioned bug.
The moderator role was not included in the tests for create_stream_policy
and invite_to_stream_policy. The tests are do_set_realm_property_test
in test_events.py and do_test_realm_update_api in test_realm.py.
This should have been added for create_stream_policy in 5b32dcd and
in 5b32dcd for invite_to_stream_policy, but was missed by mistake.
There was a bug where invite users link is shown in the Invitations
section of settings overlay, irrespective if the user is allowed to
invite or not based on the realm settings. So here we just use the
'can_invite_others_to_realm' field of page_params, that was added in
previous commit itself to hide the link accordingly.
Also added the code in server_events_dispatch.js for this field.
Though it doesn't do live update when the overlay is opened similar
to other policies like create_stream_policy, but it ensures that the
visibility of link is correct if the overlay is opened after the
settings has changed.
We should probably do the complete live update in future of the
elements related to such realm settings.
This commit adds backend code for passing can_invite_others_to_realm
field to clients using the fetch_initial_state_data in the page_params
object.
Though this field is not used by webapp as of now, but will be used
to fix a bug of incorreclty showing the invite users option in
settings overlay in the next commit.
We add moderators and full members option to invite_to_realm_policy
by using COMMON_POLICY_TYPES and use can_invite_others_to_realm helper
added in previous commit. This commit only does the backend work,
frontend work will be done in separate commit.
This commit adds can_invite_others_to_realm helper which will be used in
further in next commit when invite_to_realm_policy will be modified to
support all values of COMMON_POLICY_TYPES.
It is important for this commit's correctness that
INVITE_TO_REALM_POLICY_TYPES was initialized to use the same values.
This commit replaces invite_by_admins_policy, which was a bool field,
with a new enum field invite_by_realm_policy.
Though the final goal is to add moderators and full members option
using COMMON_POLICY_TYPES, but this will be done in a separate
commit to make this easy for review.
Our aim is to use this library to remove use of bootstrap-tooltip
for showing popovers and tooltips. This will remove our
dependency on bootstrap for showing tooltips. Thus, bootstrap
can be upgrade more independently.
Since the base hash for org settings and user settings are
different (organization and settings), the hashchange module
gets confused that we are going from one overlay to other.
A reproducer for this flow is to visit the organization "Bots" page,
click on your own profile as the owner of a bot, and then click "Edit
profile" from there.
So, we fix this by making an exception for this particular case
in the module.
Fixes part of #18011.
Displays "Several people are typing..." when more than 3 users
are typing to avoid typing notifications in streams being too noisy.
A side effect is it shows the same message in pms too.
This commit implements a subtle optimization (described in more detail
in the comment) that can save a few hundred milliseconds in when the
sender sees that their message has sent when sending to very large
streams.
Fixes#17898.
The tests for can_create_streams and can_subscribe_other_users shares a
lot of code and we deduplicate the code by extracting most of the code
as check_has_permission_policies which will now be called by the two
tests test_can_create_streams and test_can_subscribe_other_users.
This will also help in avoiding the duplication of code when we will
convert more policies to use COMMON_POLICY_TYPES.
We send the whole data set as a part of the event rather than
doing an add/remove operation for couple of reasons:
* This would make the client logic simpler.
* The playground data is small enough for us to not worry
about performance.
Tweaked both `fetch_initial_state_data` and `apply_events` to
handle the new playground event.
Tests added to validate the event matches the expected schema.
Documented realm_playgrounds sections inside /events and
/register to support our openapi validation system in test_events.
Tweaked other tests like test_event_system.py and test_home.py
to account for the new event being generated.
Lastly, documented the changes to the API endpoints in
api/changelog.md and bumped API_FEATURE_LEVEL.
Tweaked by tabbott to add an `id` field in RealmPlayground objects
sent to clients, which is essential to sending the API request to
remove one.
Similar to the previous commit, we have added a `do_*` function
which does the deletion from the DB. The next commit handles sending
the events when both adding and deleting a playground entry.
Added the openAPI format data to zulip.yaml for DELETE
/realm/playgrounds/{playground_id}. Also added python and curl
examples to remove-playground.md.
Tests added.
This endpoint will allow clients to create a playground entry
containing the name, pygments language and url_prefix for the
playground of their choice.
Introduced the `do_*` function in-charge of creating the entry in
the model. Handling the process of sending events which will be
done in a follow up commit.
Added the openAPI format data to zulip.yaml for POST
/realm/playgrounds. Also added python and curl examples for using
the endpoint in its markdown documented (add-playground.md).
Tests added.
Tweaked exports.py to add the config object there so that our export
tool can include the table when exporting. Also includes all the
changes required to import the new table from the exported data.
Helper function `get_realm_playgrounds` added to fetch all
playgrounds in a realm.
Tests amended.
Realm administrators already get creation and deletion events for all
streams, including private streams. So these should be reflected in
the initial state data.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Muted users are stored in a map with key as user ID and
the value as the timestamp of muting.
Names can be easily fetched from existing functions
in `people.js` and hence not stored.
Adds backend code for the mute users feature.
This is just infrastructure work (database
interactions, helpers, tests, events, API docs
etc) and does not involve any behavioral/semantic
aspects of muted users.
Adds POST and DELETE endpoints, to keep the
URL scheme mostly consistent in terms of `users/me`.
TODOs:
1. Add tests for exporting `zulip_muteduser` database table.
2. Add dedicated methods to python-zulip-api to be used
in place of the current `client.call_endpoint` implementation.
Because the logic in print_listeners doesn't have access to computed
settings in dev_settings.py, we need to duplicate the special
IS_DEV_DROPLET logic for computing the default hostname.
There's still a secondary problem that this URL 404s.
It does not seem like an official version supporting Webpack 4 (to say
nothing of 5) will be released any time soon, and we can reimplement
it in very little code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`call_on_each_event` now supports additional params other than
`event_type` and `narrow`; Ex: `all_public_streams` to fetch events
of all public streams.
Also add a bit of explanation of how this parameter works.
Fixeszulip/python-zulip-api#647
Within 9c9d74fd6d, page_params.is_realm_admin was incorrectly
stated while building popover which lead to admin actions not
being populated within popover.
This is so beacuse `is_realm_admin` is not a valid object
of page_params.
Rectified by renaming `is_realm_admin` to `is_admin`.
This is a prep change to eventually completely
replace the term "filter" with "linkifier" in
the codebase.
This only renames files. Code changes will be
done in further commits.
This renames the test file for muting to have
the term `topic` in it, along with an ambiguously
named helper.
This is a prep change for implementing the mute
users feature.
This is a prep commit for implementing mute users feature, which
renames this function to be more specific. This function cannot be
used as-is for user mutes, because for user-muted messages, we always
want to rerender the message list, irrespective of what
`excludes_muted_users` is.
In muting_ui.js, we also remove an unnecessary conditional that is
already handled by update_topic_muting_and_rerender itself.
Previously, the recent-topics view did not update when the webapp
received `muted_topics` events.
The final state was correct **only** on the client which was used to
mute/unmute the topic, because we update the UI even before sending
the request to the server to mute/unmute the topic.
This commit fixes that by rerendering the recent-topics table when the
client receives `muted_topics` events. While doing so can be
expensive, it is likely unavoidable, because we may want to even
remove the topic from the recent-topics table, and we don't know
exactly which topic was affected (we just get an updated list of all
muted topics from the event).
Even though rerendering is expensive, it should not affect the user
experience, because a rerender will be trriggered only in the clients
which did not do the (un)muting (and hence, the user was probably not
interacting with these clients when the event was received). The
`last_topic_update` variable makes sure that this is the case.
Previously, on clicking the "bell" icon from a recent
topics topic row, two click handlers were triggered-
1. The one for the bell in the recent-topics topic
row.
2. The one for the bell in the floating recipient bar.
This is intended to be triggered only when the user
clicks on the bell icon in the floating recipient
bar to mute a topic, but this was triggered on
clicking the recent-topics topic row bell too,
because it wasn't specific and we use the same
HTML class for both the bells.
Because both these click handlers were triggered,
the webapp sent back-to-back duplicate requests to
the server to mute the topic, which caused the sever
to throw "HTTP 400: Topic already muted" error.
This commit fixes this by making the recipient bar
click handler more specific, so that it is triggered
only when the recipient bar bell icon is clicked.
There is also the issue that the recipient bar bell
icon only allows a user to mute the topic, not unmute.
So, when someone mutes a topic, and then clicks on
the recipient bar bell icon, the webapp sends a
request to mute the topic again (not to unmute),
leading to the same error as above.
This commit does not fix that. See #15223.
Added a `include_current_item` parameter to dropdown_list_widget
which behaves as follows:
- Has a default value of `true`, which includes the current value
in the dropdown list items.
- If set to false, it excludes the current value from dropdown
list items.
This fixes a bug I introduced when merging
86eccfd20f, by not using the "new topic
button" trigger.
It would probably be wise to replace that "trigger" system with just
passing actual options, as it's pretty confusing.
This commit adds an option to create new topic in stream
popovers. Then adds a click listener on them which triggers
the compose start function with the stream specified.
Fixes#13480.
Fixes#2507.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.
We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
Previously, if a user subscribed to a stream with
history_public_to_subscribers, and then was looking at old messages in
the stream, they would not get live-updates for that stream, because
of the structure in how notify_reaction_update only looked at
UserMessage rows (we had a previous workaround involving the
`historical` field in `UserMessage` which had already made it work if
the user themselves added the reaction).
We fix this by including all subscribers with history access in the
set of recipients for update events.
Fixes a bug that was confused with #16942.
This change should make live-update code less brittle,
or at least less cumbersome.
Instead of having to re-compute calculated fields for
every change to a stream message, we now just compute
the fields right before we render stream settings UI.
This is mostly a pure code move.
In passing I remove an unneeded call to
update_calculated_fields in the dispatch code,
plus some tests that don't need them.
The current documentation doesn't clearly explain the process of
changing the organization's subdomain.
This commit adds instructions for user to get the string_id of their
organization and then lists the commands to be executed in the
management shell to change the subdomain.
Part of #17857.
This maps pageup/pagedown/home/end to close compose when used in empty
compose box, matching the existing behavior for the Up/Down arrow keys.
Currently, these keys do nothing when used in an empty compose box.
Typically, a user intends to navigate when pressing these keys (and
with empty compose, they can't be expecting to navigate within the
compose box), so it makes sense to map them to navigate the message
feed just to save users from needing to hit `Esc` in these contexts.
Fixes#17917
The `copy_handler` function that this shortcut calls is not useful
unless the body of a Zulip message is selected, so we shouldn't try
running it in other situations.
This logic correctly prevents the hotkeys implemented below it from
being active when "Recent topics" is open. We expect to change some
parts of that soon (see #17685), but in any case, we should always
return false in the hotkey code when we don't process a key, so that
default browser behavior works.
This fixes browser shortcuts like Ctrl+C, Ctrl+V, etc. when recent
topics is loaded.
This bug was introduced in 1eafb1d8b3.
Thanks to ganpa3 for noticing this bug.
This sorts the members imported within each individual declaration; we
use import/order for sorting multiple declarations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Amazon SES has a limit on the size of address fields, and rejects
emails with too-long "From" combinations of name and address. This
limit is set to 320 bytes and comes from an RFC limitation on the
size of addresses. This RFC standard states that an email address
should not be composed of a local part (before the '@') longer than
64 bytes and a domain part (after the '@') longer than 255 bytes.
It is possible that Amazon SES misinterprets this limitation as it
checks the length of the combination of the name and the email
address of the sender.
To ensure that this problem is not encountered in the send_email
module of Zulip the length of this combination is now checked
against this limit and the from_name field is removed to only
keep the from_address field when it is necessary in order to
stay below 320 bytes.
If the from_address field alone is longer than 320 bytes the
sending process will raise an SMTPDataError exception.
Tests for this new check are added to the backend test suite in
order to test if build_email correctly outputs an email with filled
from_name and from_address fields when the total length is lower
than 320 bytes and that it correctly throws the from_name field
away when necessary.
Fixes: #17558.
"Copy link to message" is both a clearer label, and also more likely
to be what someone is looking for. (It also avoids adding the
"conversation" terminology to the product).
The previous label was also somewhat inaccurate, because it actually links
to a specific message in the conversation.
We're renaming "stream deletion" language to "stream archiving"
and these pages were moved in the process, so we should keep redirects
for them for a while.
`expires_in` (remaining time before the invite expires) should
be calculated from the time at present, not from the time when
confirmation link was sent.
Added emojis and Non ASCII characters to default
stream names and descriptions.
Added raw_emojis array under streams so that
we can pass --extra-streams argument without
--extra-users as it should be.
It looks like folks are accidentally generating requests for new
country-specific languages when they sign up. This change clarifies
the instructions so that we avoid these requests.
From the commit history, this typo has always been there; because it
had the same priority as the `opacity: 0.5` for that element,
it can be nondeterministic whether the bug appeared.
Fixes#17476.
This reverts commit 9c6d8d9d81 (#16916).
This feature has known bugs, and also wants some design changes to
make it customizable like linkifiers, so we’re retargeting this to
post-4.x.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The Session middleware only adds `Vary: cookie` if it sees an access
to the from inside of it. Because we are effectively, from the Django
session middleware's point of view, returning the static content of
`request.saved_response` and never accessing the session, it does not
set `Vary: cookie` on longpoll requests.
Explicitly mark Tornado requests as varying by cookie.
For filter values which don't exist or are invalid in some
way, we return false to show user that there are no messages
in the filter user is trying to render. Our previous behaviour
was to show all the messages and ignore the filter which
isn't good.
Adding an additional `!` to the stream name each time a stream is
deactivated, to a maximum of 21 times, effectively limits number of
times a stream with a given name can be deactivated. This is unlikely
to come up in common usage, but may be confusing when testing.
Change what we prepend to deactivated stream names to something with
more entropy than just `!`, by instead prepending a substring of hash
of the stream's ID. `!`s. Using 128 bits of the hash means that it
will require more than 10^18th renames to have a 1% chance of collision.
Because too-long stream names are also truncated at 60 characters,
having this entropy in the beginning of the name also helps address
potential issues from stream names that differed only in, e.g. the
60th character.
Fixes#17016.
The popup that appears when you mute a topic is a bit hard to read,
since nothing makes the topic and stream names jump out from the rest
of the paragraph. Fix this by using bold around the stream/topic and
also cutting a bunch of unnecessary verbiage.
Tweaked by tabbott to further simplify the language.
The Available flags table in update_message_flags.md was using
markdown for certain content inside HTML, which resulted in the
table not being rendered properly.
This commit fixes the table to use proper HTML for content rendering
instead of markdown, since the table was written in markdown's HTML
syntax.
This reverts commit a00f5dd90e (#17801).
That commit introduced a regression in the portico pages as described
in commit 85b3157b47. Since that fix
introduced a regression of its own, we need to revert both commits for
now.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reverts commit 85b3157b47.
This broke the × button on Blueslip alert boxes, because @extend does
not work across different PostCSS compilation units.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The start time of the last password change was the wrong time to use,
because we could start a password change, start another request,
finish the password change, and then observe that the other request
failed due to the password change.
We could use the end time, but a counter is more robust to
sub-millisecond race conditions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
* We shouldn't use "Search" when we mean "Filter". Filter is correct
here, since we are just showing a subset of what's otherwise shown,
and won't find anything that's older (or whatever).
* The stream/topic wording was unnecessary; the things we're filtering
are topics (E.g. "Filter users" might look at name/email, and is the
right label, not "Filter name/email").
We should still display the `source` advice when not in Vagrant or a
Droplet, since that's an important hint for anyone using local
installation on Linux.
We move the "If you are using Vagrant..." text a bit after to
highlight things nicely for folks who are running tools outside
Vagrant.
Also tighten text to avoid line-wrapping on an 80 character console.
* repo => repository for more standard language.
* Delete 3 repeats of explaining the origin/upstream/local.
* Add some links.
* Update `git pull` language for rebase changes.
Instead of validating `op` value later, this commit does that
in `REQ`.
Also helps avoiding duplication of this validation when
stream typing notifications feature is added.
The only downside of this is that it makes it harder to control the
order of these tests; which isn't that important. And the structure
of naming each with its test order fundamentally requires renaming
files when adding/deleting tests, so if we want to control the default
test order, we'd be better off doing that by just hardcoding a list in
the test runner code.
Previously we were liable to have false positives in our tests here
because we did not reset the visible state for these selectors, this
commit adds a helper and relevant calls to it in order to prevent such
false positives.
There can be several cases when we don't require a reset button
with the dropdown_list_widget.
Hence, Added an abstraction for dropdown_list_widget that
renders the button only if it's corresponding text is passed.
These flags were put in place in the first commit that introduced
Tornado (9afd63692f) with unclear
utility.
Remove them, since they have never been documented, and do not have a
clear need.
The `X-Forwarded-For` header is a list of proxies' IP addresses; each
proxy appends the remote address of the host it received its request
from to the list, as it passes the request down. A naïve parsing, as
SetRemoteAddrFromForwardedFor did, would thus interpret the first
address in the list as the client's IP.
However, clients can pass in arbitrary `X-Forwarded-For` headers,
which would allow them to spoof their IP address. `nginx`'s behavior
is to treat the addresses as untrusted unless they match an allowlist
of known proxies. By setting `real_ip_recursive on`, it also allows
this behavior to be applied repeatedly, moving from right to left down
the `X-Forwarded-For` list, stopping at the right-most that is
untrusted.
Rather than re-implement this logic in Django, pass the first
untrusted value that `nginx` computer down into Django via `X-Real-Ip`
header. This allows consistent IP addresses in logs between `nginx`
and Django.
Proxied calls into Tornado (which don't use UWSGI) already passed this
header, as Tornado logging respects it.
The `widget_content` key is expected to contain a string which parses
as JSON; in the event that it does not, log the error and notify the
bot owner, instead of failing silently.
Fixes#16850.
Added non ascii and non bmp characters to stream names.
A Stream Name will now consist of a random stream name +
a number (to avoid name duplicates) + a 15% to contain
a emoji.
Added non ASCII and non bmp characters to full name.
Created a new list for non_ascii_names and emojis
to store them explicitly.
A full name will now consist of first name +
(a non ASCII name or a plain middle name) + (a emoji
or a plain last name).
First name will not have any non ASCII or non bmp text
as it is also being used as email.
This commits adds on to 9884226f, which was added to
handle a rare race condition that occurs when the
session hash is not updated by the backend during the
password change process.
It handles a variant race situation where the request was initiated
before/during the password change event and completed after it was
completed. Hence, forcing the page to redirect to the login page.
In a00f5dd90e, we needed to move the
`alert-box` styles from alerts.css to be visible in portico pages.
However, when doing so, we incorrectly moved all of alerts.css, which
also has styles for `alert` and` alert-error` designed to make it
convenient to include hidden elements for potential errors in the
webapp settings UIs directly in the HTML template (and then use
show/hide to manage them).
We fix this by moving just the alert-box scope to the common
components.css module, which is designed as the place for styles
shared between the webapp and portico pages.
This fixes an issue where the error messages for wrong password and
the like were invisible :(.
In `validate_account_and_subdomain` we check
if user's realm is not deactivated. In case
of failure of this check, we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of users with deactivated realms for non-browser
clients.
So we register a new REALM_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises RealmDeactivatedError if user's realm
is deactivated.
This error is also documented in
`/api/rest-error-handling`.
Testing: I have mostly relied on automated
backend tests to test this.
Fixes#17763.
In validate_account_and_subdomain we check if
user's account is not deactivated. In case of
failure of this check we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of deactivated accounts for non-browser clients.
So we register a new USER_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises UserDeactivatedError if user's account
is deactivated.
This error is also documented in
`/api/rest-error-handling`.
Testing: I have mostly relied on automated
backend tests to test this.
Partially addresses issue #17763.
This commit make compose_ui.autosize_textarea handle most of the autosize
logic of the textarea. It audits for any logic that is trying to do
autosize manually and replace it with compose_ui.autosize_textarea.
This allows to have better check for when the textarea is autosized.
Mainly done this so that we can have a check on #compose-textarea when
to autosize and when not to, thus helping to have all the logic in only
one function.
This commit changes some fragile selectors (like
`a[href=#link]`) to more stable selectors because they
are more prone to break from doing something normal
like adding another link in the app.
It also solves an inconsistency in `07-navigation.ts`,
where the subscription overlay was opened by clicking
on the header stream instead of the menu list.
It also fixes a rare flake (in `07-navigation.ts`), where
the close button of subscription overlay was not clicked
due to a delay in the opening. The delay was caused by
clicking the header stream to open subscription overlay
which caused unnecessary loading of the stream
setting(Verona).
As we are using the 'navigate_to' function to navigate
the links on the left sidebar, It'd be more clear to rename
the function to 'navigate_using_left_sidebar'.
Also adding '#left-sidebar' when selecting the element,
to be sure it will select the element from the left sidebar.
We recently added the commit to add the log-out call
after each test (52706908b).
This commit cleans that approach by using
just one log-out call after the test is executed at
`common.ts`
We now unconditionally enable backgroung events when 'hidden.bs.modal'
event is triggered on closing of modal. We do not need to handle them
separateley for closing modal by close_modal, data-dismiss or escape.
We handle this by single handler for modals in settings and subscription
overlay.
Fixes#16688.
This commit adds waitForFunction to wait till the background mouse events
are enabled after closing the modal in the settings test.
This change is needed to avoid the failure that will be caused after we
change the code to handle re-enabling of mouse events only at one place
using 'hidden.bs.modal' event of bootstrap, as this event is fired only
after the modal is completely hidden, and we would want the mouse events
to be enabled before using clicks in further tests.
This commit changes the code to not remove the modal element from DOM
after closing the modal for the deactivation stream modal and stream
privacy modal.
This is a prep commit for enabling background mouse events
unconditionally using 'hidden.bs.modal', because removing element
using '.remove' remove all events attached to it and this will also
remove the 'hidden.bs.modal' event which we do not want.
This remove behavior is inconsistent as we remove some of the modals
but do not remove some, so for now we are not removing the modal
after closing but it is anyway removed before opening a modal to handle
case of having two elements of same id and avoids any bugs.
This behavior of when to remove the element from DOM and when to not
remove needs to be discussed and may be modified in future.
We show a modal as a warning when unsubscribing a private stream
because it is a irreversible action and one cannot re-subscribe
tovit until added by other member of stream.
Fixes#9254.
This commit adds "close-modal-btn" class to cancel button in
the modal and cross icon.
We do this change because we would add a modal for unsubscribing
from private stream in further commit using confirm_dialog module.
We would need to avoid the unexpected closing of stream settings
on closing the modal which can be done by calling 'e.stopPropagation'
to prevent propagating of events in other elements.
Thus, adding this class will mean that the handler used for stream
privacy modal for this same task will be used for unsubscribe modal
also.
This commit adds 'close-modal-btn' class to cancel button and cross icon
in the deactivation stream modal.
This change is a prep commit for enabling background events on
'hidden.bs.modal' event. As we would enable background events in furhter
commit using the 'hidden.bs.modal' event, we would need to remove the
'hide.bs.modal' event of deactivation_stream_modal which removes the
element from DOM.
When we remove this event we would need a e.stopPropagation call to avoid
unexpected closing of stream settings which was not a problem previously
because the element was being removed from DOM before actual closing of
modal.
So instead of adding a different handler, we can use the handler used
for stream privacy modal here by adding this new class.
This commit renames the class of both cancel button and the cross
icon to close-modal-btn.
This change is a prep commit for enabling background events on
'hidden.bs.modal' event. As we would enable background events in
furhter commit using the 'hidden.bs.modal' event, we would need to
remove the 'hide.bs.modal' event of deactivation_stream_modal which
removes the modal element from DOM.
When we remove this we would need a e.stopPropagation call to avoid
unexpected closing of subscription settings, which was not a problem
before as the element was removed from DOM before the actual closing
of modal.
So instead of adding a separate `e.stopPropagation' call, we can use
the same handler that is being used for stream privacy modal and this
is the reason the class name of cancel button of privacy modal is
being changed.
We do not remove the stream row instantly from the subscribed list in
subscription overlay when unsubscribing from public streams in most
of the cases but we do so when unsubscribing using hotkey.
This commit makes it consistent by not removing the stream row on
unsubscribing using hotkey.
We also not remove the 'active' class as streams settings is still
open in the right seciton and this behavior is also consistent with
the other ways of unsubscribing.
Note that this behavior is only for public streams, we remove the
stream row in case the stream is private, as user cannot
resubscribe himself and this behavior is consistent across all ways
of unsubscribing.
- Rectifies broken label tag having a misleading 'for' attribute.
- Removed 'name' attribute from unlabelled span tag.
- Removed label expression from DropdownListWidget to built an,
abstraction for control group only.
Fixes#17311.
Restructured dropdown_list_widget template to unwrap label tag
from the control group, hence defaulting the edit_bot
dropdown items to their original text size.
We add a TUTORIAL_ENABLED setting for self-hosters who want to
disable the tutorial entirely on their system. For this, the
default value (True) is placed in default_settings.py, which
can be overwritten by adding an entry in /etc/zulip/settings.py.
Updated database query to filter out deactivated streams from the
return of the get_topic_mutes method. Added optional
include_deactivated parameter to the method to make the behavior
default but overrideable. Added test case in test_muting for these
changes. Fixes blueslip warnings thrown by muting.js set_muted_topics
when passed deactivated streams via page_params.
We now consistently set the PM counts for the right
sidebar toggle in unread_ui, similar to what we
do for the overall counts in the left sidebar toggle.
(Use a thin window to see the code in action.)
This breaks a dependency cycle.
In passing I improve the test coverage for the
actual job that pm_list still does (updating its
own total count in the "Private Messages" section).
With the previous two commits deployed, we're ready to use the
denormalization to optimize the query.
With dev environment db prepared using
./manage.py populate_db --extra-users=2000 --extra-streams=400
this takes the execution time of the query in
bulk_get_subscriber_user_ids from 1.5-1.6s to 0.4-0.5s on my machine.
The new comment explains the issue in some detail, but basically if we
deactivate the bots first, then an error partway through is corrected
by a retry; if we deactivate the user first, then we may leak
undeactivated bots if a failure occurs.
This adds the is_user_active with the appropriate code for setting the
value correctly in the future. In the following commit a migration to
backfill the value for existing Subscriptions will be added.
To ensure correct user_profile.is_active handling also in tests, we
replace all direct .is_active mutation with calls to appropriate
functions.
These procedures should be done atomically overall, with the exception
of the code that sends events to avoid block if there's a delay
communicating with Tornado.
We add the savepoint=False on underlying function that already
executes inside an atomic context - to avoid the overhead of creating
savepoints where they aren't needed.
This commit adds a new option of STREAM_POST_POLICY_MODERATORS
in stream_post_policy which will allow only realm admins and
moderators to post in that stream.
We extract a helper which checks whether to allow the sender to send the
message to a stream according to the stream_post_policy. The purpose
of extracting it out is to avoid additional code for checking the access
for bot owners in case of bot sending the messages and instead calling
the handler two times - one time for sender and one time for bot owner if
sender is a bot.
The moderators-only option was actually added in the previous
commit for create_stream_policy as we use the same function
'has_permission' for both the policies. But we add the error
handling code and tests for moderators-only option in this
commit.
This commit modifies the has_permission function to include
realm moderator role. Thus this adds a new option of moderators
only for create_stream_policy.
Though this automatically adds this option for invite_to_stream_policy
also, but we will keep other code for showing error and for tests
in a separate commit.
This commit adds an assert statement in the last block of
has_permission which checks whether the policy_value is
POLICY_FULL_MEMBERS_ONLY. This assert statement is added
for readability.
For messages which we don't have stored locally, we don't update
our data structures. This was actually our behaviour before
59e5f2d8fc, which introduced this
bug.
Not doing this caused a major bug where we ran into errors
moving messages for topics for which we didn't have all
the messages available.
The current API documentation uses call_endpoint to upload a file;
since we've added a custom helper in python-zulip-api, we should
document that cleaner approach.
This data structure has never been one that we actually render into
the DOM; instead, its role is to support clicking into view that
contain muted streams and topics quickly.
This downgrade makes that situation much more explicit, and is also
useful refactoring to help simpify the upcoming changes in #16746.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The session object provides a common place to set headers on all
requests, no matter which implementation.
Because the `headers` attribute of Session is not a true static
attribute, but rather exposed via overriding `__getstate__`, `mock`'s
autospec cannot know about it, and thus throws an error; in tests that
mock the Session, we thus must explicitly set the `session.headers`.
The existing organization, of returning an opaque blob from
`build_bot_request`, which was later consumed by
`send_data_to_server`, is not particularly sensible; the steps become
oddly split between the OutgoingWebhookWorker, `do_rest_call`, and the
`OutgoingWebhookServiceInterface`.
Make the `OutgoingWebhookServiceInterface` in charge of building,
making, and returning the request in one method; another method
handles extracting content from a successful response. `do_rest_call`
is responsible for calling both halves of this, and doing common error
handling.
The comments in stream-policy tests in test_message_send.py specifies
the restriction of creating streams based on stream_post_policy. But
this restriction was removed in 9aaa61963 and we now allow everyone to
create all type of streams. So this commit fixes the stream creation
parts in comments.
The STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS option of stream_post_policy
was explained as "Only new members can post" in the api docs. It should
instead be "Only full members can post" and this commit fixes it.
This commit fixes the 'Your account' settings page to show
correct role of organization owner. Previously it was
incorrectly displayed as administrator.
Commit message tweaked by sahil839.
I lift this function out of message_store to
break some dependencies, and it's also more
consistent with the rest of the codebase:
alert_words.process_message
pm_conversations.process_message
recent_topics.process_messages
recent_senders.process_message_for_senders
We can do further cleanup to make these names
consistent (and possibly have them all work in
bulk), but that's out of the scope of the current PR.
We move the message_store.add_message_metadata function
(and all its dependencies) into a new module called
message_helper and rename the function to process_new_message.
(It does a bit more than adding message metadata, such
as updating our message store.)
We also have a "protected" interface now in message_store,
so that message_helper can access the message store:
update_message_cache
get_cached_message
Because update_message_cache is identical to
the former create_mock_message, we just renamed it
in the tests.
Most callers should use these functions:
message_helper.process_new_message (setting)
message_store.get (getting)
It's slightly annoying that the setter interface
is in a different module than the getter interface,
but that's how you break a bunch of dependencies.
We also extract the tiny message_user_ids class:
user_ids()
add_user_ids()
All the code moves here are pretty trivial, and
the code that was moved maintains 100% line
coverage.
The module name `message_helper` is not ideal, but it's a single
function and it'll save time to just do the topology change now and
leave thinking through the right name to later.
We had used 2>&1 to redirect stderr to stdout so it could be piped
into ts, but commit dd3cdd6ec5 (#17611)
removed ts, so we no longer need the redirection.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
As the hotkey might cause the page to be scrolled, thus capturing
the wrong area, I view the message by id instead of using
'#narrow/near' and remove the 'selected_message' class from the
message box to remove the selection highlight on it.
Fixes: #17878
This function should be named similarly to the well-named
end_inline_topic_edit; previously the main hint that this was the
inline code path was just that the recipient_row was the parameter.
When migrating from Casper to Puppeteer, some tests
were missed for adding log-out calls at the end.
This commit adds log-out calls to those missed tests.
Also, As we are resetting the test database after each
run (from 99f8be6a12) it will better to log out because we are
not resetting Tornado's internal state. It'd help us prevent
any future flakes.
This likely still needs work on updating the list of highlights, as
well as an editing pass, but we shouldn't need to read the whole
`git log --stat` again.
This helps us reduce time to update dependencies on every CI
build since the previous containers used to take about 1 minute.
`sudo` had a bug due to which we were not able to create directories.
See https://github.com/sudo-project/sudo/issues/42.
We used these directories to restore caches.
Upgrading the focal dependencies via this commit naturally fixes that
bug.
Fixes#17854
The /events and /register endpoints were excluded from schema validations,
because they were earlier not completely documented. However, they can
now be added for proper checking. Removed them from excluded endpoints list
and fixed the documentation for /register and /events after the checking.
Fixes#17796.
Backend logic for handling user mention was cluttered
because it was handled at two stages first in
get_possible_mentions_info while fetching mention data
based on the messsage and then later in UserMentionPattern
which handles processing of text for mention.
Ideally UserMentionPattern should depend on
get_possible_mentions_info only for data but there was a
shared logic between these two that made it hard to debug
any possible bugs.
Updates in this commit make both of these functions
coherent in terms of logic and also add appropiate
comments to improve readability of these functions.
There was also a hidden bug that if a user A is
mentioned in with @**name|id** then @**invalid|id**
again mentioned A because of the way we handled mentions
earlier. It is solved as a result of this refactor and
appropiate test has been added for this.
This has been tested manually as well as by adding new
test to address missing case.
I have updated the `Documenting an Integration` documentation to be more
accurate on the usage of `tools/generate-integration-docs-screenshot` as
well as the requirement to fill out the `DOC_SCREENSHOT_CONFIG` section
of `zerver/lib/integrations.py` and a few other minor changes.
Before this we did not have remove event in server_events_dispatch.js
for the user group delete event even though server had. This was
leading to blueslip errors. Extracted the logic which was used in
success() of channel.del for user_groups into the remove case in
server_events_dispatch. Also removed the redundant reload call as
we already do that in server events.
We support Debian as an OS for setting up the Zulip server. But the CI
does not run on pull request to test the setting up of the server on
Debian. Hence, add the check to CI.
This is a mostly verbatim extraction.
I re-phrased one line of code to work around a lint
false alarm. (Look for `preamble` in the diff.)
There are about 8 lines missing coverage here, so
the new module might be a good candidate to get
100% line coverage on.
Before this change, you would need to remove 74
edges from our dependency graph to make it
acyclic. Now it's 72.
This change does not impact the overall complexity
of our dependency graph (at least in terms of the
number of edges that we would need to remove to get
a tree), but it does clarify the picture a bit.
Commit 0200f48a12 (#17407) removed the
navigate global variable. Use the K hotkey instead of evaluating
navigate.up().
Signed-off-by: Anders Kaseorg <anders@zulip.com>
I noticed this because the test_events.py tests had the extremely
weird pattern of calling the actual change function, and then testing
the `notify` function's state changes (which should always be noops),
rather than actually testing the state change function.
Fixing the test made it clear that the actual logic in events.py
simply did not handle deleting custom_profile_field_value elements
from user objects when a custom_profile_field object was deleted.
So we fix that bit of logic as well.
It appears this bug was unique -- at least we don't have any other
notify_* functions being used directly in test_events.py, and the
handful of state_change_expected=False entries are all events for data
not present in page_params.
* `op` (operation) field, added in f6fb88549f, was never intended for
`custom_profile_fields` event. This commit removes the `op` as it doesn't
have any use in the code.
* As a part of cleanup, this also eliminates the schema check warnings
for `custom_profile_fields` event, mentioned in #17568.
Rename few functions referencing the "CHOICE" field to
instead use the new "SELECT" name. This is done so that they
can be reused in "SELECT_MULTIPLE" field.
For pages that don't have page_params, the default_page_params now
ensures that debug_mode will correctly follow settings.DEBUG.
This allows blueslip exception popups to work on portico pages for
development environment.
Fixes: #17540.
In 1a12e112d9, this page was converted
to use portico styling, but we intentionally left this page not using
the portico_content class since we didn't want the header/footer.
We still don't want the header/footer clutter, so instead, we achieve
that same goal using the isolated_page flag.
When running some tests multiple times in the same call,
were failing because of the data duplication.
This commit resolves that issue by resetting the test
environment (i.e: Re-cloning test database and clearing
cache) after each run.
Fixes#17607.
Since the list of streams returned by a query which is not sorted
can vary, the tests which use it become flaky.
NormalActionsTest.test_default_stream_groups_events became
flaky due to this and hopefully sorting the streams should
fix it.
When a user entered an invalid character (whitespace or characters not
present in a name), the cleaned-up array, and hence the query,
would be empty which resulted in an error.
Fixes#17542
I have updated the capistrano docs so that the indentations are all
correct and align with the numbers correctly.
This one, if wanting to pout backticks round the code whilst indented,
doesn't render properly. Has to be double indented.
Fixes#17633.
I have updated the Puppet docs to include numbers for increase of
readability and have removed a thank you message from the bottom to
bring in line with the rest of the docs.
Fixes part of #17633.
I have the updated the documentation page for the hello world
integration to include numbers to bring it up to standard and make it
more readable.
Fixes part of #17633.
We were loading filters from localstorage after rendering the
filters block which caused the incorrect icons being displayed
for filters.
Since usually recent_topics renders a couple times when it
is loaded directly and the filters were being loaded as part of
show_selected_filters after we rendered recent_topics filters,
it meant the correct filters were being displayed in the
second render.
But, when user loads any other narrow directly when Zulip is loaded,
and opens recent_topics, recent_topics is only rendered once,
hence the bug gets noticed.
Fixes#17496
This commit migrates some of the backend tests in test_import_export
to use assertLogs(), instead of mock.patch() as planned in #15331.
Logs for tests in this file are suppressed and are not asserted as
that made changes to import/export codebase more fragile. As we
already have checks for the actual functionalities, it made less
sense to assert those logs.
Currently, there was no markdown page for deactivate-own-user API
endpoint. Created deactivate-own-user.md for the API page and
created a new owner client in test-api to reactivate the client
deactivated during testing.
Also changed endpoint name from deactivate-my-account to
deactivate-own-user, for better consistency with other endpoints.
Fixes#16163.
This is no longer used in any important place,
get_user_profile_by_email is meant to be used only in manage.py shell
now and thus there's no point in this function being cached.
Emails are not unique, so we can only sensibly cache using keys formed
with both email and realm.
This requires adding a new cache key function for caching by delivery
email - user_profile_delivery_email_cache_key.
Extend our markdown system to support mentioning of users
by id also. Following these changes, it would be possible
to mention users with @**|user_id** and silently mention
using @_**|user_id**.
Main intention for extending the mention syntax is to make
it convenient for bots to mention a users using their ids. It
is to be noted that previous syntax are also supported.
Documentation tweaked by tabbott for better readability.
The changes were tested manually in development server, and also
by adding some new backend and frontend tests.
Fixes: #17487.
We add support to shorten links and test their shortening in
well-organized, clean manner that makes it trivial to extend the
GitHub approach for GitLab and perhaps other services.
We only shorten basic types of GitHub links (issue, PR, commit) that
fit a set of simple common patterns; the default behaviour of Autolink
is kept for everything else.
Logic added in frontend and backend Markdown Processor is identical.
This makes easy to extend the logic for other services like GitLab.
Fixes#11895.
The only reason to use typeof foo === "undefined" is when foo is a
global identifier that might not have been declared at all, so it
might raise a ReferenceError if evaluated. For a variable declared
with const or let or import, a function argument, or a complex
expression, simply foo === undefined is equivalent.
Some of these conditions have become impossible and can be removed
entirely, and some can be replaced more idiomatically with default
parameters (note that JavaScript does not share the Python misfeature
of evaluating the default parameter at function declaration time).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The schema for unread_msgs was missing additionalProperties: false
that was causing tests to pass even with undocumented parameters
which were validated as an additional property. Set
additionalProperties to false and added documentation for missing
count variable.
Fixes#17728.
This commit changes the list_to_streams function to raise error
according to create_stream_policy value when a user cannot create
streams instead of same error for all cases.
This commit modifies test_user_settings_for_subscribing_other_users
to check all the possible cases including the cases when a user
can successfully subscribe other users along with the already
tested failure cases. This commit also adds checks for guest users
which was not present before.
This commit replaces the code which directly changes user.role,
realm.create_stream_policy and realm.waiting_period_threshold
with do_change_user_role and do_set_realm_property functions
in test_can_create_streams. This makes the code similar to the
other tests.
We refactor test_can_create_streams and test_can_subscribe_other_users
in test_subs.py. We want to follow a specific order in such tests
which is just set the policy value one by one and then checking
that the role in policy returns true and role just below that returns
false. This approach is explained in detail below.
Following hierarchy of roles is considered for these tests -
1. Realm admin
2. Full members
3. Members
4. Guests.
Then if the policy is set to admins only, we check that the having
role as admin returns true and the role just below that, i.e. full
member returns false. Similarly, if the policy is set to members
only, we check that a member should return true and role below it
which is guest should return false. We basically follow these as
we can assume that if a user with particular role cannot do the
required task, then user with role below in the hierarchy would
be not allowed to do the task too.
This commit refactors the above mentioned two tests to have above
explained workflow.
This commit removes the unnecessary do_change_user_role function
in test_can_subcribe_other_users. This was added in 1aebf3cab
which replaced the multiple functions like do_change_is_admin
and do_change_is_guest with do_change_user_role.
Previously two functions do_change_is_admin and do_change_is_guest
were used because there were two flags is_realm_admin and is_guest
which were used to determine the role of a user. But then we added
a single field role to UserProfile and removed the multiple flags
and thus also replaced the different functions with a single
do_change_user_role. With addition of a new field role, two
different do_change_* functions were not needed as we only have
a role field instead of different flags, but this was missed in
1aebf3cab and this commit fixes it.
These were introduced in ff9a929d7a
with no explanation of why they were necessary.
Generally you only render a few things, and it's
important that they're up to date.
We weren't doing a good job of invalidating the cache.
Eliminating the cache will fix bugs (like presence circles
being out of date) and break some dependencies.
I removed some very fragile test code that was relying
on invalid values taken out of the cache. (We now have
less line coverage, but if we want to test our rendering,
there are much cleaner ways to do it.)
As part of testing this, I renamed Hamlet to "aaron", so
that there are two aarons, and then I logged on as Iago
to see the "secondary" code in action that shows their
emails to distinguish them.
Development environment docs contains an outdated
statement stating that the non-vagrant setup can't
be used on windows, while it can now be used on windows by
using wsl. This commit fixes the statement and points
the link towards the non-vagrant setup process page.
Fixes#17721.
The number is really low, since we haven't told anyone about our
GitHub Sponsors site, so the badge probably isn't actually helpful,
but I suppose that's OK.
This add the schema checker, openapi schema, and also a test for
realm/deactivated event.
With several block comments by tabbott explaining the logic behind our
behavior here.
Part of #17568.
We discovered recently that some ops for events were just not
implemented in events.py (specifically, realm/deactivated).
Since our goal is for events.py to be complete, we add this bit of
hardening to ensure that it stays that way.
This helper was added in eac6463031 and
used by the "message.handlebars" file. This is no current call for
this helper in the codebase, hence it is removed to improve coverage.
This commit also marks template.js to have 100% test coverage.
Modifies `StreamPattern` and `StreamTopicPattern` to inherit
from InlineProcessor instead of Pattern. This change is done
because Pattern stopped checking for matching patterns as soon
as it found a match which was not a valid stream. Due to this
all the subsequent mention failed, even if they were valid.
This bug was only present in backend renderring due to
markdown.inlinepatterns.Pattern.
Due to above changes verbose_compile is no longer used for
precompiling STREAM_LINK_REGEX, STREAM_TOPIC_LINK_REGEX as
adds ^(.*?) and (.*?)$ which cause extra overhead of matching
pattern which is not required. With new InlineProcessor these
extra patterns at beggining and end are not required.
So, StreamPattern and StreamTopicPattern now define their own
__init__ method for precompiling the regex.
Fixes#17535.
These changes were tested locally in dev server and by adding
some new markdown tests to test these.
Modifies `UserGroupMentionPattern` to inherit from InlineProcessor
instead of Pattern. This change is done because Pattern
stopped checking for matching patterns as soon as it found
a match which was not a valid user group. Due to this all
the subsequent user group mention failed, even if they were
valid. This bug was only present in backend renderring due to
markdown.inlinepatterns.Pattern.
This was reported as issue #17535.
These changes were tested locally in dev server and by adding
some new markdown tests to test these.
Modifies `UserMentionPattern` to inherit from InlineProcessor
instead of Pattern. This change is done because Pattern
stopped checking for matching patterns as soon as it found
a match which was not a valid user. Due to this all the
subsequent user mention failed. This bug was only present in
backend renderring due to markdown.inlinepatterns.Pattern.
This was reported as issue #17535.
These changes were tested locally in dev server and by adding
some new markdown tests to test these.
This adds support for unstarring all (starred)
messages from a particular topic, from the topic
popover.
The earlier implementation of this in #16898
was reverted in bc55aa6a01 (#17429)
because it had two problems-
1. The crash reported in bc55aa6a01
was due to message_store returning undefined. This happens
when the message itself hasn't been fetched from the server
yet, but we know that the message is starred from the ids
in `page_params` in `starred_messages.js`.
This commit handles this case explicitly.
Note that, we simply ignore those messages which
we haven't fetched, and because of this, it may
happen that we don't unstar some messages from that
topic. The correct implementation for this would
be to ask the backend for starred IDs in a topic.
2. The earlier implementation actually unstarred **all**
messages. This was because it grabbed the topic and stream_id
from the topic popover `data` attributes, after the topic
popover had been closed. This passed `undefined`, which
the function then interpreted as an action to unstar all
messages.
With this commit, we use the confirm_dialog widget,
which eliminates the need to store this data in the DOM.
* Currently, the confirm_dialog is used only in
the settings pane, which already has the `new-style`
class in the main `settings_overlay.hbs` file. So,
the confirm modal is rendered correctly there. But to
make it available for use outside of the settings pane,
we add the `new-style` class to the confirm container
itself, without which, the buttons look ugly.
* The other change here is the click handler for
removing the modal element. Previously, when the
modal was closed (with any of the "yes"/"no"/"cross"
buttons), there was a small time interval of around
a second during which the modal had disappeared,
but the background content was still in the faded-out
state. This change fixes this glitch. This glitch was
probably not noticable earlier, because the settings
pane itself causes the background to be slightly faded
out.
This code was added in 2d414fa897, after
the `window.exports` variable was removed from
`stream_popovers.js`, while converting it to an ES6 module
in c71af35461. This resulted
in opening the starred messages or all messages
popovers throw `Error: exports in undefined.`.
In the openapi specs, the update_message_flags event is documented as
having a `operation` (deprecated) field, alongside the modern `op`.
This causes check_schemas warnings like this:
NEED SCHEMA to match OpenAPI update_message_flags_add_add_event
NEED SCHEMA to match OpenAPI update_message_flags_remove_remove_event
as check_schemas uses both `op` and `operation` for constructing the
event name.
Being deprecated (and really only still there for
backwards-compatibility with the original error of having it present),
`operation` will be removed eventually, therefore we can safely
ignore it from being used in openapi schema validation.
Part of #17568.
This removes the `add` from op list of stream event, as we do not
actually generate the stream/add event in the API, and when a stream
is created we identify it using the `create` operation.
(This was likely just a mistake introduced as a result of the fact
that `create` does not fit the normal naming scheme; probably
long-term we should actually migrate this to "add", but more important
for now is to document what's accurate).
Part of #17568.
Follow up to #14768. This feature was already non-functional due to
.alert-display { display: none; }, and if we want to reimplement it,
we should do it using a modern library.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Remove the unused notifications-area wrapper. Remove the feature
detection code as all browsers recognize the <audio> element. Create
the <audio> statically with the page template. Use multiple <source>s
to let the browser detect the appropriate format instead of trying to
do its job for it. Remove the absurd loop="yes" attribute, which had
fortunately been specified on the wrong element.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is mostly a refactoring to break the unnecessary
dependency of bot_data on settings_bots.
This is a bit more than a refactoring, as I remove all
the debounced calls to render bots during the
initialization of bot_data. (The debouncing probably
meant we only rendered once, but it was still needless
work.)
We don't need to explicitly render bots during
bot_data.initialize(), which you can verify by loading
"#settings/your-bots" as the home page. It was just an
artifact of how add() was implemented.
Note that for the **admin** screen, we did not and
still do not do live updates for add/remove; we only do
it for updates. Fixing that is out of the scope of this
change. The code that was moved here affects
**personal** bot settings.
Note that the debounce code is quite fragile. See my
code comment that explains it. I don't have time to go
down the rabbit hole of a deep fix here. The puppeteer
tests would fail without the debounce, even though I
was able to eliminate the debounce in an earlier
version of this fix and see good results during manual
testing. (My testing may have just been on the "lucky"
side of the race.) I created #17743 to address this
problem.
The keyboard-shortcuts icon currently has a fix position
causing design related bugs such as overlapping with userlist
in the sidebar.
The fix wraps the invite-more-users link and keyboard icon inside
a div with display property as flex instead of just using the anchor
tags inside the side-bar items.
This also fixes the suggestions for the following words: disabled,
disables, disabling, implemented, implementing, implements, kept,
made, took, using.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This mainly extracts a new module called
browser_history. It has much fewer dependencies
than hashchange.js, so any modules that just
need the smaller API from browser_history now
have fewer transitive dependencies.
Here are some details:
* Move is_overlay_hash to hash_util.
* Rename hashchange.update_browser_history to
brower_history.update
* Move go_to_location verbatim.
* Remove unused argument for exit_overlay.
* Introduce helper functions:
* old_hash()
* set_hash_before_overlay()
* save_old_hash()
We now have 100% line coverage on the extracted
code.
I moved four functions, verbatim, to a new module.
They were in message_util before, which led to
filter.js having several accidental indirect
dependencies.
I considered just putting these four functions in
filter.js, but I think it's a nice abstraction boundary
that filter.js delegates actual message parsing, and
the original author apparently had a similar thought
process.
I also wanted to make it so that a casual reader of
filter.js doesn't think we are manipulating DOM. It's
true that we still indirectly require jquery here, but
it's only for parsing, and it seems plausible we would
eventually use a more low-level parser.
I can see us maybe using these functions in something
like MessageListData in the future, so speculatively
splitting them out might future-proof us from some
cyclical dependencies.
I also think it's plausible that we will just modify
our two markdown processors to attach that kind of
metadata to the messages.
Last but not least, I think there might be opportunity
here to simplify the filter tests and remove some of
the zjquery hacks. We would instead just mock the
message_has_* helpers for the filter tests, and then
do more detailed direct testing on the functions
themselves.
The only caller for this function was settings_config,
so we put it there.
For the stream_edit test we no longer mock the function.
(The reason we mocked the function was more about avoiding
the heavy settings_notifications import than the function
itself.) This gives some incidental coverage, but then I
also add some more real coverage on it.
We extract compose_fade_users and compose_fade_helper.
This is a pretty verbatim extraction of code, apart from adding a few
exports and changing the callers.
This change makes the buddy_data module no longer sit "above" these
files in the dependency graph (at least not via compose_fade):
* jquery
* lodash (not a big deal)
* compose_state
* floating_recipient_bar
* message_viewport
* rows
The new moules have dependencies that buddy_data already
had directly for other reasons:
* people
* util
And then buddy_data still depends on stream_data indirectly through
the compose-fade logic for stream_data. Even without compose-fade, it
would depend indirectly on stream_data via hash_util.
Note that we could have lifted the calls to compose_fade out of
buddy_data to move some dependencies around, but it's useful to have
buddy_data fully encapsulate what goes into the buddy list without
spreading responsibilities to things like activity.js and
buddy_list.js. We can now unit-test the logic at the level of
buddy_data, which is a lot easier than trying to do it via modules
that delegate drawing or do drawing (such as activity.js and
buddy_list.js).
Note that we still don't have 100% line coverage on the
compose_fade.js module, but all the code that we extracted now is
covered, mostly via buddy_data tests.
This is preparatory work for investigating reports of missing unread
messages.
It's a little surprising that not test failed after adding the code
without API documentation.
Co-Author-By: Tushar Upadhyay (tushar912).
This commit adds vertical-align: middle to .message_failed in zulip.css
which was necessary as the alignment of .message_failed wasn't matching
with rest of the message controls like .edit_content. This makes the
look of the message controls better that they don't look shifted.
Follow up #17666
This is a prep commit which modifies the
`send_message_moved_breadcrumbs` function to take
message strings as input.
This is done to reuse the function in other places
like the /digress command.
Previously, it was tedious to create actual message
objects in message_store for use in node tests.
This was mainly because, `add_message_metadata`
in message_store has many dependencies and
validation checks. Since it was difficult to create
actual message objects, many tests just mocked
the `message_store.get()` method to return the desired
message.
This commit adds a new helper method (`create_mock_message`)
to message_store, for use in node tests. This just stores
the object passed to it in the `stores_messages` map,
without any validation. We do not add any
default fields to the message object before saving
it from this helper, because doing so would decrease
the utility of this helper, and, if a test
depends on some field having a particular value,
then it would be better to just pass the field: value
pair from the test itself, for readability, rather
than relying on the helper to add the field for us.
This helper allows us to write deeper tests.
This commit also replaces some instances of mocking
`message_store.get()` to use this new helper method.
Previously, if a user had zero total starred messages,
we would still show the "Unstar all messages" in the
left sidebar on opening the starred messages popover.
This commit adds a check to show button only if the
user had non-zero starred messages. This is done
because-
1. The button, when shown when the user has zero
starred messages, is redundant and may be confusing.
2. Clicking on the button when having zero starred
messages sends a zero-length array to the backend,
resulting in HTTP 400 error.
Computed indexing into an object, especially with a user-provided key,
can be dangerous in JavaScript because of nonsense features like
obj["__proto__"]. In this case there’s no vulnerability because the
possible keys are strictly limited by the regex, but it’s always
better practice to use a Map for computed indexing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit removes the unless msg/locally_echoed condition for the
edit content div, which has the consequence of making the "view
message source" widget always available for locally echoed
messages. This ensures that the message source can be seen if a very
long message has been drafted and it fails due to a server-side error
(See #17425 for the original report).
Fixes#17650.
Structurally, exception, failure_message, and status_code are mutually
exclusive in how this function is called, and it's best for the
function's flow to represent that.
The message from the bot which triggered the 407 error message notifies
the bot owner about the exceptions as well in the error message. This
commit handles it more gracefully and shows a generic message.
The messages from the bot which were triggered by the outgoing_webhooks
didn't have the bot name in them. This commit adds the bot name to it
and makes the corresponding changes in the tests.
Had this been in normal route, this would have been an XSS bug, as we
were passing what the developer clearly believed to be plain text into
an HTML 404 page.
The affected routes have @require_server_admin, a permission that we
do not expect any self-hosted users to have ever enabled (as it is
undocumented and doing so is only possible manually via a `manage.py
shell`, and we believe to only be useful for running a SaaS service
like zulip.com). So the security impact is limited to a handful of
staff of zulip.com and this isn't a candidate for a CVE.
Thanks to GitHub's CodeQL for finding this.
This adds an option for restricting a ldap user
to only be allowed to login into certain realms.
This is done by configuring an attribute mapping of "org_membership"
to an ldap attribute that will contain the list of subdomains the ldap
user is allowed to access. This is analogous to how it's done in SAML.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This verifies that the proxy is working by accessing a
highly-available website through it. Since failure of this equates to
failures of Sentry notifications and Android mobile push
notifications, this is a paging service.
This commit takes the blocks of code from "build_message_groups" that are the
same as "_rerender_message", and move those into a function called
"set_calculated_message_container_variables". This helps to avoid bugs in
future as in #17663. Like timestr was being updated in one of them, but needed
in both. So, it takes care that message variables are correctly set.
Part of #17663
This commit updates the _rerender_message to update the message_time
string with the current timestamp on the message rerender.
When we locally echo a message, we store a local timestamp that will
generally not be used as it is replaced by the server time in
echo.process_from_server when we confirm receipt of the message.
echo.process_from_server correctly updates the .timestamp field on
the message and triggers a rerender but that rerender reuses
the message_container object without recomputing the
message_container.timestr due to which wrong older timestr was shown
on the message box.
This commit fix this by calling set_timestr in the rerender code path,
alongside calls to update similar data structures like
this._maybe_format_me_message.
Fixes#17655
In responsive narrow windows where the left sidebar is an overlay, clicking the \vdots menus for
'All messages' and 'Starred messages' would result in the navigation closing and the menu appearing
somewhere weird.
We fix this the same way that we address this issue with the similar stream/topic menus, by calling
the function to show this sidebar after closing all popovers.
Fixes: #17537.
On replying to an email notifcation from a stream where the user
does not come under the stream_post_policy will subsequently result
in a failure. In such a case, the user does not receive feedback
regarding the failure.
Notify the user via notification bot if their email
message failed to send.
Fixes#16642.
The custom-profile-fields-form element custom_user_field contains
the textarea for Biography that expands. The textarea treats the
user-avatar-section as an disturbing obstacle when expanded beyond
the certain width. To fix this, the custom-profile-fields-form
is placed out from the account-settings-form.
Fixes#17617.
Fixes#17466
This commit will change encoding logic. Initial logic
was not encoding parenthesis, and this creates conflicts
with the markdown link format. To resolve this while encoding,
we're now replacing parenthesis with ".28" and ".29."
There is no need to change decoding logic because before
decoding any URL, we first convert all the “.” to “%.”
optimization: No need to replace parenthesis in popovers.js.
I have updated the docs for the SVN integration to properly indent the
code block etc as well as using 1. for the numbers rather than 1. 2. and
so on.
Fixes part of #17633.
I have updated the docs for the Jenkins integration to include number
for ease of read as well as switching over to the new
{create-a-bot-indenmted.md} template to allow the continuation of
numbers.
Fixes part of #17633.
GitHub Actions gives us 2 cpus (probably shared) to run the
jobs. Specifying 6 processes here doesn't make a difference
since both jobs run in around 5 minutes right now.
We basically move all the tests from backend and frontend test
files to zulip-ci workflow. This results in GitHub Actions
nicely displaying all the tests separately.
Timestamps are logged automatically by GitHub Actions and can be
made visible using log settings easily. Hence we remove the
unnecessary timestamps here to make the logs look much cleaner.
We have disabled CircleCI and are using GitHub Actions for automated
testing.
docs: Changed context from CircleCI to Github Actions and wrote
some documentation specific to GH Actions.
tools: Replaced env checks for CIRCLECI with GITHUB_ACTION.
README: Use GitHub Actions build status badge.
GitHub Actions supports doing more than just CI,
and so in some contexts it's less obvious that we're
talking about just the CI if we refer to it instead of CircleCI.
The scroll position of recent topics table is according to the
element which is in focus.
While this behaviour is correct, when
user clicks on an element in recent topics after scrolling to
a different position, the scroll position is lost as the focus
was not being set on the element. This commit ensures that
we set focus on the element when user clicks on it. Thus, the
scroll position being lost is naturally fixed.
Fixes#17587
If the client has an old version of the code which is not present on
the server, don't throw a 500; instead, default to the same `unable to
look up in source map` message is used when the line numbers don't
line up.
All of `/var/log/nginx/` is chown'd to `zulip` and the nginx processes
themselves run as `nginx`, and would thus (on their own) create new
logfiles as `zulip`. Having `logrotate` create them as the package
default of `www-data` means that they are momentarily unreadable by
the `zulip` user just after rotation, which can cause problems with
logtail scripts.
Commit the standard `nginx` logrotate configuration, but with the
`zulip` user instead of the `www-data` user.
This commit removes the option to add more streams out of scrollbar
as it is not visible on mobile devices or organizations with large number of
streams until scrolled down.
A few years ago I introduced the anti-pattern
of automatically calling success functions within
channel.get stubs. It's better to just capture
the success function and call it explicitly.
Also, we now have override(), so it's easy to
inline these types of things in a safe way.
The title in li.top_left_private_messages applies the title
to private-container too. This causes to show "Private message"
title to private message elements. To fix this, the title is
placed at div.private_messages_header.
There is no element on the settings page with
id="admin_page_filters_loading_indicator", so
the indicator doesn't appear. And even if we make
a div for an indicator, it would be hardly visible,
because we don't call the server to fetch linkifiers
in this page, and there generally won't be too many
linkifiers to render.
Previously, if there weren't any linkifiers set, only
non-admins could see the "No linkifiers set." message
below the list header. We do not hide the linkifiers
(when there are some of them) from non-admins, so there
seems to be no reason hide the above message when there
aren't any linkifiers.
Add input to filter in sortablejs config. This prevents drag
and drog from being called on clicking input field. Also
set preventOnFilter to false. This prevents disabling the
default behaviour on the click event.
Fixes#17619
This is a code cleanup commit.
Earlier, install-node used to fail to clone using ssh in CircleCI,
so we removed the git config file.
Now we no longer observe the nvm git clone error,
so the issue seems to be resolved.
Previously we could navigate the user info popover on messages by
using the up/down arrow keys, but we could not use the enter key to
select an item, this commit fixes the bug.
Fixes: #17589.
These sigils will help make it easier to see that this is a special
Zulip syntax feature, not just something the other user typed, which
might help set the expectation that we're showing the time in the
user's timezone.
Tweaked by tabbott to improve variable/template naming.
Apparently, we never tested the unlikely behavior of deleting the last stream,
and doing so would result in exceptions being thrown (and thus no UI update).
Fixes: #16691
As demonstrated with the recent Zabbix integration, our line-wrapping
of numbered lists was busted in the presence of 2-digit numbers of steps.
Fixes#17634.
I have updated the documentation for the Zabbix integration to give the
correct instructions for the latest version of Zabbix (5.2). The old
instructions are now obsolete.
I have also updated the message that is PMd to a user if the webhook
doesn't receive a complete payload to also align with the new
instructions.
The "Add streams" button text made users think this was what you needed to create
additional streams in the organization. While that's correct for the first user/organization
administrator, it's not correct for most other users.
0663b23d54 changed zulip-puppet-apply to
use the venv, because it began using `yaml` to parse the output of
puppet to determine if changes would happen.
However, not every install ends with a venv; notably, non-frontend
servers do not have one. Attempting to run zulip-puppet-apply on them
hence now fails.
Remove this dependency on the venv, by installing a system
python3-yaml package -- though in reality, this package is already an
indirect dependency of the system. Especially since pyyaml is quite
stable, we're not using it in any interesting way, and it does not
actually add to the dependencies, it is preferable to parsing the YAML
by hand in this instance.
This is needed not because the DOM isn’t ready here (we’re in a
<script defer>), but because we want to wait an asynchronous tick
until after all the other callbacks that waited an asynchronous tick
for the DOM to be ready.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 5bd73ce190 (#17367)
unintentionally truncated more of the traceback rather than less when
there’s more than one Module._compile frame.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We have generally gone away from using $(...)
initialization in modules that we test with
zjsunit, but there are a few remaining special
cases related to our billing and portico
codebases.
I remove an obsolete comment--we use get_streams()
for the `n` key now.
I also remove a guard statement from sort_groups()
that returned `undefined` for empty lists.
That guard statement would break this code:
const stream_groups = stream_sort.sort_groups(streams, get_search_term());
if (stream_groups.same_as_before && ...
The calling code prevents the situation anyway:
const streams = stream_data.subscribed_stream_ids();
if (streams.length === 0) {
return;
}
I modify the "no_subscribed_streams" test to test
the new behavior. (Even though stream_list currently
short-circuits the call here, that may change in the future.)
I also introduce the test() wrapper to explicitly clear
our data.
This is prep toward allowing individual tests
to fail while continuing to run the test suite.
Most of the steps in namespace.finish() could
be put in a `finally` block, but once a test
fails, there's no reason to complain about
unused mocks, since there are bigger things
to address.
Don't focus on search box when user is at end or start of
the table and is using vim keys. This ends up being a
bad UX as once user is inside the search box, vim
navigation keys cannot be used to take user out of
the search box.
This reverts commit 211232978f. The
`rabbitmq` user does not exist yet on first install, and the goal is
to create the `rabbitmq-env.conf` file before the package is
installed.
In production, the `wildcard-zulipchat.com.combined-chain.crt` file is
just a symlink to the snakeoil certificates; but we do not puppet that
symlink, which makes new hosts fail to start cleanly. Instead, point
explicitly to the snakeoil certificate, and explain why.
The email subject used to be translated to the language of
the user who requested the sponsorship. This was a bug since
the recipient of the emails are Zulip's support staff and
not the user who requested the sponsorship.
When typing the password in Firefox,
it shows a "Not Secure" warning which was
hiding the "#get_api_key_button". You can see
the screenshot of it in #17136.
This commit fixes that issue by focusing on the button.
Factor out mock_cjs from mock_esm because adding __esModule prevents
mocks for CJS modules from being imported correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Use fully resolvable request paths because we need to be able to refer
to third party modules, and to increase uniformity and explicitness.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This introduces the make_stream_message()
helper to avoid all the strange
`messages[0] === message1` confusion.
We also clear data explicitly at the beginning
of the test.
It's still a messy test.
A bunch of tests were mutating the same message.
Now they just make their own copy.
I introduce the name "sample_message" for the common
message, but I remove the two bogus reactions, and
the "warning" test now just creates its own
test-specific data.
I considered using set_info({}, 0) to clear data,
but it has a lot of machinery that could lead
to accidental line coverage and/or extra test
complexity.
This commit adds a new class for typeahead items that do not need
a presence circle. It is changed to support addition of new items
that do not require presence circle.
There should not be any visual change due to this.
This commit removes presence circles for special users like
all, stream, and everyone. This was discussed at
#design>Presence circles in typeahead, and this was justified
as presence circles for these special users will always be grey
circle and do not convey any information about presence of anyone.
* move one test below setup
* add a test() wrapper to clear mutes
* split up a test
* use two different dates for java vs. gossip
* deterministically sort by date
All these changes make it so that the order of
the tests is a lot less fragile.
@typescript-eslint/recommended disables no-undef on the theory that
TypeScript knows which globals are in scope. However, some TypeScript
definitions include declarations for globals that may or may not
actually be available depending on the environment. So we will need
to prohibit some globals that TypeScript knows about, and enforce that
they are properly imported.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Directives in `location` blocks may or may not inherit from
surrounding `location` blocks; specifically, `add_header` directives
do not[1]:
> There could be several add_header directives. These directives are
> inherited from the previous configuration level if and only if there
> are no add_header directives defined on the current level.
In order to maintain the same headers (including, critically,
`Access-Control-Allow-Origin`) as the surrounding block, all
`add_header` directives must thus be repeated (which includes the
`include`).
For clarity, un-nest and repeat the entire `location` block as was
used for `/static/`, but with the additional `add_header`. This is
preferred to the of an `if $request_uri` statement to add the header,
as those can have unexpected or undefined results[2].
[1] http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header
[2] https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/
{visible: false} just redundantly specifies the default behavior,
which is to wait for the selector to be present regardless of
visibility. We want to wait for these selectors to be hidden.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Using get_user_profile_by_email is invalid, as it omits the realm, and
also fetches via .delivery_email - our convention is that .email is
supposed to be used for user-facing purposes like this.
self.example_user("hamlet") uses get_user_by_delivery_email, so it
doesn't actually cache anything. This should use a cached function, like
the test below: test_do_change_realm_subdomain_clears_user_realm_cache.
This is part of our general process of replacing emails, which are not
static with time, with user_ids when referring to users in the API.
We still keep the `email` reference option, since it can be useful for
linking third-party applications to Zulip on an intranet that might
have a user's corporate email handy and not want to do the extra round
trip to lookup the user.
The name of the parameter, user_id_or_email, was chosen to to make it
clear that the default/preferred option is user_id.
Fixes#14304.
Use confirm_dialog here as this change is destructive and thus not
easy to undo.
We may want to consider using settings_ui.do_settings_change()
instead.
Fixes#17073.
Split the logic of check_profile_incomplete into two functions
show_profile_incomplete and check_profile_incomplete.
The latter is passed to the former which shows the message if the
profile is incomplete.
Use a regex to check for the pattern "Organization
imported from ..." instead of the previous approach
where we just checked if pattern startswith "Organiztion
imported from".
This allows users to extend the description from the original
"Organization imported from Slack." with a few extra sentences without
this warning remaining indefinitely.
Fixes#17463
TextField is used to allow users to set long stream + topic narrow
names in the urls.
We currently restrict users to only set "all_messages" and
"recent_topics" as narrows.
This commit achieves 3 things:
* Removes recent topics as the default view which loads when
hash is empty.
* Loads default_view when hash is empty.
* Loads default_view on pressing escape key when it is unhandled by
other present UI elements.
NOTE: After this commit loading zulip with an empty hash will
automatically set hash to default_view. Ideally, we'd just display
the default view without a hash, but that involves extra complexity.
One exception is when user is trying to load an overlay directly,
i.e. zulip is loaded with an overlay hash. In this case,
we render recent topics is background irrespective of default_view.
We consider this last detail to be a bug not important enough to block
adding this setting.
When user presses escape but there is no action that recent topics
can perform on it, it returns false.
The final state of focus is the focus on topics table. If the focus
is on table and not on RT search or filters, we return false to
indicate that the key was unhandled by recent topics.
This will allow escape to take user to another view if recent topics
is not the default view.
In the documentation of the "Sending messages," path for the `check_message` and `do_send_message` function is not provided. So, I added the path of both for future contributors.
Redis is not nagios, and this only leads to confusion as to why there
is a nagios domain setting on frontend servers; it also leaves the
`redis0` part of the name buried in the template.
Switch to an explicit variable for the redis hostname.
The query string parameter authentication method is now deprecated for
newly created Slack applications since the 24th of February[1]. This
causes Slack imports to fail, claiming that the token has none of the
required scopes.
Two methods can be used to solve this problem: either include the
authentication token in the header of an HTTP GET request, or include
it in the body of an HTTP POST request. The former is preferred, as
the code was already written to use HTTP GET requests.
Change the way the parameters are passed to the "requests.get" method
calls, to pass the token via the `Authorization` header.
[1] https://api.slack.com/changelog/2020-11-no-more-tokens-in-querystrings-for-newly-created-appsFixes: #17408.
We want to be careful about MIT code, as it leads
to more lax checks in the filter code.
And then we just use with_field in a couple places
and clear subscriptions.
For most functions that we were using __Rewire__ for,
it's better to just use the override helper, which
use __Rewire__ under the hood, but also resets
the reference at the end of run_tests.
Another nice thing about override() is that it reports
when you never actually needed the mock, and this
commit fixes the instances found here.
I didn't replace every call to __Rewire__. The
remaining ones fall under these categories:
* I looked for ") =>" in my code sweep,
so I missed stuff like "noop" helpers.
* Sometimes we directly update something
in a module that's not a function. This
is generally evil, and we should use setters.
* Some tests have setup() helpers or similar
that complicated this code sweep, so I
simply punted.
* Somes modules rely on intra-test leaks. We
should fix those, but I just punted for the
main code sweep.
I now let folks override the same target multiple
times inside of `with_overrides` (and then indirectly
inside of `run_test`.)
I may re-impose this restriction in the future, since
most violations are due to code smells, but there are
a few legitimate use cases for this, and the code
can handle it, plus I want to remove some other
ugliness first.
This is a deceptively ugly diff. It makes
the actual code way more tidy.
I basically inlined some calls to mock_module
and put some statements in lexical order.
We now just use a module._load hook to inject
stubs into our code.
For conversion purposes I temporarily maintain
the API of rewiremock, apart from the enable/disable
pieces, but I will make a better wrapper in an
upcoming commit.
We can detect when rewiremock is called after
zrequire now, and I fix all the violations in
this commit, mostly by using override.
We can also detect when a mock is needlessly
created, and I fix all the violations in this
commit.
The one minor nuisance that this commit introduces
is that you can only stub out modules in the Zulip
source tree, which is now static/js. This should
not really be a problem--there are usually better
techniques to deal with third party depenencies.
In the prior commit I show a typical workaround,
which is to create a one-line wrapper in your
test code. It's often the case that you can simply
use override(), as well.
In passing I kill off `reset_modules`, and I
eliminated the second argument to zrequire,
which dates back to pre-es6 days.
I created this test in Jan 2019 with
6b7a4f8611.
At the time I created this we were still migrating modules
over from using the $(() => {...}) style of initialization,
so the test served a purpose as an easy smoke test that we
weren't doing something really stupid in
ui_init.initialize().
Now that that migration is complete, the test's warts start
to outweigh its benefits. Because it's just a smoke test,
we get credit for artificial line coverage where we're
exercising code without actually verifying anything other
than that it runs without crashing.
There are still possibilities for bugs within
ui_init.initialize() where we initialize modules in the
wrong sequence, but I am not confident that the node test
would catch them with any more likelihood (or clarity) than
either the puppeteer tests or czo bug reports.
And then on top of that, the zjquery setup in the node test
is hard to maintain. It's also hard to verify that some
of the setup isn't cruft.
If you want to see how much of a maintenance headache this
has been in recent times, just look at how many of the
recent commits have been related to things like es6
migrations or code cleanup sweeps.
We should just kill it.
"Alert Words" is one of Zulip's oldest settings UI elements, and as a
result is buggy. This commit converts it to use our standard
progressive-table-wrapper system used for settings tables, which has
the side effect of fixing a bug that mad ethe tables look pretty bad
if one adds a very long word.
Fixes#17172.
ipython < 7.20.0 is incompatible with jedi >= 0.18.0; it fails to
tab-complete in `./manage.py shell`, as described in
ipython/ipython#12740.
We cannot bump the ipython dependency because ipython 7.20.0 requires
Python 3.7, and we must support Python 3.6 due to Ubuntu 18.04
support. Our only solution is thus to cap the version of `jedi` to
the last one before its API changed.
Previously, these were interpreted relative to the current directory.
This was notably causing dozens of extra var/mypy-cache directories to
be created in different places when mypy was run from Emacs Flycheck.
Use the $MYPY_CONFIG_FILE_DIR variable added in mypy 0.800.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit addresses the problem of user's status visibility to
some extent. It adds presence circles, like we have in buddy_list to the
typeahead suggestions that are given for mentioning users in messages.
Tweaked by tabbott to adjust vertical alignment of group mentions as well.
Testing for the changes is done manually in the developement server,
and also by updating frontend tests to address these changes.
Fixes: #17138
Fixes#16674.
The commit hides the tooltip for touch events on touch enabled device.
Touch events trigger both click and hover action,
leaving the hover action sustained until next click.
Hence it is better to hide the tooltip to avoid the clutter in UI.
This === comparison between two Date objects added by commit
9896782fd1 (#17220) always returned
false, so the body of timerender was running every minute instead of
every day.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 13915740bb (#5199) added a loop
in update_timestamps that appended an entry to update_list once for
each element that its className matched. If there were two such
elements, this would double the length of update_list each time the
body of update_timestamps ran. So let’s not do that.
Also fix the incorrect elements !== null check from the same commit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit works around an issue with Puppeteer tests on Firefox
where `page.url()` does not show the URL fragment, by adding a
temporary function that solves the issue.
This commit sets the system default locale to `en_US.UTF-8` before
running the tests.
This was necessary because when running tests with Firefox, it was
failing on some test verifies the results of locale-aware sorting.
This commit introduces the `is_firefox`
option in CommonUtils to identify if a browser
is Firefox or not.
This helps us to avoid calling some tests
that is yet not compatible with Firefox.
When we run puppeteer with Firefox,
the `--window-size` option does not work,
which makes the bottom part of
the page cut off.
This commit fixes this issue
by setting the screen default viewport
to the maximum size of the window.
Minimized code duplication by integrating POSTRequestMock into
HostRequestMock and then updating the required files with
HostRequestMock.
Fixes part of #1211.
A deprecated import shouldn’t be used even in a migration, since the
migration will need to remain runnable in the future. We never needed
a migration for this switch anyway; we just needed to edit the old
migration, since no actual state changes are involved.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When exception is raised inside an exception handler, Python 3
helpfully prints both tracebacks separated by “During handling of the
above exception, another exception occurred:”. But when we’re using
an exception handler to retry the same operation, multiple tracebacks
are just noise. Suppress the earlier one using PEP 409 syntax.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
./tools/test-all had a short option -f which was used for running it
by skipping provision check. This short option paired well with its
long version --force, but in an effort to provide more clarity over
the use of force option it is renamed to --skip-provision-check.
Following this rename -f option for test-all does not pair well with
its long part which is now --skip-provision-check. Also, this option
is rarely used. So the short form -f is removed in this commit. We
now provide only its longer version --skip-provision-check.
This is a part of commits done to address issue #17455.
This commit changes help message for skip-provision-check argument
used with various test commands. The message is changed with an
intention to be more clear about what this option actually does.
This commit is in series of various changes done to provide clarity
over the use of --force option which is renamed to
--skip-provision-check.
Fixes: #17455
This commit renames --force argument used with various tests to
--skip-provision-check. As a consequence of this name change all other
files that set --force option for the test commands have been updated.
This change is done in order to provide more clarity for using this
option for runnning tests.
This commit addresses issue #17455.
This commit moves --force option used with various tests to
test-scripts.py to have it alongside the logic that does provisioning
status assertion.
This is a step towards providing more clarity over use of this
argument with tests as asked in issue #17455.
This commit updates the stream creation, subscribing others to
stream, wildcard mention settings and stream post policy to allow
realm moderators even if they are new and the respective setting
is set to allow full members only.
This commit renames the is_new_member property in models.py
to is_provisional_member which will return true for any user
who is not a full member. We will add a condition in further
commit such that this returns 'False' for a moderator as we
will initially give all the rights to moderator that a full
member has.
This is more broadly useful than for just Kandra; provide
documentation and means to install Smokescreen for stand-alone
servers, and motivate its use somewhat more.
Hover does not work for touch-based devices like mobile phones.
Hence the icons on the right sidebar do not appear, making the
user unaware of its presence on such devices. The following
media property displays the icon by default for such behaviour.
This commit adds about Remote - SSH extension (in VS Code), which
helps us develop remotely by providing a similar interface as if we
are developing locally. We also simplify the documentation for RMate
to use the new standards.
Tweaked by tabbott for simplification/formatting.
This was inaccurate after testing the implementation, and there's an
argument that we shouldn't move it as it will simplify migrating to a
world where (some) private message threads can have topics.
Hover does not work for touch-based devices like mobile phones.
Hence the the icons does not appear, making the user unaware of its
presence on such devices. The following media property displays the
icon by default for such behaviour.
These are all referring to email_gateway_bot, when they're supposed to
refer to the notification and welcome bots, respectively. The values are
the same though, so the tests were passing anyway.
Found by running the tests after
sed -i 's/\.with(/.toBeUsed().with(/g' frontend_tests/node_tests/*.js
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Its likely that we would implement new hotspots that aren't
a part of the tutorial hotspots, in the future. For instance,
a hotspot to advertise new features. Hence, grouping them into
categories like INTRO_HOTSPOTS would be a good start. We also
have an aggregate of all types of hotspots we may add in the
future, under ALL_HOTSPOTS.
This reverts commit 6fba17599f (#16898).
@chrisbobbe reported this crash:
Uncaught TypeError: Cannot read property 'stream_id' of undefined
at starred_messages.js:43
at Array.filter (<anonymous>)
at Object.e.get_topic_starred_msg_ids (starred_messages.js:40)
at stream_popover.js:221
at HTMLSpanElement.<anonymous> (stream_popover.js:358)
at HTMLUListElement.dispatch (jquery.js:5429)
at HTMLUListElement.v.handle (jquery.js:5233)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes#17238.
In process_new_human user, the queries were wrong, revoking all invites
sent to the email address, even in other realms than the one where the
new account just got created.
The textarea in Settings/User Profile was overflowing in
smaller width devices.
This commit fixes that issue by adding appropriate media queries.
Fixes part of #16817.
This reverts commit 34ada11448.
That commit traded a minor visual glitch for a major usability
regression at my most common browser width (960px).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, exact matches could be pushed off the typeahead list in the
case where there were more prefix matches that happened to rank first,
which is confusing to the user: if an emoji, for instance, falls into
this category, it will never show up in typeahead, which is easy to
confuse with the emoji not existing.
This isn't a perfect fix — there are still cases where it's hard to find
emojis because the prefix-space is very crowded, but it does fix a
category of surprising and frustrating behaviour.
This doesn't come completely without downside - it means that the exact
match emoji will jump to the front of the list, which changes what is
currently conceptually a "filtering" operation to a "filtering and
sorting" operation, but it seems on the whole to be a more ideal
experience. This is particularly notable in the non-typeahead emoji
picker, which uses the same codepath, but this change seems somewhat
desirable even there, since it allows the user to type the name of an
emoji and press enter and have that emoji show up, without having to
visually confirm that they aren't inadvertently selecting a
prefix-matching emoji.
A better solution to this in the long term might be ordering emoji
results by shortest-first as a tiebreaker for alphabetical ordering,
since that should provide the same behaviour while keeping the mental
model as "filtering" (since the sort order won't change as the user
types), but this seems like a reasonable first pass, and changing to
shortest-first ordering after making this change won't break any muscle
memory for existing users.
We change the text. This is an attempt to make the text space occupied by
the col header of last message timestamp smaller so that
it doesn't overflow to next line in some languages.
Also, add some extra padding.
We don't need to handle user clicking on Zulip logo since
changing the hash via the `a` tag takes care of it automatically.
Also, cleanup the narrow.restore_home_state function since
it is no longer being used.
We manually trigger a re-render of RT after a stream is muted
to update the list of topic in RT for the active filter.
This fixes the bug that RT doesn't update correctly
after a stream is muted.
On safari, after search box wraps to next line on smaller widths,
it is not visible due to some flex box default property difference
between chrome and safari. We fix this by resetting default
property.
If user is in private message narrow, we reduce height of stream
list to allow height for pm list in the left sidebar. We need
to recalculate it when moving out of pm narrow and moving in
rt narrow.
We need to increase a bit of spacing around text in rows at <750px
because row height is reduced after hiding avatars. We use
padding instead of line height so that this plays nice when text
is wrapped.
Note more padding is also required for >750px now because text
can be wrapped now and take more width than avatars.
Since All messages narrow is no longer home page for webapp,
we change its icon to align-left which also shows a concept of
interleaved topics / messages.
When idle, we try to backfill messages and in the end reselect
the closest message in the list, which can be a unread message
if present.
When recent topics is open, we can backfill messages; but
shouldn't select the message_id otherwise it will mark the
message as read if the message is unread while triggering
`message_selected.zulip`.
User can go from recent topics to stream / topic narrow via various
means, but all go through narrow.activate, hence we make sure all the
state changes we do in recent_topics.hide are actually applied when we
hide recent topics and go to another narrow.
This fixes the bug that narrowing from left sidebar to a stream
takes user to the top of the narrow.
The top row of the RT can be hidden sometimes after scrolling down
and then scrolling up. This is because the focus is applied before
the row is rendered. Applying the focus after the topic row is
rendered by the browser makes sure it is always visisble when
it needs to be.
For inputs to recent topics which were unhandled, we return false
so that the browser can handle them.
This also fixes the issue of search box not able input `t` key.
We land user on the first row of the table instead of the search
box because here user can access hotkeys like `w`, `q`, `/`, etc,
which will not be directly available if user is focused in
recent topics search box.
For tests:
We set focus to search by default to avoid mocking a lot of
table html for getting the tests passing.
This fixes the bug where a user cannot type vim keys in the
general search box / user search / stream search,
since they are captured by recent topics.
The behaviour was flaky for stream search, but can be reproduced
consistently after previous commit fixing the popovers.any_active
output.
Previously the filter would be reset every time the page was
refreshed. This commit adds persistence via localstorage, the tests
follow the pattern used in tests for drafts.
Fixes: #15676.
When user directly has hash for overlay in the URL when app loads,
we need to still show recent topics in the background. This
doesn't need to happen in other cases when user is accessing
the overlay after UI is loaded.
Go to Recent Topics on "#", no hash and "#recent_topics".
Go to Recent Topics as the last destination for escape key.
Map `a` key to All messages and change its hash to
`#all_messages`.
throttled mousewheel handler marks messages as read in the
message_list regardless of if the message_list is visible or not.
We don't trigger it if recent_topics is visible.
Recent Topics is no longer an overlay now, but note that it is
also not a typical messages narrow. It can reside between
an overlay and a Filter in the sense that it is dispalyed as
a typical Filter narrow but has properties of an Overlay.
Compose box is not visible in this view as it will be confusing
to many users and hence compose shortcuts have also been disabled.
Keyboard shortcuts that apply on messages have also been disabled.
The remaining shortcuts that apply to a narrow are still accessible
here.
This issue adds the appropriate padding to the
copy_generate_invite_link class. This fixes the copy link icon which
seemed to be shifted when clicked.
Fixes#16868
This commits gives a right margin to the stream description
so that it does not collide with the icon on its right and
also become better visually appealing.
Fixes the following issues:
- Rectifies broken label tag having a misleading 'for' attribute.
- Removed 'name' attribute from unlabelled span tag.
- Removed label expression from DropdownListWidget to built an,
abstraction for control group only.
Fixes#17311.
The description of request parameter of update-subscription-settings was
wrongly pasted in yaml and wasn't completely removed from the md file.
Made appropriate fixes in yaml and md file.
When interacting with popovers in the night theme using the keyboard
UI (e.g. the `i` menu for a message), the background color was
incorrectly white, resulting from the bootstrap `nav > li > a:focus`
rule. We had already fixed this for `nav > li > a:hover`; we just
need to add `nav > li > a:focus` to the relevant block of CSS rules as
well.
Replaces #17195 and #17353.
Rewritten to use a cleaner solution by tabbott.
We now only assign target once, rather than
assigning it then overwriting it for the
not-sent-by-me use case.
I tried to extract a "selector" here but the linter
complained.
Splitting up the tests here ensures we don't
needlessly do extra work here. (In the prior
clumsy implementation, the second test that
I split out here would fail due to the lack
of us setting up the jquery stub.)
On realms with large numbers of custom emoji, the typeahead emoji picker
often isn't useful. This is exacerbated by the fact the picker prefers
longer matches, so if there are five emoji that share a prefix, and an
emoji that is just the prefix, the only-prefix emoji will never show in
the typeahead emoji picker. This means that if someone thinks that there
is an emoji that shares a prefix with many other emoji, but they don't
remember for sure, they cannot use the typeahead emoji picker to check
that the emoji that they are entering exists.
There are two "real" fixes to this, neither of which this commit
addresses:
First, we should adjust the emoji ranking code such that exact string
matches for existing emojis are always shown in the picker. This would
be an improvement overall (the current behaviour is surprising and
frustrating), but it doesn't fundamentally solve the problem - if there
are many matching emoji, some of them will be pushed off the list.
Second, we should allow scrolling through the entire list of matching
entries in a typeahead, instead of only looping through the top N
matches. This will completely fix the problem (although there is some
UI/UX consideration in how to make it clear that the box is scrollable),
but seems like significantly more work to implement.
However, increasing the typeahead box size should improve the user
experience here independently of either of those changes.
I've chosen 8 as the max size for no particularly principled reason -
the fact that it's larger than 5 makes it more useful, but it's not so
large that it covers an obnoxious amount of the screen. Possibly it
would make sense to make it a bit bigger, but 8 seems like a good place
to start.
I've tested this on my laptop, which has a Intel i5-7200U CPU (~4.5 years
old, middle of the line when it was released) on a test instance with
5000 users, as well as on chat.zulip.org, and didn't see any noticeable
performance regression in completing @-mentions or emoji on either.
This means that in steady-state, `zulip-puppet-apply` is expected to
produce no changes or commands to execute. The verification step of
`setup-apt-repo` is quite fast, so this cleans up the output for very
little cost.
This warning was added in #6551. It’s not for any version of the
current Electron app, which we warn about on the server side with
DESKTOP_WARNING_VERSION, but rather some pre-Electron app so ancient I
don’t even know what it is. Apparently it communicated using the
window.bridge global, so eradicate that too.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Appling i18 to reaction tooltips (#16585) caused usernames to be
double-escaped, for instance, if there is a single-quote in a username.
This disables escaping of usernames by i18next, since they're escaped
again later by the rendering code.
Fixes: #16785
Usually we increase the opacity of an interactable icon on hover, but the search-icon at the top of the
right sidebar was missing the behavior.
Co-authored-by: ritik <ritikcn05@gmail.com>
test_signup: This test was wrong, because the inviter UserProfile was
from a different realm. Such a PreregistrationUser shouldn't be
considered valid.
test_tutorial: The direct call to internal_send_private_message was
using sender's realm as the realm argument which is not valid. It
doesn't lead to any error because the codepath seems to mostly not care
about the realm arg if the sender is a cross-realm bot. From my reading
of the code I think that wrong realm arg here would break user mentions,
because it makes its way to check_message() and then to
build_message_send_dict - but overall the message gets sent without
errors. Either way, this was a bug in the test and should be fixed.
Currently, the ID and Type fields didn't have a description,
and weren't being displayed. Added a schema component to add
descriptions, and display on the api page. Fixes part of #15967.
This commmit includes ROLE_MODERATOR in realm_user_count_by_role.
We also update test_change_role in test_audit_log.py to include
changes for moderator role as well.
Note that at this point, it's not possible to create moderator users;
this just will make it easier to write tests for logic involving them
as we develop the feature.
Have not included "ROLE_MODERATOR" in UserProfile.ROLE_TYPES
in this commit because did not want to update the openapi
docs at this stage as it will be a user-facing change and
not updating the openapi docs with moderator role included in
UserProfile.ROLE_TYPES gives error in ./tools/check-schemas.
‘function’ and ‘=>’ are not equivalent because they bind ‘this’
differently. For these functions, the ‘function’ semantics are
intentional.
This reverts part of commit 1a241cef88
(#17388).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The `message_id` was made an `str` object because
the request expected `Dict[str, str]`. The request is now
casted to `Dict[str, Any]` to fix the issue and removed
typecast of `message_id` to str.
python-zulip-api reference:
https://github.com/zulip/python-zulip-api/pull/653
We now call $.clear_all_elements at the top
of run_test.
We have to exempt two modules from the new regime:
compose
settings_user_groups
Also, if modules do set_global("$", ...) we don't
try to call the non-existent function.
It's possible we'll want to move to something like
this, but we might want to clean up the two
sloppy_$ modules first:
// AVOID THIS:
// const $ = require("zjquery")
run_test("test widget", ({override, $}) => {
override(foo, "bar", ...);
$.create(...);
// do stuff
});
It was checking whether the selector string is itself null, not
whether it selects anything!
Use page.waitForSelector(…, {hidden: true}) instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Move clear_zulip_refs into restore, and rewrite it without lodash. We
no longer need the requires array, and zrequire is now nothing more
than a wrapper around require.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We weren't exercising this method in any
meaningful way during the tests, and when
do add coverage, we probably want to just
test it directly.
We also kill off stub_selector(), which was
never well-documented.
Callers can either explicitly pass in children,
stub out $(...)[0] as needed, or just
circumvent jQuery complications with override.
Note the reactions test was broken before,
since $(...)[0] was always returning the same
stub.
Like using $('input') is too broad a selector and shouldn't
be used in the codebase. With this error messages, contributors
can easily understand that now.
We no longer export make_zjquery().
We now instead have a singleton zjquery instance
that we attach to global.$ in index.js.
We call $.clear_all_elements() before each module.
(We will soon get even more aggressive about doing
it in run_test.)
Test functions can still override $ with set_global.
A good example of this is copy_and_paste using the
real jquery module.
We no longer exempt $ as a global variable, so
test modules that use the zjquery $ need to do:
const $ = require("../zjsunit/zjquery");
The "silent" option was kind of evil, as it had
$(...).find(...) passing back "self" instead of a stub.
Now we just use $(...).set_find_results(...) or
override(...) to simulate/bypass drawing code.
(It turns out hash_util didn't even need this option.)
Currently there are only tests for verifying the error case and there
are no tests to check the case where messages are sent successfully
in 'STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS' stream.
This commit adds tests for checking that full members and bots owned
by them can send message successfully in streams with post policy as
'STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS'.
We currently not allow new bots to send message in stream with post
policy as 'STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS', but we should
allow them to send messages if their owner is a full member.
This will make it consistent with behavior in stream with post
policy as 'STREAM_POST_POLICY_ADMINS_ONLY' where we allow non admin
bots with owner as admin to send messages.
According to tests we should not allow bot without owners to
post in streams with STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS.
But the code does not handle this and the related test passes
and raises error for case of bots without owner because the bot
is itself a new member.
This commit fixes this by adding a condition to check if there
is no bot owner and then raise error if there is no owner.
Now we just update the whole row any time a sub
changes. This prevents a whole class of bugs.
As the TODOs indicate here, some of the post-processing
that we have to do on rows after rendering the
template will soon go away.
I audited all the functions in stream_ui_updates and
added TODO comments to functions that are clearly just
updating rows in the left panel of Manage Streams.
In an upcoming commit I will simplify the approach so
that we just re-render the entire row.
The tooltips for the left panel of stream settings
have been broken since November 2018 due to my
commit 8f915da2ca.
The code prior to 2018 was restoring tooltips
right inside the loop where we were detaching
the row from the DOM to put it back into the
DOM at another place. And then I tried to
just add them in bulk, forgetting that I was
in the middle of all the DOM manipulation (and
hence my selector for the loop was a noop).
Also, I don't think we've ever had them for live
events that add streams. (I fixed that too.)
It's not clear to me that this code is actually
necessary, as we get hover help without
calling $(...).tooltip(...) properly.
This is probably why we didn't notice any
breakage when we merged my 2018 commit.
Checking for the button was a brittle way to do this.
Note that the code on master is flawed insofar as
we don't respect the search filters. I don't fix that
bug here. This is a tactical change to eliminate
another function.
Upcoming changes will make it so that all the bugs
related to "notdisplayed" will simply go away.
These aren't relevant to most users, but in the interest of
transparency, we also don't want the existence of these to feel like a
secret. And maybe publishing their existence will result in folks who
we forget to add to these private streams asking about them.
We generally want to avoid clicking on DOM elements
that may not actually be visible due to the prior
operation. Instead, we can just find the visible
element after each step.
I also introduce a couple helper functions to
de-clutter the click/unclick/click steps, and I do
a couple extra clicks for good measure.
You can verify that the test will fail if you
add an early return to update_check_button_for_sub.
The "Narrow to PM with" notification above the composebox was
double-escaped, mangling names with single quotes in them. This removes
the escaping in i18next, causing the name to be escaped only in
handlebars.
These optimizations only makes sense when all connections at a TCP
level are coming from the same host or set of hosts; as such, they
are only enabled if `loadbalancer.ips` is set in the `zulip.conf`.
This is a minor refactor which renames the
notify_topic_moved_streams function to
send_message_moved_breadcrumbs.
This is done because this function will be also used
for other things in the future, when moving streams
or when using the /digress command, for example.
Added assertion to check that if a deprecated flag is in a field's
schema, then it should have deprecated mentioned in description
as well, and moved these checks to a separate function.
Fixes part of #15967.
We just want to reset the scrollbar here, which
we still do via ui.reset_scrollbar.
You don't want to preserve scroll position if
you are filtering or re-sorting.
We have long had this annoying two-pass way of building the
DOM that I am trying to eliminate.
The function names that I introduce here describe the current
situation more accurately.
In passing I make it so that we only throttle redraws when
users are actually typing. Using a throttled redraw when
you click on the sort icons is at best unnecessary, and it
may actually aggravate double clicks.
Lists that were followed by a paragraph (i.e. our p+ul, p+ol CSS rule)
in messages had negative top margin of -3px. Adjusting the margin
here is important, because the default styling would result in an
excessive gap that made bulleted lists weirdly far from the previous
paragraph. See #12113 for background.
However, the -3px negative margin was so large that it reduced spaces
between paragraph and lists, such that there was too little visible
separation between the two. We fix this by going with a 0px
margin-top instead.
This has been tested for various structures of messages:
1. text + bulleted list
2. bulleted list + unbulleted list(or two lists)
3. only list.
And it looks good in all cases.
Fixes#17284.
The code blocks and response blocks had small and unreadable font,
because they were using the bootstrap defaults without adjustment for
the size of content on the rest of the page. Fixes part of
zulip#15967.
This both is better copy and also cleared for translators, who in
languages with gendered nouns don't need to guess what gender to use
for "one" without context.
This bug was caught thanks to the earlier commit which
introduces the "Restart tutorial" feature. To reproduce
the bug,
1. Restart tutorial
2. Click "Got it!" on the intro_reply hotspot
3. Repeat steps 1 and 2
The hotspot for intro_reply won't disappear the second
time around and the intro_stream hotspot would be displayed
simultaneously.
The reason for this was the intro_reply's "Got it!"
button codepath never removing the item completely from
the DOM. This would then conflict with the new intro_reply
hotspot which would get assigned a different 'id'.
Add new rest api endpoint GET users/{email} for looking up a user by
email, which is useful especially for corporate API applications that
might already have a user's email address.
Fixes#14302.
It looks like this ritual was born when a type comment wasn’t working
because it was mistyped without the colon.
Signed-off-by: Anders Kaseorg <anders@zulip.com>'
This finds three sets of related settings to extract from the
"Miscellaneous" settings section:
- Service configuration (PostgreSQL, RabbitMQ, Redis, Memcached)
- Previews (image, URL, and Twitter)
- Logging and error reporting
Adds a "unstar messages in topic foo" option to the topic sidebar
popover, if there are any starred messages in that topic, known
to the frontend.
Altered existing "unstar all messages" confirmation modal to mention
the topic name, in the case that it was opened by the topic sidebar
codepath.
This is just a v1, and will not unstar old messages from that
topic, if they have not been fetched by the frontend.
Fixes#12194
Co-authored-by: Abhijeet Bodas <abhijeetbodas2001@gmail.com>
A few internal fields used for tracking which types of notifications
have already been sent for a given message, like `hander_id` and the
`push_notified` bundle of fields were being incorrectly included in
message events delivered to clients clients.
One could argue these fields might be useful hints to clients, but
because notifications can be triggered later on via
`missedmessage_hook`, they have no useful purpose in the API.
This commit move these extended event field on a `internal_data`
object within the event object, and delete this field in `contents()`
for call points that would serve data to clients.
Tweaked by tabbott to provide a cleaner interface.
We're not bumping API_FEATURE_LEVEL because these fields have always
been documented as being present only due to a bug, so no clients
should be expecting or relying on them.
Fixes: #15947.
This splits the one big `run_test` block into
add/edit/delete parts, for better clarity.
This also removes the `with_field` calls and
uses the override style instead, which is
preferred because it makes sure the stubbed
function is actually called.
This commit replaces `with_field` calls to
use the override style instead.
`override` is preferred since it makes sure
the stubbed function is actually called,
while `with_field` doesn't, which makes it
hard to spot dead code.
This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design. It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
This commit replaces all `with_stub` calls and
explicitly calls `make_stub` instead.
The `with_stub` helper does not add much clarity
hence we now use scoped stub objects instead.
This de-indents some blocks where scoping isn't
required, for example when there is a single
stub object inside a `run_test` function.
With this change, we also need to explicitly
assert `num_calls`.
This is required for unattended upgrades to actually run regularly.
In some distributions, it may be found in 20auto-upgrades, but placing
it here makes it more discoverable.
Match the order of the variables between `default_settings.py` and
`settings.py`, and move the defaults into `default_settings.py` so
the section does not require any uncommented lines in `settings.py` if
LDAP is not in use.
We can relax this restriction in the future, but
basically every time it came up for me, the test
code was just disorganized, or it had an easy
workaround.
These are still kind of a mess.
The old code combined the worst of both worlds:
- we had one monolithic test
- we called the events multiple times,
verifying a different stub each time
Now I make the tests more granular.
We could actually re-combine the tests, but
in a nicer way, so that we just set
up multiple stubs and verify that all stubs
get correctly invoked.
There is also some code cleanup here--in dispatch_subs,
we don't stub stream_data, so it's easier to write
deeper tests that actually validate the data changes.
This prevents a bug where we interpret "2something"
as a modern slug instead of a legacy stream name.
The bug was probably somewhat unlikely to happen in
practice, since it only manifests if 2 is an actual
stream_id.
We still need to write to these globals with set_global because the
code being tested reads from them, but the tests themselves should
never need to read from them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit makes it so that MessageListData
methods always attempt to filter muted messages.
We later, in a new function
(`messages_filtered_for_topic_mutes`)
check if `excludes_muted_topics` is true or not,
and skip the filtering work if it isn't.
This new function consistently returns a new list.
This refactor will later allow us to write clean
and concise code as part of mute users.
This commit also refactors the muting tests
for MessageListData, which were earlier
spread across two `run_test` functions.
These tests should remain organized,
since similar tests will be added as part of
user mutes in future commits.
Previously, the `muting_enabled` property of
MessageListData class was used to indicate whether
some messages in the message list need to be
filtered due to topic muting, depending on the
narrow. For example, we exclude messages belonging
to muted topics from stream narrows, but not from
search narrows.
The name `muting_enabled` is a bit confusing, and hence is
changed to `excludes_muted_topics`.
It is also important that the name be specific, since
a similar new property will be added for user mutes
in future commits.
user_profile.id was confused for user_profile.recipient_id. These bugs
are particularly sneaky as they can go undetected by tests due to ids of
objects accidentally coinciding. We add a mitigation for this class of
mistakes by shifting the Recipient.id sequence in test db.
This was introduced in dda3ff41e1.
On the rare occasion where user_profile.id would coincide with
recipient_id passed to the function, we would return the wrong value.
That is, instead of correctly returning recipient_id, we would return
sender.recipient_id - recipient id of the sender of the message, thus
possibly returning user_profile.recipient_id (if user_profile is the
sender) - exactly the situation the function wanted to avoid
with the `if recipient_id == my_recipient_id:` if. Ultimately resulting
in incorrect/malformed data in
state['raw_recent_private_conversations'].
nlargest is the natural fit for selecting n biggest items
from an unsorted list. It's more readable as well as more
efficent (even though we don't care much about the efficeny
in this particular case).
The current logic doesn't display data types when the additionalProperties
variables are not object, but are array of strings, etc. Changed the if
condition to allow rendering in such cases.
I have a local branch with a hacked up version of
zjquery that lets you basically detect when zjquery
stubs are never actually invoked by real code.
There are some nuances to that kind of audit, so
I haven't pushed the auditing code, but these
are low hanging fruit.
I now just use inline the code to create stubs
for the line items in the markdown_content
container, and I don't add methods to the
zjquery stubs.
And then I use the new "children" feature in
zjquery's `$.create(sel, opts)` to set up
$(".markdown_content"), which means I don't
have to stub `each` any more.
It's actually pretty rare in our codebase to
call methods like `$(...).map` or `$(...).each`,
but we now support them better in zjquery.
You can pass a list of child elements now to
`$.create(...)`.
Fixes the sorting button labels in stream settings, which were
regressed by commit f8fbae4d8e (because
the HTML was not marked as being HTML).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
zerver/lib/users.py has a function named access_user_by_id, which is
used in /users views to fetch a user by it's id. Along with fetching
the user this function also does important validations regarding
checking of required permissions for fetching the target user.
In an attempt to solve the above problem this commit introduces
following changes:
1. Make all the parameters except user_profile, target_user_id
to be keyword only.
2. Use for_admin parameter instead of read_only.
3. Adds a documentary note to the function describing the reason for
changes along with recommended way to call this function in future.
4. Changes in views and tests to call this function in this changed
format.
Changes were tested using ./tools/test-backend.
Fixes#17111.
Merge the two "misc" sections into one and place all the service
configurations next to each other. Place the TERMS_OF_SERVICE
and PRIVACY_POLICY at the bottom of the "misc" section.
This is a minor refactor in the muting test in
`message_list`.
The `unmuted_messages` function filters out messages only
considering topic mutes, and not stream mutes.
The test previously made it look like we were testing
stream muting, by stubbing the `is_topic_muted` on the
basis of `stream_id`.
This also replaces the stub and uses real data instead.
This is a prep commit, which renames some variables
and functions involved in topic muting to include
the word "topic" in them.
This is done to have clarity when similar code
will be added as a part of the mute-user in
future commits.
If we don't pass `date_muted`, we shouldn't calculate
date_muted * 1000. This code used to work because of
how javascript treats `undefined`.
This commit deals with the `date_muted=undefined` case
in a cleaner manner.
This commit fixes a small bug in the
settings/muted-topics pane.
When there are zero muted topics, the
"You have not muted any topics yet." message
was not shown.
This is fixed by adding the `required-text`
class to the table body.
The bug was introduced in 3bc818b9f7.
Replaced methods/functions of moment.js with date-fns library.
The motive was to replace it with a smaller frontend timezone library.
Date-fns ~ 11.51 kb
moment.js ~ 217.87 kb
Some of the format strings change because date-fns encodes them
differently from how moment did.
Fixes#16373.
Previously, the data type of responses wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation. Here we add a recursive function to
render the data types visibly in API Documentation.
Fixes part of #15967.
The responses for the API weren't being rendered from yaml, and were
incorrectly formatted in yaml. The parameters also weren't completely
included in yaml and needed to be moved. Made appropriate fixes in
yaml and markdown file.
This commit adds the Zulip logo to the readthedocs
documentation by adding an .svg file (taken from
zulip.com) to the `docs/images` directory.
Morever, it removes the name `Zulip` which was
written at the top of the sidebar because the
logo already has that.
Also fix incorrect 2011 copyright years; the original Zulip, Inc. that
was merged into Dropbox, Inc. was incorporated in 2012, not 2011.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A LICENSE file is supposed to be an unmodified copy of the license,
and the license appendix explains that the copyright line should be
included with the notice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
On a high-DPI display or with a non-default zoom level, the browser
viewport may have a width strictly between md_max = 767px and md_min =
768px. Use only the *_min bounds for consistency.
This requires queries with strict inequalities to express upper
bounds (width < md_min). Fortunately, that functionality is provided
by range context queries. Unfortunately, those are not supported in
all browsers. Fortunately, we can compile them away using
postcss-media-minmax. Unfortunately, postcss-media-minmax currently
subtracts 1px for strict inequalities anyway to work around a Safari
rounding bug. Fortunately, 0.02px should be sufficient for that, so I
submitted a PR:
https://github.com/postcss/postcss-media-minmax/pull/28
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 434094e599 (#11321) changed this
from an Extension to a subclass of Markdown, so it no longer has any
reason to use a config dict structured like that of an Extension.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The 100 invite per day restriction is only for the free plan. Also,
the value 100 is configurable using
settings.INVITES_DEFAULT_REALM_DAILY_MAX. On top of this, newly
created realms on free plan combined can only send
INVITES_NEW_REALM_LIMIT_DAYS number of invites. So it's better not
to hardcode 100 in the doc.
It's not clear to me why this code was necessary,
and I assume it was either originally written
with a bit of misunderstanding of how zjquery
works or it became unnecessary with some refactoring
of the "real" code.
This commit migrates some of the backend tests to use assertLogs(),
instead of mock.patch() as planned in #15331.
Tweaked by tabbott to avoid tautological assertions.
There were some tests that had mock patches for logging, although no
logging was actually happening there. This commit removes such patches
in `corporate/tests/test_stripe.py`, `zerver/tests/test_cache.py`,
`zerver/tests/test_queue_worker.py`,
and `zerver/tests/test_signup.py`.
We move some of the data setup to the top of the file.
We also remove some get_sub() calls that aren't really
necessary now that peer_data and stream_data are more
independent.
The maybe_clear_subscribers() function was an artifact of
when we used to attach subscribers to the "sub" records in
stream_data.js. I think it was basically a refactoring
shim, and due to some other recent cleanup, it was only
used in test code.
We also change how we validate stream ids.
Going forward, peer_data just looks up stream_ids with the
normal stream_data API when it's trying to warn about
rogue stream_ids coming in. As I alluded to in an earlier
commit, some of the warning code here might be overly
defensive, but at least it's pretty self-contained.
In my recent commit to introduce get_user_set() I
inadvertently skipped one place to call it.
I also remove a return statement that was made
unnecessary by the new get_user_set() helper.
Now when we want to measure how long a block
of code takes to execute, we just wrap it with
`blueslip.measure_time`, instead of the awkward
idiom from my original commit of getting a callback
function.
My rationale for the original scheme was that I
wanted to minimize diffs and avoid changing
`const` to `let` in a few cases, but I believe
now that the function wrapper is nicer.
In a few cases I just removed the blueslip timing
code, since I was able to confirm on czo that
the times were pretty minimal.
We now use the same code in all places to
get the bucket of user_ids that correspond
to a stream, and we consistently treat
a stream as having zero subscribers, not
an undefined number of subscribers, in
the hypothetical case of us asking about
a stream that we're not tracking.
The behavior for untracked streams has
always been problematic, since if a
stream is untracked, all bets are off.
So now if we don't "track" the stream,
the subscriber count is zero. None of
our callers distinguish between undefined
and zero.
And we just consider the stream to be subscribed
by a user when add_subscriber is called,
even if we haven't been told by stream_data
to track the stream. (We also stop
returning true/false from add_subscriber,
since only test code was looking at it.)
We protect against the most likely source
of internal-to-the-frontend bugs by adding
the assert_number() call.
We generally have to assume that the server
is sending us sensible data at page load
time, or all bets are off.
And we have good protections in place
for unknown ids in our dispatch code
for peer_add/peer_remove events.
For the rare case where you're doing a link to a private
stream from a larger private stream that is a superset of
the former, we have bypassed warnings that you are linking
to a private stream.
I'm not sure we need this exemption for any situation
(just let the user bypass the warning), but we definitely
don't want false positives for the exemption.
For now I am closing down this loophole specifically for
Zephyr users.
Zephyr users are special in that we might not get
subscriber info on certain streams.
The current behavior for this edge case is a little
unclear. The current implementation of
peer_data.is_subscriber_subset returns false if both
streams are untracked, but most streams are tracked if we
have a sub for them and just get treated as having an
empty set of subscribers. And an empty set is a subset of
itself. Upcoming changes to our server data are gonna
make this edge case even more annoying to maintain.
We also streamline some of the error handling code
by doing everything up front. This will prevent
scenarios where a single bad stream_id/user_id causes a
bunch of the same warnings in an inner loop.
This removes a bit of complexity. If a piece of
settings code needs to render a stream with
subscribers, it just asks for it.
We no longer have the brittle, action-at-a-distance
mechanism of mutating the subscriber count on to
the stream_data version of a sub.
Stream subs are pretty small, so making copies of
them is cheap, and the blueslip timings from the
previous commit can help confirm that.
There is some discussion of putting `subscriber_count`
on the Stream model, which may eventually get us
away from tracking it in `peer_data.js`, but we will
cross that bridge when we get there. See
https://github.com/zulip/zulip/issues/17101 for
more details.
The weekly stream traffic is a better tiebreaker
for stream typeaheads than subscriber count, as
it's more directly a measure of a stream's current
relevance.
Normally stream traffic and subscriber counts are
closely correlated, but a good example for me is
the #twitter feed on czo, which only has 80 subscribers,
but which gets more traffic than our #integrations
stream (with 16k subscribers). I would rather
see #twitter win the tiebreaker (if it even got
to the tiebreaker).
The main motivation behind this fix, though, is
to break our dependency on peer_data, which has
some upcoming changes that will introduce some
performance tradeoffs, and I want one less place
to audit.
Also, it will be easier long term to share this
code with mobile if we don't require mobile
to pull in our peer_data dependency. (The webapp
has different forces than mobile that dicate
our data structures.)
The spinner icon is not visible until the user clicks on topic_edit_save,
so the space alloted to spinner-icon looks empty for rest of the time.
To improve the design, the spinner icon is only shown when the user
clicks on topic_edit_save.
EmailLogBackend used to create a new EmailMessage and copy
only certain values from the original EmailMultiAlternatives
object. This resulted in the loss of information and made
it harder to test PRs like
https://github.com/zulip/zulip/pull/17121.
So instead of creating a new EmailMessage, tweak and send the existing
EmailMultiAlternatives object.
Depending on PostgreSQL’s query plan, it was possible for the value
condition to be evaluated before the field_type condition was checked,
leading to errors like
psycopg2.errors.InvalidDatetimeFormat: invalid value "stri" for "YYYY"
DETAIL: Value must be an integer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This isn't quite the right model, because we're not actually going
through the upload code path, but it does at least provide some inline
image previews in the data.
Fixes part of #14991.
The changes are as follows:
• Fix one day offset in all western zones.
• Correct CST from -64800 to -21600 and CDT from -68400 to -18000.
• Disambiguate PST in favor of -28000 over +28000.
• Add GMT, UTC, WET, previously excluded for being at offset 0.
• Add ACDT, AEDT, AKST, MET, MSK, NST, NZDT, PKT, which the previous
code did not find.
• Remove numbered abbreviations -12, …, +14, which are unnecessary.
• Remove MSD and PKST, which are no longer used.
Hardcode the dict and verify it with a test, so that future
discrepancies won’t go silently unnoticed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We haven't actively used this plugin in years, and so it was never
converted from the 2014-era monitoring to detect the hostname.
This seems worth fixing since we may want to migrate this logic to a
more modern monitoring system, and it's helpful to have it correct.
Commit e941ee4a15 (#16680) incorrectly
converted this from 775px to xl-max = 1199px instead of md-max =
767px, causing misplacement of the FRB for browser widths between
these values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We use day_old calculated based on day instead of hours to
render last seen values. This fixes us incorrectly quoting
anything 24 - 48 hours ago as Yesterday and
incorrectly quoting `time` that are Yesterday
but < 24 hours ago in 'x hours ago' format.
We were adding `expanded` class to left-sidebar when searching
for streams even if the left-sidebar was not in the popover state.
This cased confusion with popovers.any_active returning true,
when actually it is not.
We explicitly bind `this` to MessageState class which otherwise
was defaulting to `window`.
This resulted in variables like `this.received` and `this.local_id`
being incorrectly interpreted by called function
as `window.(received | local_id)` which are `undefined`.
Hence, frontend thinks that the message was never received.
It was noticed since this was the common log message when
a double message send bug was observed. This change in no
was indicates fixing of the double send bug, but is hopefully
one step forward in that direction.
This reverts commit 5ad831fdf0 (part
of #2395).
This block is five years old and predates multiple Ubuntu upgrades and
our switch from LXC to Docker; now it reportedly breaks provisioning
on SELinux systems instead of fixing it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
https://docs.djangoproject.com/en/3.1/releases/3.1/
- django.contrib.postgres.fields.JSONField is deprecated and should be
replaced with models.JSONField
- The internals of the implementation in the postgresql backend have
changed a bit in
f48f671223
and thus we need to make an ugly tweak in test_runner.
- app_directories.Loader.get_dirs() now returns a list of PosixPath so
we need to make a small tweak in TwoFactorLoader for that (PosixPath
is not iterable)
Fixes#16010.
Adjustments made due to changes in Django 3.0:
(https://docs.djangoproject.com/en/3.0/releases/3.0/)
- test_signup: INTERNAL_RESET_URL_TOKEN was moved to
PasswordResetConfirmView.reset_url_token
- test_message_fetch:
"add_never_cache_headers() and never_cache() now add the private
directive to Cache-Control headers."
- "django.utils.html.escape() now uses html.escape() to escape HTML.
This converts ' to ' instead of the previous equivalent decimal
code '." - this requires adjusting the expected decimal code
in some of the string fixtures in tests.
79931051bd allows outgoing emails from
localhost, but outgoing recipients are still subjected to virtualmaps.
This caused all outgoing email from Zulip with destination addresses
containing `.`, `+`, or starting with `mm`, to be redirected back
through the email gateway.
Bracket the virualmap addresses used for local delivery to the mail
gateway with a restriction on the domain matching the
`postfix.mailname` configuration, regex-escaped, so those only apply
to email destined for that domain.
The hostname is _not_ moved from `mydestination` to
`virtual_alias_domains`, as that would preclude delivery to
actually-local addresses, like `postmaster@`.
topic_generator previously included an entire lazy generator
combinator library that was used four times. These straightforward
equivalent loops might not be as fun but they are way simpler.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since the bots list breaks at 625px with left col of settings
hidden and at 850px with left col of settings visible
without this media query,
having this media query trigger at 991px shouldn't produce
any problems.
This media query aligns filter buttons on tablets. Since tablets
range are usually less than 991px in width, this breakpoint makes
more sense. Also, the filter buttons looks nice between 1033px
and 991px.
Instead of adjusting the width of settings container at
1130px, we adjust it at 991px which is the standard for smaller
screens. Adjusting it 1200px(xl) didn't make sense, so the
next reasonable breakpoint was used.
Since alert-box has size of 900px on large devices, we had to
reduce its size for devices less than 900px. Setting this
breakpoint to 991px, shouldn't break anything.
The changes were not live tested since search pill is not enabled.
However, this should look fine since the properties changed
were not depended upon 500px.
We use 767px for hiding left column.
The components changed here were tested to be working fine.
This change is not likely to introduce any regression as the
calculations in the components here were not dependent upon the
breakpoint being at 775px.
We use 1199px for hiding right column.
The components changed here were tested to be working fine.
This change is not likely to introduce any regression as the
calculations in the components here were not dependent upon the
breakpoint being at 1165px.
This code was intended to hide the stream description on stream
settings overlay on display <350px; but, the css selector for
should be `.stream-row` instead of `.stream_row`. Anyway, the
overlay looks fine on small devices with stream description.
Hence, we just remove this dead code.
'>' and 'e' are added as hotkey hints for 'Quote and
reply or forward' and 'View source / Edit topic' options
in actions popover, to help make these hotkeys more discoverable.
This commit updates the Zulip User-Agent to
'Mozilla/5.0 (compatible; ZulipURLPreview/{version}; +{external_host})'
as the older User-Agent was rendering Markdown YouTube titles as
'YouTube - YouTube'.
Fixes#16970.
c2526844e9 removed the `signups` queue
worker, and the command-line tool that enqueues to it -- but not the
automated process that enqueues during signups itself.
Remove the signup, since it is no longer in use.
Previously, the data type of parameters wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation. Here we add a recursive function to
render the data types visibly in the API documentation.
This only covers the request parameters; we'll want to do something
similar for response parameters in a follow-up PR.
Fixes part of #15967.
While adding custom emojis, when a user clicks on the submit
button without providing a name to the emoji, the submit button
becomes unresponsive. This commit fixes that.
Fixes#16921
When we were getting an apply_event call for
a subscription/add event, we were trying not to
mutate the event itself, but this clumsy code
was still mutating the actual event:
# Avoid letting 'subscribers' entries end up in the list
for i, sub in enumerate(event['subscriptions']):
event['subscriptions'][i] = \
copy.deepcopy(event['subscriptions'][i])
del event['subscriptions'][i]['subscribers']
This is only a theoretical bug.
The only person who receives a subscription/add
event is the current user.
And it wouldn't have affected the current user,
since the apply_event was correctly updating the
state, and we wouldn't actually deliver the event
to the client (because the whole point of apply_event
is to prevent us from having to piggyback the
super-recent events on to our payload or put
them into the event queue and possibly race).
The new code just cleanly makes a copy of each
sub, if necessary, as we add them to state["subscriptions"].
And I updated the event schemas to reflect that
subscribers is always present in subscription/add
event.
Long term we should probably avoid sending subscribers
on this event when the clients don't set something
like include_subscribers. That's a fairly complicated
fix that involves passing in flags to ClientDescriptor.
Alternatively, we could just say that our policy is
that we never send subscribers there, but we instead
use peer_add events. See issue #17089 for more
details.
It's always cleaner to work in id space. It probably
would have required a perfect storm to have broken
the existing code, but using ids is obviously more
robust in theory, and just as simple.
We now require keywords, so that there is no
pitfall for mixing up boolean parameters.
Positional parameters are basically evil
when you have a bunch of bools.
I also make user_profile the first argument.
Finally, the code is more diff-friendly.
I eliminate the defaults, since the existing code
was already specificying values for most things.
I move all the booleans to the bottom for both
parameters and arguments.
I require explicit keywords for everything but
user_profile (which is now first).
And, finally, I format the code in a more
diff-friendly manner.
We eliminate some redundant checks.
We also consistently provide a `subscribers` field
in our stream data with `[]`, even if our users
can't access subscribers. We therefore bump
the API version and tweak the docs. (See further
down for a detailed justification of the change.)
Even though it is sometimes fine to have redundant code
that is defensive in nature, some upcoming changes are gonna
move subscriber-related logic out of build_stream_dict_for_sub
for certain codepaths as part of our effort to streamline
the payload for subscribers within page_params.
So we can't rely on the code that I removed here
inside of build_stream_dict_for_sub.
Anyway, it makes more sense to do these checks explicitly
in the validate function.
The code in build_stream_dict_for_sub was almost effectively
a noop, since the validation function was already preventing
us from getting subscriber info. The only difference it
made was sometimes converting `[]` to `None`, and then
subsequently omitting the subscribers field.
Neither ZT nor the webapp make any distinction between
`[]` or <missing key> for the `subscribers` data in
`page_params`.
The webapp has had this code for a long time (and now
equivalent code elsewhere in this PR):
if (!Object.prototype.hasOwnProperty.call(sub, "subscribers")) {
sub.subscribers = new LazySet([]);
}
The webapp calculates access based on booleans, anyway:
sub.can_access_subscribers =
page_params.is_admin || sub.subscribed ||
(!page_params.is_guest && !sub.invite_only);
And ZT would choke if `subscribers` were missing, except that
it never gets to the relevant code due to other checks:
def get_other_subscribers_in_stream(<snip>):
assert stream_id is not None or stream_name is not None
if stream_id:
assert self.is_user_subscribed_to_stream(stream_id)
return [sub
for sub in self.stream_dict[stream_id]['subscribers']
if sub != self.user_id]
else:
return [sub
for _, stream in self.stream_dict.items()
for sub in stream['subscribers']
if stream['name'] == stream_name
if sub != self.user_id]
You could make a semantic argument that we should prefer
<missing key> to `[]` when subscribers aren't even available, but
we have precedent from the way that `bulk_get_subscriber_user_ids`
has traditionally populated its result:
result: Dict[int, List[int]] =
{stream["id"]: [] for stream in stream_dicts}
If we changed `stream_dicts` to `target_stream_dicts` we
would faciliate a move toward `None`, but it would just cause
headaches for other server code as well as the frontends
(which, to reiterate, already prefer the empty array
for convenience).
As my comment indicates, I would prefer to handle
this explicitly by raising JsonableError in an
else statement here, but it's not a big deal.
This function can probably be simplified with a
bit of work, mostly on the testing side to make
sure we are covering all edge cases, but that
is out of the scope of my current PR.
By moving the relevant logic from realm.get_bot_domain to
get_fake_email_domain we will make realm.host be used (if possible) for
dummy user addresses. That is, instead of user11@zulipchat.com, the
address will become user11@subdomain.zulipchat.com.
With the change in d70e1bcdb7,
bots get email like bot@zulip.com with EXTERNAL_HOST="zulip.com",
rather than bot@subdomain.zulip.com, which was the old format. That's
not desirable, so with this commit, realm.host will be used when
possible and only falling back to FAKE_EMAIL_DOMAIN if needed.
We often send only one field (away or status_text)
to be updated.
So we have to make our schema support optional
keys.
As a result of the more flexible schema, we no
longer need to exempt the node fixtures from
our schema checks.
Commit f0f6138f01 deleted the
translation for "Saved. Please <a class='reload_link'>reload</a> for
the change to take effect." which caused puppeteer test 16-settings to
fail as it had hardcoded the translation for German. This commit
changes the expected text and hence fixes the failing puppeteer test.
When we switch subscription overlay from two column to one
column overlay, we also set stream buttons to show in next line.
575px because it the breakpoint used by bootstrap 4 for small
screens.
Since recipient_id (id of the PERSONAL Recipient of the user) was
denormalized into the UserProfile model, this query can be simplified by
getting rid of the zerver_recipient JOIN.
This makes us more efficient when handling
multiple users. We don't have to keep
sending the same two queries to the database.
Note that as part of this we eliminated
a failure mode for the obscure population
of users from whom both `user.is_guest` and
`user.can_access_public_streams()` returns
False. We know this would have only affected
Zephyr users (by looking at the code), and
we know we don't actually process Zephyr
users for email digests (or else we would
have raised exceptions in the old code).
We mostly need realm_id, but when we go to build
message lists, we need realm.uri.
We could probably be more aggresive about using
`only` here, but for now I am just trying to
reduce hops to the database.
The `deployment` key was only set in `do_report_error`, which is now
only used in one codepath (the queue worker). The logging handlers on
staging call notify_server_error directly, which omits the
`deployment` key.
Remove the odd one-of key, and instead simply do dispatch in
`do_report_error`.
After this change all peer_data functions consistently
use stream_id rather than some "sub" object whose
data type is complicated by all sort of fields that
don't really concern how we track subscribers.
The goal here is to make all our peer_data functions
basically work in id space. Passing a full `sub`
to these functions is a legacy of when subscriber
info was attached to a full stream "sub" object,
but we don't care about anything sub-related
(color, description, name, etc.) when we are
dealing with subscriptions.
When callers pass in stream_id, you can be more
confident in a quick skim of the code that we're
not mutating anything in the "sub".
This de-clutters stream_data a bit. Since our
peer data is our biggest performance concern,
I want to contain any optimizations to a fairly
well-focused module.
The name `peer_data` is a bit of a compromise,
since we already have `subs.js` and we use
`sub` as a variable name for stream records
throughout our code, but it's consistent with
our event nomenclature (peer/add, peer/remove)
and it's short while still being fairly easy
to find with grep.
This sets us up to use better system-wide data structures
for tracking subscribers.
Basically, instead of storing subscriber data on the
"sub" objects in stream_data.js, we instead have a
parallel data structure called stream_subscribers.
We also have stream_create, stream_edit, and friends
use helper functions rather than accessing
sub.subscribers directly.
We now use add_sub only in tests.
The line to defensively initialize subscribers does
not get copied from add_sub, since we know that
create_sub_from_server_data always initializes
subscribers via set_subscribers.
The codepath for moving a topic changes the message.recipient_id to the
id of the new recipient, but later, in update_messages_for_topic_edit,
it uses message.recipient when querying for messages with the matching
topic in the *old* stream (because those are the other messages that
need to be moved). This is a bug which happens to work fine, because in
Django 2, if message.recipient gets fetched first and then
message.recipient_id is mutated, message.recipient will not be altered
and thus will retain the outdated, previously fetched value.
In Django 3 changing .recipient_id causes .recipient to be updated to
the new Recipient objects, which is the Recipient of the *new* stream.
That will cause the bug to manifest.
This is a bugfix preparing for the upgrade to Django 3.
Support for saving it in the session is dropped in django3, the cookie
is the mechanism that needs to be used. The relevant i18n code doesn't
have access to the response objects and thus needs to delegate setting
the cookie to LocaleMiddleware.
Fixes the LocaleMiddleware point of #16030.
We now require explicit keywords for all arguments
to fetch_initial_state_data except user_profile.
We provide reasonable defaults to keep the test
code concise.
In the case of reusing a registration link, reuse the
redirect_to_email_login_url helper. This does have the side effect of
now showing a "you've already registered" note, which did not happen
previously, but that seems probably for the best, since the user did
just click a "register" link.
ecfafc05c0 shifted to using a different paramter name to hint that
the user had previously signed up -- and in so doing also stopped
pre-filling the "email" box. Also send along the email box, to save
users time.
Checking for `validate_email_not_already_in_realm` again (after the
form already did so), but only in the case that the form fails to
validate, means that we may be spending time pushing totally invalid
emails to the DB to check. In the case of emails containing nulls,
this can even trigger a 500 error from PostgreSQL.
Stop calling `validate_email_not_already_in_realm` in the form
validation. The form is currently only used in two places -- in
`accounts_home` and in `maybe_send_to_registration`. The latter is
only called if the address is known to not currently have an account,
so checking in there is unnecessary; and in the former case, we wish
different behaviour (the redirect) than just validation failure, which
is all the validator can do.
Fixes#17015.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
Add a `--allow-reserved-subdomain` flag which allows creation of
reserved keyword domains. This also always enforces that the domain
is not in use, which was removed in 0258d7d.
Fixes#16924.
When changing the subdomain of a realm, create a deactivated realm with
the old subdomain of the realm, and set its deactivated_redirect to the
new subdomain.
Doing this will help us to do the following:
- When a user visits the old subdomain of a realm, we can tell the user
that the realm has been moved.
- During the registration process, we can assure that the old subdomain
of the realm is not used to create a new realm.
If the subdomain is changed multiple times, the deactivated_redirect
fields of all the deactivated realms are updated to point to the new
uri.
Instead of just storing the edit history in the message which
triggered the topic edit, we store the edit history in all
the messages that changed. This helps users track the edit history
of a message more reliably.
This change updates the GitHub Integration webhook
get_opened_or_update_pull_request_body method so that
the description is only printed if it actually changes.
If the update event is a result of some other
attribute update, such as an asignee change, then the
description is not included in the message sent to
the zulip stream.
Fixes#16345
Concretely, we'll use this with a `UserId` type which is an
"opaque type alias" of `number` -- it's secretly implemented as
simply `number`, and it can be consumed by anything that wants a
`number` (in other words, it's a subtype of `number`), but the
fact that it secretly just is `number` is private to the module
that defines the type.
As far as the typing_status code is concerned, allowing this to
be a subtype of `number` just means that the code doesn't ever
try to inject new numbers of its own into the recipients arrays
that it passes around.
This type means that code consuming this value promises not to
mutate it. It's useful partly for the sake of simply controlling
mutation, so that arrays can be passed around without making
defensive copies; and partly because it makes the type covariant
in the elements, rather than invariant.
That is, if a function takes a plain Array<number | null>, then you
can't pass it an Array<number>, because it might add a `null` to it.
But if it takes $ReadOnlyArray<number | null>, then you can.
In general, Array<S> <: $ReadOnlyArray<S> <: $ReadOnlyArray<T>
for any S <: T, where `<:` means "is a subtype of".
Marking this type as read-only means we can pass in a read-only
array without adding a fixme (equivalent to a mypy type-ignore) to
locally disable the type-checker, nor a redundant defensive copy.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
Fetchings rows with end_time within the last 25 hours would result
in the realmcount queries returning two rows for each realm
if the analytics page was opened within an hour since the
count stats were updated.
This is a prep commit. Currenty we only pass CountStat.property
to last_successful_fill function. But it needs access to
CountStat.time_increment as well. We can pass the entire CountStat
object to the function as a workaround. But making last_successful_fill
a property of CountStat seems to be much more cleaner.
Allowing any admins to create arbitrary users is not ideal because it
can lead to abuse issues. We should require something stronger that
requires the server operator's approval and thus we add a new
can_create_users permission.
We change the return type of check_message to be dataclass instead of
Dict[str, Any]. This refactoring helps us to understand the context of the
data structure returned by check_message clearly which was not possible
when using Dict.
SendMessageRequest class is added in zerver/lib/message.py inspite of it
not being used in that file itself just to maintain consistency as other
TypedDicts and dataclasses are defined in that file and to avoid circular
dependency as SendMessageRequest is being used in lib/widget.py as well.
We also rename local variable to 'send_request' for accessing
SendMessageRequest objects.
The {addr} part isn't directly useful, since connections to Tornado
are done on localhost anyway, and made the development environment
output a bit more confusing.
Also, use the same phrasing for restarts we use for Django.
We run this tool at DEBUG log level in production, so we will still
see the notice on startup there; this avoids a spammy line in the
development environment output..
This logging is really only potentially interesting in a development
environment when the numbers are nonzero.
In production, it seems worth logging for consistency reasons.
Probably we'll eventually redo this block by change the log level, but
this is good enough to despam the development environment startup
output.
It seems the Ubuntu base image we use now has a new enough VirtualBox
Guest Additions to trigger the ETXTBSY bug even when it’s not upgraded
by the vagrant-vbguest plugin. Provide and document a way to
downgrade it.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
We always want to do these at the same time. Previously, message
editing did too much stripping (fixes#16837) and failed to check for
NUL bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously we were just returning a dict containing a message id when
trying to mirror a already sent message in 'zephyr_mirror' cases.
This commit changes this behaviour to raise an exception when trying
to mirror an already sent message by adding a new exception class
ZephyrMessageAlreadySentException and then the caller returns the
message_id directly, instead of calling do_send_messages which also
returns a list of size one containing the message_id only.
This is a prep commit for changing the return type of check_message to
be a dataclass instead of a Dict as now we have only single output for
check_message.
This commit renames the content variable in do_widget_post_save_actions
to message_content and is a prep commit for changing the return type of
check_message from Dict to dataclass.
This change is required because content variable is used two times in
this function - one for message content and other for submessage
content, so when we change the return type of check_message to
dataclass, the type of content variable is considered as str and then
when dict is assigned to content in the submessage case, mypy raises
'Incompatible types in assignment' error.
This issue is not faced before the dataclass migration because there is
no type checking for the values of dict returned by check_message as the
return type of check_message is 'Dict[str, Any]'.
The message_dict['wildcard_mention_user_ids'] should be empty set instead
of empty list when there are no wildcard mentions similar to the case
when there are wildcard mentions, where it is equal to set of user ids and
not list of user ids.
After exiting lightbox view by pressing the browser back button,
future requests to open images were failing. This was because the
handler called on back button press- close_for_hash_change() was not
closing the currently open overlays gracefully. This commit fixes the
problem by calling the close_handler function inside
close_for_hash_change().
Fixes#16726
The set status modal to add/remove/update user status was not
visible properly on devices with a small width. This commit fixes
the issue by adding appropriate media queries to the css to make
the modal mobile responsive.
Fixes part of #16817.
Discount is applied relative to the price per license of our normal
plans. For fixed price plans, the concept of discount doesn't make
any sense since we manually assign a price for the entire realm
irrespective of the number of users in the realm.
f82cc4ed06 started checking all
zulip/zulip GitHub links in CI. Instead, it should have checked only
zulip/zulip file and directory links since checking other
links require making requests to GitHub servers.
This fixes a bug where the typeahead did not include the
zulip specific langs (such a `quote`, `spoiler` and `math`)
as these weren't passed to the typeahead's source.
Introduced in af64c52166.
Fixes#16862.
I reformatted the tests and view to include information about who
acknowledged and closed the alert. Only includes the information about
the owner if there was an owner.
Made a few small changes to the refactored bit as requested in review.
Moved time formatting check and conversion to
zerver/lib/webhooks/common.py. Updated tests slightly to match new
output. Removed duration from the calculation because the difference
is less than the precision of output and it complicated the error
handling.
The Slack API always (even for failed requests) puts the access scopes
of the token passed in, into "X-OAuth-Scopes"[1], which can be used to
determine if any are missing -- and if so, which.
[1] https://api.slack.com/legacy/oauth-scopes#working-with-scopes
An HTML document sent without a charset in the Content-Type header
needs to be scanned for a charset in <meta> tags. We need to pass
bytes instead of str to Beautiful Soup to allow it to do this.
Fixes#16843.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
If a user visits a realm which has been deactivated and it's
deactivated_redirect field is set, we should have a message telling the
user that the realm has moved to the deactivated_redirect url.
We export a realm's data, and disable the realm, because the user
is moving from Zulip Cloud (e.g. https://example.zulipchat.com/) to
self-hosting or another platform (e.g. https://zulip.example.com/)
which we do not control. This commit adds a field in the realm object
called deactivated_redirect to store the url to which the realm has
moved.
At /devtools 'Connecting to the local PostgreSQL database',
path for `dev-secrets.conf` should zulip/zproject/dev-secrets.conf
instead of zulip/zerver/dev-secrets.conf.
I added these hooks in Zulip Desktop 5.5.0; handling these events in
the frontend will let us remove the janky desktop-side fallback code
that uses fake click events on menu items with specific indexes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 13c11ec5f3 (#16699) already fixed
the generated curl examples, but missed this, which is the only
hard-coded one.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This handles the conditions when anchor values are larger than
LARGER_THAN_MAX_MESSAGE_ID by clamping them down to it. Also added
tests for the function parse_anchor_value.
Fixes#16768.
This simplifies the code, as it allows using the mechanism of converting
JsonableErrors into a response instead of having separate, but
ultimately similar, logic in RateLimitMiddleware.
We don't touch tests here because "rate limited" error responses are
already verified in test_external.py.
In 1bcb8d8ee8 I made
it so the webapp doesn't include "streams" in its
state from `fetch_initial_state_data`, but I didn't
address all the places in apply_event.
By default all Stripe API amounts are in the currency's smallest unit.
It's upto us to convert it to a bigger unit and show it to the end user.
And refund event used to show the currency in the smallest unit which makes
the output wrong when it comes to most currencies like USD, Europ, INR etc
which uses a bigger unit(eg Dollar instead of Cents) as the standard.
rustc's default edition is 2015 to preserve backwards compatibility, and
the playground appears to follow this scheme. However, 2018 edition Rust
is the current standard and is the default that Cargo uses when
initializing new projects. It adds support for various features,
including async/await and a new module system. As a result, I think
Zulip should default to 2018 edition when linking to the playground.
Users can always select a different edition once in the playground if
they would like.
There are actually two dropdowns for wildcard mention setting,
which would have been added mistakenly while changing the label
and position during merging the original commit for adding this
setting.
This commit removes the extra dropdown dropdown in Other
Permissions subsection and retains the one in Stream
Permissions subsection.
This refactoring should have no functional effect for any call points,
but makes the function behave more naturally. The comments explain
the situation, but specifically:
* There's the page_params.narrow hack that affects both narrows and
home_msg_list.
* There's the shared data for home_msg_list and all_msg_list that
requires we modify the query from home_msg_list.data.public_operators().
And otherwise the logic should just use the operators associated with
the message_list.data object (allowing us to remove the force_fetch
hack added in the last commit).
Hopefully in some future refactoring, we'll be able to migrate those
hacks to live in the Filter object construction and eliminate this
block of conditionals entirely.
In commit ebea17b9a6,
we added an extra fetch to get accurate data for the top
items in recent topics table.
But the `narrow` parameter wasn't passed to the endpoint,
this resulted in fetching the user's overall message
history including the muted streams/topics which aren't
required by the recent topics table.
`operators` can be replaced as we set the same value for
the `narrow_state` module and the narrowed message list's
filter, when activating the narrow.
`wal-g wal-push` has a known bug with occasionally hanging after file
upload to S3[1]; set a rather long timeout on the upload process, so
that we don't simply stall forever when archiving WAL segments.
[1] https://github.com/wal-g/wal-g/issues/656
Update the New Relic webhook and tests to match the format specified
in the New Relic documentation. The new format sends a json body
instead of using url parameters. The old format is no longer supported
by New Relic according to their support staff; as a result, the fixtures for
the old test cases were removed. Added fixtures for new test cases.
Fixes: #16393.
We seem to periodically get contributors who rebase upstream commits
onto their pull request rather than the other way around, resulting in
a lot of GitHub noise. The PRs where this happens were made from
branches named master. We have always documented that you should work
on a feature branch, but not from this page; maybe this will help
reduce that kind of confusion.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The changes made in this commit are as follows:
* The `remove_messages` is moved to the `message_events.js`
file from `ui.js`.
* We refactor `MessageListData.change_message_id` to no
longer require an `opts` parameter as this function
just returns whether we need to rerender or not.
The blueslip error block can be removed since we made
the change to no long defer the data updates in
commit 3b5ba6b2c1,
this case can no longer occur.
The changes made in this commit are as follows:
* We remove the now unused `ui.find_message` which was added
in commit 1666403850.
* We change the function paramter to now accept message ids
instead of messages to eliminate redundant message ids to
message convertion as only the id is required.
* The remove method in MessageListData did not remove the
messages from the hash, it removed only from the items,
this fixes it.
* This commit also fixes a bug where messages are not added
to the current message list if an event is recieved where
messages are moved to this current narrow.
Only the message removal logic was present, which has been
refactored in this commit.
For 3000 messages and 400 users, this saved
about 30 seconds.
We only do two queries per batch of messages
now, and the algorithm is easier to analyze,
as it's just three nested loops.
Note that we are much more efficient about finding
active users here:
- we do one query per realm (instead of per-user)
- we pass the cutoff date to the database
- we get back just a list of distinct ids
This function is going away completely soon. It is
querying everybody's entire UserActivity history instead
of passing the cutoff date to the database!
The query counts increase here for somewhat
contrived reasons. The tests before this
commit reflected a successful trip to the
UserProfile cache, but that's not actually
realistic in practice.
We don't need to mock the dates here. We also
explicitly clear out all streams first, and then
we explicitly test with both the stream being
current and the stream being old.
We can use the _enqueue_emails_for_realm helper
to avoid all the Tuesday-related logic here.
We also don't bother to create UserActivity
records, since the bot gets excluded by virtue
of its being a bot. (Also, the date ranges
here were sketchy due to the time mocking.)
We can avoid all the date mocking now for all
but a couple tests that exercise the is-it-Tuesday
logic.
And this test now correctly tests that we exclude
recently active users.
And this allows us to remove the other test.
Logging `Host` is useful for determining access patterns to realms,
especially if ROOT_DOMAIN_LANDING_PAGE is set. Total response time is
useful in debugging access and performance patterns.
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
Sentry allows adding simple webhooks without going through the process
of creating an Internal Integration in Sentry's Integration
Platform[1] (which our docs recommend).
The payload from sent from such a (simple) webhook integration is
slightly different from the payload sent by an Internal Integration
webhook. This commit tries to wrangle this payload into a form that is
usable by our webhook handler to send a notification message.
[1]: https://sentry.io/integration-platform/
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.
This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.
Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.
Fixes#14845.
The code we deleted here was no longer
doing anything.
Maybe the code was always dead, or maybe it
was written during a time when topics_by_diversity
and topics_by_length actually had different keys.
But now it's clearly cruft.
If we have 4 or more topics, then the code above
it would already have populated the list with 4
elements, and the `if num_convos < 4` condition
would evaluate to False.
And if we had 3 or fewer topics, then we would
have already put all possible topics into our
result, and the `topics_by_diversity[num_convos:4]`
slice would be empty.
It's possible that we should just have a simple
heuristic for topic hotness like `10*num_senders
+ messages`, so we don't have to maintain this
fiddly function, and we can just do something like
`topics_by_score[:4]`.
I now use sets for stream_ids in more of the digest
code.
As part of this I replaced exclude_subscription_modified_streams
with streams_recently_modified_for_user.
It's easier for the caller to just ask for ids
to delete from its callee than it is to pass
in a set/list to mutate.
The simpler boundary between the functions makes
the tests easier to write--you can see the
`filtered_streams` logic goes away in this diff.
I also make the tests a bit more thorough by using
combinations of Cordelia/Othello and Verona/Denmark
to try to find multiple possible flaws.
And I make the time intervals longer than 1s to
avoid false negatives from slow CI boxes.
Currently, the Stream Name change isn't reflected in the streams
sidebar when a stream is renamed if the order of streams in the
sidebar remains unchanged, because the optimization to avoid
rerendering when nothing changes about the order prevents the
rerendering code from running.
We fix by this adding a flag in build_stream_list and
update_streams_sidebar functions to force a rerender, and pass that
when a stream is renamed.
Fixes#16026.
The comment explains the problem statement in some detail, but
basically this algorithm ensures that the top items in "Recent Topics"
on page load are always the very most recent topics the user has
received messages in (well, ignoring muted topics in this iteration).
The line number was outdated and was linking to totally unrelated
section. I think the best way to handle this case would be to
link directly to search.
If we have multiple users, this reduces the amount
of queries we need to do, because we get all
subscriptions for all users in a single query
to Subscription.
For the single-user case, we are introducing an
extra query hop, but the database is doing
roughly the same work, because we are just breaking
up this complex query into two hops:
messages =
select ... from message
where recipient__type_id in (
select stream_id from subscription
where ...
)
Now it's more like:
stream_ids =
select stream_id from subscription
where ...
messages =
select ... from message
where recipient__type_id in stream_ids
Note that we are not changing anything semantically
or algorithmically yet. The only overhead here
for the single-user case is boxing and unboxing
data into single-item dicts and lists.
The interfaces for callers in the view and the
queue processor remain the same for now.
We didn't need the enough-traffic mock.
We also continue to prep for testing multiple users.
I also finally remove a comment that is about to
be addressed (and which inaccurately refers to huddles).
This extraction will make a bit more sense when
we start doing bulk operations on a realm to
get digests, but even now, it encapsulates the
slightly complex way we cherry-pick the top 4
topics for a user.
This prep step is mostly for diff hygiene; the next
commit will make the code a bit nicer.
The original code here had the nice property that
most (but not all) of the DB work happened up
front in `handle_digest_email`, and none of the
DB work was delegated to the callers. But I
prefer the tradeoff of making the helpers a bit
more cohesive--let them get the data they need.
And we have query-count coverage in our tests,
so there's no real danger of having helpers
down in the stack insidiously doing a bunch of
extra DB hops.
In 709493cd75 (Feb 2017)
I added code to render_markdown that re-fetched the
sender of the message, to detect whether the message is
a bot.
It's better to just let the ORM fetch this. The
message object should already have sender.
The diff makes it look like we are saving round trips
to the database, which is true in some cases. For
the main message-send codepath, though, we are only
saving a trip to memcached, since the middleware
will have put our sender's user object into the
cache. The test_message_send test calls internally
to check_send_stream_message, so it was actually
hitting the database in render_markdown (prior to
my change).
Before this change we were clearing the cache on
every SQL usage.
The code to do this was added in February 2017
in 6db4879f9c.
Now we clear the cache just one time, but before
the action/request under test.
Tests that want to count queries with a warm
cache now specify keep_cache_warm=True. Those
tests were particularly flawed before this change.
In general, the old code both over-counted and
under-counted queries.
It under-counted SQL usage for requests that were
able to pull some data out of a warm cache before
they did any SQL. Typically this would have bypassed
the initial query to get UserProfile, so you
will see several off-by-one fixes.
The old code over-counted SQL usage to the extent
that it's a rather extreme assumption that during
an action itself, the entries that you put into
the cache will get thrown away. And that's essentially
what the prior code simulated.
Now, it's still bad if an action keeps hitting the
cache for no reason, but it's not as bad as hitting
the database. There doesn't appear to be any evidence
of us doing something silly like fetching the same
data from the cache in a loop, but there are
opportunities to prevent second or third round
trips to the cache for the same object, if we
can re-structure the code so that the same caller
doesn't have two callees get the same data.
Note that for invites, we have some cache hits
that are due to the nature of how we serialize
data to our queue processor--we generally just
serialize ids, and then re-fetch objects when
we pop them off the queue.
The visual noise from the blue border has bothered me forever and I
finally decided to do something about it. I don't know if this is the
best solution, but I do think it's a lot better than the status quo!
This prevents Zulip CI from eventually consuming large amounts of
storage on one's GitHub account.
I picked a longer retention period for the Puppeteer artifacts because
humans look at those; the production tarballs are unlikely to be used
10 minutes after the run completes as they are just for the next stage
fo the build; certainly 14 days seems ample for any debugging.
The passwords generated for our development environment / test suite
include the `+` character, which needs to be quoted when encoded as an
HTTP POST parameter.
This is hopefully sufficient to fix the CI failures we've seen with
the tests for POST /api/v1/fetch_api_key; I haven't reproduced the
failure so am not completely sure.
This tells users how autofix errors for linters which support it.
This is important since only way to fix prettier errors is
running lint with `--fix` which now the linter will gladly print
with the error.
Steve asked me to remove this, since the tictactoe game was always
intended as a proof of concept. Now that we have poll and todo
widgets, the sample code for tictactoe has much less value.
We replace the content and type in test_widgets.py to maintain
coverage.
Strings should be escaped at the point of interpolation into a
template, not before. In this case, the early escape was hiding the
bug that code_language was only escaped if it was not found in
pygments_data.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reverts commit 564b199fe6, which
was part of #16308.
Escaping is either required or incorrect; it is never “defensive”.
This escaping is incorrect. lxml already escapes attributes during
serialization (any other behavior would be a serious bug), and
additional escaping just results in double escaping.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A convenient copy-to-clipboard button was added in the Invite users to
Zulip modal, to make it slightly more convenient to share the
generated links.
The formatting is extracted to a template to make i18n and variable
substitution simpler.
Tweaked by tabbott significantly to simplify JS, HTML, and CSS.
Fixes#16442.
When compose box is open we already set
```
$(".new_message_textarea").css("min-height", "3em");
```
in compose_actions.js.
So, this property actually reduces the min-height slightly which
hides the topic when it is long enough to span in two lines.
This property never gets used and is not necessary since it
is overridden by
```
.button.small {
font-size: 1em;
padding: 3px 10px;
}
```
in compose.css
which looks good enough.
Initally, when writing two or more quotes, having
a blank line in between them, merges those quotes.
This created confusion especially in "quote and reply".
This commit fixes such issues. Now two or more quotes
having a blank line in between them, will not get merged.
This change is correct both for usability and for improving our
compatibility with CommonMark.
Fixes#14379.
By registering a post_delete handler to clear appropriate caches in a
nicer way, we can get rid of the ugly flush-memcached call in the
delete_realm command.
Color-picker overflows the screen width when an user
attempts to change color of the stream in small devices.
Fixed by making it fullscreen in narrow devices.
Fixes#16477
Floating upwards caused a weird flickering effect if the mouse floated
onto the tooltip's body, and it's still reasonable UI floating left
(and also there's guaranteed to be space).
Fixes#16438.
This commit adds migration which removes default status of exisitng
default private streams, i.e. private stream exists but they are no
longer default.
This commit removes mock.patch with assertLogs().
* Adds return value to do_rest_call() in outgoing_webhook.py, to
support asserting log output in test_outgoing_webhook_system.py.
* Logs are not asserted in test_realm.py because it would require to users
to be queried using users=User.objects.filter(realm=realm) and the order
of resulting queryset varies for each run.
* In test_decorators.py, replacement of mock.patch is not done because
I'm not sure if it's worth the effort to replace it as it's a return
value of a function.
Tweaked by tabbott to set proper mypy types.
This commit moves the wildcard mentions documentation to a top-level page.
Edited by tabbott to deduplicate with the existing docs, and add cross-links.
Then because the ID is now part of the draft dict, we can
(and do) change the structure of the "drafts" parameter
returned from `GET /drafts` from an object (mapping ID to
data) to an array.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Sometimes we don't need to specify the expected_drafts field.
So by removing it, we can reduce the clutter a bit.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Now the timestamp returned in a draft dict will always be an int.
The endpoints will still accept either an int or a float.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This fixes a bug where the autocomplete for topics
deleted all the text content, if the topic jump is used
without entering any text.
The topic typeahead is automatically set up, on entering
the ">" key for stream completions. Therefore there is a
case where the user can select a typeahead item without
entering any text.
Thus the token length will be 0 and `beginning.slice(0, -0)` returns
"" instead of the `beginning` string. The case is only relevant for
"topic_list" completion as we don't set up the typeahead for empty
strings.
Fix this by reverting a hunk of
48f5e5179a, adding a test.
Fixes#16599.
Co-authored-by: Rohitt Vashishtha <aero31aero@gmail.com>
Our test-backend validation confirms that we don't log anything to
stdout in the tests, so the fact that CI passes with this removes
shows there was nothing being logged.
Refactor test_video_link_compose_clicked into seperate tests for:
No video provider.
Jitsi as the provider.
Zoom as the provider.
BigBlueButton as the provider.
Rename zoom_xhrs to video_call_xhrs.
Rename abort_zoom to abort_video_callbacks.
Delete callbacks from video_call_xhrs when they have been aborted.
Move generation of video_call_id in the .videolink handler into
the Jitsi video call handling block as it is the only place it is
referenced.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.
We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.
Fixes#16246.
The class names need to be renamed even if we are not about to run
puppet ourselves; otherwise, deployments which rely on running puppet
themselves will still have the wrong class names.
These are respected by `urllib`, and thus also `requests`. We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
The `no_proxy` parameter does not work to remove proxying[1]; in this
case, since all requests with this adapter are to the internal Tornado
process, explicitly pass in an empty set of proxies to disable
proxying.
[1] https://github.com/psf/requests/issues/4600
Not all of the workers are known to be safe to interrupt; they might
leave inconsistent state. As such, terminating them with timeouts
should currently only be a last-resort against stalled queues, not a
regular occurrence.
Since the exception can be triggered at arbitrary places in the stack
based on whenever the alarm happens to fire, they do not often group
together.
Explicitly group them together, grouped only by which queue the work
is in.
While working on shifting toward native browser time zone APIs
(#16451), it was found that all but very recent Chrome and Node
versions reject certain legacy timezone aliases like US/Pacific
(https://crbug.com/364374).
For now, we only canonicalize the timezone property returned in user
objects and not the timezone setting itself.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.
Fixes#12490.
Using `config_file.write()` only writes out what python stored of the
file; as such, it strips all comments and whitespace.
Use `crudini --set`, which only modifies the line whose contents are
changed.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name. This is minor
cleanup, but enabled by the other renames in the previous commit.
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes. Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.
Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.
[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
This also removes direct includes of `zulip::common`, making
`zulip::base` gatekeep the inclusion of it. This helps enforce that
any top-level deploy only needs include a single class, and that any
configuration which is not meant to be deployed by itself will not
apply, due to lack of `zulip::common` include.
The following commit will better differentiate these top-level deploys
by moving them into a subdirectory.
Relying on `defined(Class['...'])` makes the class sensitive to
resource evaluation ordering, and thus brittle. It is also only
functional for a single service (thumbor).
Generalize by using `purge => true` for the directory to automatically
remove all un-managed files. This is more general than the previous
form, and may result in additional not-managed services being removed.
During the new user creation code path, there can be no existing
active clients for the user being created, so we can skip the code to
send events to that user's clients.
The tests here reflect that we need to send fewer events, and do fewer
queries that would have been spent computing data for these..
Fixes#16503, combined with the long series of recent changes by Steve
Howell to fix super-linear behavior in this code path.
We no bulk up peer_add/peer_remove events by user if the
same user has subscribed to multiple streams (and just
that single user).
This mostly optimizes the new-user codepath, but the
algorithm is a bit more general in nature.
This test was flaky due to some date-related
non-determinism. I make all the Message objects
current to make add_new_user_history reliably
try to bulk-update UserMessage rows to read.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.
Tweaked by tabbott to improve the logging output and update documentation.
Fixes#16586.
Because of the very large `oneOf` clause of the formats of events
possible in Zulip's `GET /events` system, we had issues with
`test-backend` failures for missing documentation for a new event
format being like 1000 lines of output, which was very much unhelpful.
Fix this by limiting the output use only the oneOf variants that are
broadly similar to the actual payload received.
Fixes#16023.
Restarting servers is what can cause service interruptions, and
increase risk. Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
d2aa81858c replaced the `apt::source` to set up debathena with
`Exec['setup-apt-repo-debathena']`, but mistakenly left the
`apt::source` in place in `zmirror` (but not `zmirror_personals`).
The `apt::source` resource type was later removed in c9d54f7854,
making the manifest to apply on `zmirror`.
Remove the broken and unnecessary `apt::source` resource.
See commit 8b002040e0 and #86. The
development environment bug that necessitated this handler has long
been irrelevant.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The comment still pointed to 'vacate' event flow, but
we have removed the vacate event in a9356508ca.
This commit fixes the comment to depict the correct
purpose of below lines, i.e. to test the remove
event flow.
We were including 'realm_user' in event_types along with 'subscription',
but we don't send event of type 'realm_user' when subscribing to a new
stream. This was added in 1c332f5d6a.
This commit removes 'realm_user' from event_types.
The name used to be included in the id_token, but this seems to have
been changed by Apple and now it's sent in the `user` request param.
https://github.com/python-social-auth/social-core/pull/483 is the
upstream PR for this - but upstream is currently unmaintained, so we
have to monkey patch.
We also alter the tests to reflect this situation. Tests no longer put
the name in the id_token, but rather in the `user` request param in the
browser flow, just like it happens in reality.
An adaptation has to be made in the native flow - since the name won't
be included by Apple in the id_token anymore, the app, when POSTing
to the /complete/apple/ endpoint,
can (and should for better user experience)
add the `user` param formatted as json of
{"email": "hamlet@zulip.com", "name": {"firstName": "Full", "lastName": "Name"}}
dict. This is also reflected by the change in the
native flow tests.
This property is not related to the base zulip install; move it to
zulip::postgres_common, which is already used as a namespace for
various postgres variables.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
We now can send an implied matrix of user/stream tuples
for peer_add and peer_remove events.
The client code basically does this:
for stream_id in event['stream_ids']:
for user_id in event['user_ids']:
update_sub(stream_id, user_id)
We used to send individual events, which gets real
expensive when you are creating new streams. For
the case of copy-to-stream case, we should see
events go from U to 1, where U is the number of users
added.
Note that we don't yet fully optimize the potential
of this schema. For adding a new user with lots
of default streams, we still send S peer_add events.
And if you subscribe a bunch of users to a bunch of
private streams, we only go from U * S to S; we can't
optimize it down to one event easily.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:
British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어
Fix the sort to use the locale-independent Unicode Collation
Algorithm:
British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Updated create-bot-construct-url-indented.md file with
guidelines on how to URL-encode stream name and topic
name. The hyperlink added will solve the issue for those
who use emoji in stream name or topic name.
Tweaked by tabbott to edit the copy and update the non-indented
version as well.
Fixes#16430.
In Firefox, other Keyboard shortcuts stop working once the users
presses escape to exit the search bar.
Fix this by explicitly focusing the main panel after we exit search.
Fixes#16394.
All the fields of a stream's recipient object can
be inferred from the Stream, so we just make a local
object. Django will create a Message object without
checking that the child Recipient object has been
saved. If that behavior changes in some upgrade,
we should see some pretty obvious symptom, including
query counts changing.
Tweaked by tabbott to add a longer explanatory comment, and delete a
useless old comment.
This saves us a query for edge cases like when
you try to unsubscribe from a public stream
that you have already unsubscribed from.
But this is mostly to prep for upcoming
optimizations.
This doesn't change anything yet, but the goal is
to eventually optimize events for the case where
one user (typically a new user) gets subscribed
to multiple public streams.
Initially markdown titles were overridden by Youtube and Vimeo preview titles.
But now it will check if any markdown title is present to replace Youtube or
Vimeo preview titles, if preview of linked websites is enabled.
Fixes#16100
Upstream has slightly changed the whitespace around stashes. Take
this opportunity to clean up the extra blank lines we were outputting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Add `data-simplebar` attribrute to `preview_message_area` div in
`templates/zerver/app/compose.html`.
This will cause preview_message_area div to use simplebar scrollbar
instead of normal scrollbar.
Fixes#16468.
There was no need to put "stream_id" on the sub
dictionary here. It's kinda annoying to introduce
the little helper here, but I feel
that's better than crufting up the sub data
structure.
The is_web_public flag is already in Stream.API_FIELDS,
so there is no reason for all this complicated logic.
There's no reason to hack it on to the subscription
object.
We replace all_streams_id with a map.
We also use it to populate never_subscribed_streams.
And all_streams_map is a superset of stream_hash,
which we will soon kill off as well.
Apparently I put these parens in the code as
part of 73c30774cb
during 2017.
It looks like I extracted is_public during
the middle of my change and forgot to remove
the unnecessary parens. (The code was correct,
but it makes it look like a tuple if you're
skimming it too quickly.)
That class is an artifact of when Stream
didn't have recipient_id. Now it's simpler
to deal with stream subscriptions.
We also save a query during page load (and
other places where we get subscriber
info).
As per https://stackoverflow.com/questions/1865837/ location.href
should be preferred to location.replace in some places due to the
fact that location.replace violates browser history and breaks back key.
Let the callers access stream.recipient as needed.
It costs the same, and some of the callers can
actually stop caring about the actual Recipient
object.
We already trust ids that are put on our queue
for deferred work. For example, see the code for
"mark_stream_messages_as_read_for_everyone"
We now pass stream_recipient_id when we queue
up work for do_mark_stream_messages_as_read.
This generally saves about 3 queries per
user when we unsubscribe them from a stream.
We get two speedups:
* The query to get existing subscribers only
gets the two fields we need. We no longer
need all the overhead of user_profile
and recipient data being returned in the
query.
* We avoid Django making extra hops to the
database to get user info.
Because the command is part of a pipe sequence, the exitcode defaults
to the last in the sequence, which is not the most important one here.
Set pipefail, which sets the exit status to the exit code of the last
program in the sequence to exit non-zero, or 0 if all succeeded. This
prevents the upgrade from barreling onward and setting
`postgres.version` improperly if the database upgrade step failed.
Use https://github.com/stripe/smokescreen to provide a server for an
outgoing proxy, run under supervisor. This will allow centralized
blocking of internal metadata IPs, localhost, and so forth, as well as
providing default request timeouts (10s by default).
Previously, the transaction.atomic() was not properly scoped to ensure
that RealmAuditLog entries were created in the same transaction,
making it possible for state changes to not be properly recorded in
RealmAuditLog.
When apps like mobile register for "streams", we
will now just use active streams as our baseline,
rather than "occupied" streams.
This means we will send a stream that is active,
even if it happens to have zero occupants. It's
actually pretty rare that a stream has zero occupants,
and it's not exactly clear that we want to exclude
a non-occupied but otherwise active stream from
our list of streams.
It also happens to be fairly expensive to compute
whether a stream is occupied.
This change only affects API clients (including
possibly our mobile app). The main webapp never
used the data from this codepath.
We replace get_peer_user_ids_for_stream_change
with two bulk functions to get peers and/or
subscribers.
Note that we have three codepaths that care about
peers:
subscribing existing users:
we need to tell peers about new subscribers
we need to tell subscribed user about old subscribers
unsubscribing existing users:
we only need to tell peers who unsubscribed
subscribing new user:
we only need to tell peers about the new user
(right now we generate send_event
calls to tell the new user about existing
subscribers, but this is a waste
of effort that we will fix soon)
The two bulk functions are this:
bulk_get_subscriber_peer_info
bulk_get_peers
They have some overlap in the implementation,
but there are some nuanced differences that are
described in the comments.
Looking up peers/subscribers in bulk leads to some
nice optimizations.
We will save some memchached traffic if you are
subscribing to multiple public streams.
We will save a query in the remove-subscriber
case if you are only dealing with private streams.
Installing an updated linux kernel package, as can happen during the
`apt dist-upgrade` done by the installer, can cause grub to pop up a
prompt to update its configuration file. In an unattended headless
configuration, this will stop the installation.
Explicitly configure apt to be non-interactive, and prefer the newest
configuration, during the install.
This will ensure that we always fully execute the database part of
modifying subscription objects. In particular, this should prevent
invariant failures like #16347 where Subscription objects were created
without corresponding RealmAuditLog entries.
Fixes#16347.
We don't need the select_related('user_profile')
optimization any more, because we just keep
track of user info in our own data structures.
In this codepath we are never actually modifying
users; we just occasionally need their ids or
emails.
This can be a pretty substantive improvement if
you are adding a bunch of users to a stream
who each have a bunch of their own subscriptions.
We could also limit the number of full rows in this
query by adding an extra hop to the DB just to
get colors (using values_list), and then only get
full sub info for the streams that we're adding, rather
than getting every single subscription, in full, for each user.
Apart from finding what colors the user has already
used, the only other reason we need all the columns
in Subscription here is to handle streams that
need to be reactivated. Otherwise we could do
only("id", "active", "recipient_id", "user_profile_id")
or similar. Fortunately, Subscription isn't
an overly wide table; it's mostly bool fields.
But by far the biggest thing to avoid is bringing
in all the extra user_profile data.
We have pretty good coverage on query counts here,
so I think this fix is pretty low risk.
This class removes a lot of the annoying tuples
we were passing around.
Also, by including the user everywhere, which
is easily available to us when we make instances
of SubInfo, it sets the stage to remove
select_related('user_profile').
We used to send occupy/vacate events when
either the first person entered a stream
or the last person exited.
It appears that our two main apps have never
looked at these events. Instead, it's
generally the case that clients handle
events related to stream creation/deactivation
and subscribe/unsubscribe.
Note that we removed the apply_events code
related to these events. This doesn't affect
the webapp, because the webapp doesn't care
about the "streams" field in do_events_register.
There is a theoretical situation where a
third party client could be the victim of
a race where the "streams" data includes
a stream where the last subscriber has left.
I suspect in most of those situations it
will be harmless, or possibly even helpful
to the extent that they'll learn about
streams that are in a "quasi" state where
they're activated but not occupied.
We could try to patch apply_event to
detect when subscriptions get added
or removed. Or we could just make the
"streams" piece of do_events_register
not care about occupy/vacate semantics.
I favor the latter, since it might
actually be what users what, and it will
also simplify the code and improve
performance.
The query to get "occupied" streams has been expensive
in the past. I'm not sure how much any recent attempts
to optimize that query have mitigated the issue, but
since we clearly aren't sending this data, there is no
reason to compute it.
Using web_public_guest for anonymous users is confusing since
'guest' is actually a logged-in user compared to
web_public_guest which is not logged-in and has only
read access to messages. So, we rename it to
web_public_visitor.
We should eventually add templating for the set of hosts here, but
it's worth merging this change to remove the deleted hostname and
replace it with the current one.
This is a more thorough test of adding multiple
streams for multiple users, including streams
that users have already subscribed to.
The extra queries here are due to the fact
that we call `principal_to_user_profile` in
a loop in the view. So that's an example
of O(N) overhead. We may be able to bulk-fetch
these users eventually.
This is a pure extraction, except that I remove a
redundant check that `len(principals) > 0`. Whenever
that value is false, then `new_subscriptions` will
only have one possible entry, which is the current
user, and we skip that in the loop.
We no longer do O(N) queries to get existing streams.
This is a somewhat contrived use case--generally, we
are not trying to re-subscribe a user to several
streams. Still, we want to avoid this.
This commit also makes `test_bulk_subscribe_many`
do more work, and the change to the test helped
me discover this bug.
If a user asks to be subscribed to a stream
that they are already subscribed to, then
that stream won't be in new_stream_user_ids,
and we won't need to send an event for it.
This change makes that happen more automatically.
Let
U = number of users to subscribe
S = number of streams to subscribe
We were technically doing N^3 amount of work
when we sent certain events, or to be more
precise, U * S * S amount of work. For each
stream, we were looping through a list of tuples
of size U * S to find the users for the stream.
In practice either U or S is usually 1, so the
performance gains here are probably negligible,
especially since the constant factors here
were just slinging around Python data.
But the code is actually more readable now, so
it's a double win.
We rename needs_new_sub (which sounds like
a boolean!) to new_recipient_ids, and we
calculate it explicitly within the loop, so
that we don't need to worry as much about
subsequent passes through the loop mutating it.
This allows us to also remove recipient_ids,
which in turn lets us remove recipients_map,
albeit with a small tweak for stream_map.
I also introduce the my_subs local, which
I use to more directly populate used_colors,
as well as using it as the loop var.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.
This may be overkill--I would also be happy if we
just included the assertions from this commit.
This function now does all the work that we used
to do with notify_subscriptions_added happening
inside a loop.
There's a small fine-tuning here, where we only
get recent traffic on streams that we're actually
sending events for.
We now just pass in all_subscribers_by_stream, rather
than a callback.
We also move sub_tuples_by_user closer to the
loop where we call notify_subscriptions_added.
Disabled on webservers in 047817b6b0, it has since lingered in
configuration, as well as running (to no effect) every minute on the
loadbalancer.
Remove the vestiges of its configuration.
This banner shows on lb1, advertising itself as lb0. There is no
compelling reason for a custom motd, especially one which needs to
be reconfigured for each host.
This shows the normal popover instead of extended profile.
We use the standard event handler attached to the body element in
`popovers.js` instead of attaching a new one.
Since the denmark stream stub has subscribed = true,
we include the current user id in it's subscribers set.
We remove user `me` from sweden stream stub for the same
reason.
This preserves the alpha layer on GIF images that need to be resized
before being uploaded. Two important changes occur here:
1. The new frame is a *copy* of the original image, which preserves the
GIF info.
2. The disposal method of the original GIF is preserved. This
essentially determines what state each frame of the GIF starts from
when it is drawn; see PIL's docs:
https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#saving
for more info.
This resolves some but not all of the test cases in #16370.
ssh always runs its command through a shell (after naïvely joining
multiple arguments with spaces), so it needs an extra level of shell
quoting. This should have no effect because we already validated user
with a regex, but it’s better for escaping to be locally correct in
case the context changes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
do_send_messages has side effects outside the database and may not
work reliably if its database effects are reordered by being inside a
transaction.
This also fixes a bug where we were doing the update incorrectly on
the Message table.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since this was using repead individual get() calls previously, it
could not be monitored for having a consumer. Add it in, by marking
it of queue type "consumer" (the default), and adding Nagios lines for
it.
Also adjust missedmessage_emails to be monitored; it stopped using
LoopQueueProcessingWorker in 5cec566cb9, but was never added back
into the set of monitored consumers.
This low-level interface allows consuming from a queue with timeouts.
This can be used to either consume in batches (with an upper timeout),
or one-at-a-time. This is notably more performant than calling
`.get()` repeatedly (what json_drain_queue does under the hood), which
is "*highly discouraged* as it is *very inefficient*"[1].
Before this change:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11158 / sec
Dequeue rate: 3075 / sec
```
After:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11511 / sec
Dequeue rate: 19938 / sec
```
[1] https://www.rabbitmq.com/consumers.html#fetching
`loopworker_sleep_mock` is a file-level variable used to mock out the
sleep() call in LoopQueueProcessingWorker; don't reuse the variable
name for something else.
Despite its name, the `queue_size` method does not return the number
of items in the queue; it returns the number of items that the local
consumer has delivered but unprocessed. These are often, but not
always, the same.
RabbitMQ's queues maintain the queue of unacknowledged messages; when
a consumer connects, it sends to the consumer some number of messages
to handle, known as the "prefetch." This is a performance
optimization, to ensure the consumer code does not need to wait for a
network round-trip before having new data to consume.
The default prefetch is 0, which means that RabbitMQ immediately dumps
all outstanding messages to the consumer, which slowly processes and
acknowledges them. If a second consumer were to connect to the same
queue, they would receive no messages to process, as the first
consumer has already been allocated them. If the first consumer
disconnects or crashes, all prior events sent to it are then made
available for other consumers on the queue.
The consumer does not know the total size of the queue -- merely how
many messages it has been handed.
No change is made to the prefetch here; however, future changes may
wish to limit the prefetch, either for memory-saving, or to allow
multiple consumers to work the same queue.
Rename the method to make clear that it only contains information
about the local queue in the consumer, not the full RabbitMQ queue.
Also include the waiting message count, which is used by the
`consume()` iterator for similar purpose to the pending events list.
This styles the vertical scrollbar similar to the horizontal
one for <pre/> (which can be seen in narrow windows). Strictly
speaking, this change shouldn't go in rendered_markdown.scss, but
placing it there helps unify the two scrollbar stylings rather than
duplicating them.
Since both the original button svg's have their own individual height
and width, this commit tweaks it to values height=20 and width=16 which
works well for both buttons.
Z-index is added to the base class. This doesn't affect copy_code_button
in any way.
Attributes dropped/changed:
- background-color
- Base class on-hover property is now used.
- height, width, padding is now the base classes.
We can also remove the TODO now.
The base class will contain common styling which is used by both
copy_codeblock and copy_message buttons. This sets us up nicely
for following commit(s) which aims to unify the two button styling.
We modify access_stream_for_delete_or_update function to return
Subscription object also along with stream. This change will be
helpful in avoiding an extra query to get subscription object in
code for updating subscription role.
For streams in which only full members are allowed to post,
we block guest users from posting there.
Guests users were blocked from posting to admin only streams
already. So now, guest users can only post to
STREAM_POST_POLICY_EVERYONE streams.
This is not a new feature but a bugfix which should have
happened when implementing full member stream policy / guest users.
In addition to being generally more correct, this works around a bug
in Node.js that causes webpack-dev-server to corrupt the terminal
state when exiting as a background process.
https://github.com/nodejs/node/issues/35536
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Otherwise, if consume_func raised an exception for any reason *other*
than the alarm being fired, the still-pending alarm would have fired
later at some arbitrary point in the calling code.
We need two try…finally blocks in case the signal arrives just before
signal.alarm(0).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Replaced ImageOps.fit by ImageOps.pad, in zerver/lib/upload.py, which
returns a sized and padded version of the image, expanded to fill the
requested aspect ratio and size.
Fixes part of #16370.
SIGALRM is the simplest way to set a specific maximum duration that
queue workers can take to handle a specific message. This only works
in non-threaded environments, however, as signal handlers are
per-process, not per-thread.
The MAX_CONSUME_SECONDS is set quite high, at 10s -- the longest
average worker consume time is embed_links, which hovers near 1s.
Since just knowing the recent mean does not give much information[1],
it is difficult to know how much variance is expected. As such, we
set the threshold to be such that only events which are significant
outliers will be timed out. This can be tuned downwards as more
statistics are gathered on the runtime of the workers.
The exception to this is DeferredWorker, which deals with quite-long
requests, and thus has no enforceable SLO.
[1] https://www.autodesk.com/research/publications/same-stats-different-graphs
Currently, drain_queue and json_drain_queue ack every message as it is
pulled off of the queue, until the queue is empty. This means that if
the consumer crashes between pulling a batch of messages off the
queue, and actually processing them, those messages will be
permanently lost. Sending an ACK on every message also results in a
significant amount lot of traffic to rabbitmq, with notable
performance implications.
Send a singular ACK after the processing has completed, by making
`drain_queue` into a contextmanager. Additionally, use the `multiple`
flag to ACK all of the messages at once -- or explicitly NACK the
messages if processing failed. Sending a NACK will re-queue them at
the front of the queue.
Performance of a no-op dequeue before this change:
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10847 / sec
Dequeue rate: 2479 / sec
```
Performance of a no-op dequeue after this change (a 25% increase):
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10752 / sec
Dequeue rate: 3079 / sec
```
For the lines of code that I changed here, we were
getting field reports that the below code
was getting `undefined`:
emoji.all_realm_emojis.get(r.emoji_code)
It's not really clear to me how this could happen,
but we definitely should fail softly here. We
still report it as an error, but we let the function
return and don't trigger a TypeError.
If there's a legitimate reason for realms to delete
realm emojis, we should either downgrade this to a
warning or consider a strategy of back-fixing messages
when realm emojis get deleted.
This enables core-js modules for proposals marked as finished between
core-js 3.0 and 3.6. It’s recommended upstream, although it has no
current effect for us.
The PROVISION_VERSION bump is skipped because there is no actual
dependency change.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This has the consequence of turning on loose mode for
@babel/proposal-class-properties, which generates slightly smaller
code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Part of #16094.
Strings constructed by _() were not being
translated in the /stats page.
This was because session variable was not set.
Ideally this should have been a part of b82bda9.
Part of #16094.
Moved the language selection preference logic from home.py to a new
function in i18n.py to avoid repetition in analytics views and home
views.
For users who are not authenticated, we don't need to 2fa them,
we only need it once they are trying to login.
Tweaked by tabbott to be much more readable; the new style might
require new test coverage.
We set wildcard_mention_policy in the test database so that we can
avoid future changes in mention puppeteer tests, as the default
membership of streams in the Zulip development organization is large
enough to prevent random users from using wildcard mentions.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.
We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
We rename all_everyone_warn_threshold to
wildcard_mention_large_stream_threshold as we would
be adding wildcard_mention_policy and this
constant will also be used to show error
in case when wildcard_mention_policy is set
to admins only.
This prevents the memcached connection from being shared across
multiple processes, and hopefully addresses unexpected behavior from
cached functions like get_user_profile_by_id invoked inside the worker
processes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
For dropdown elements, use bootstrap styling. The styling
was not applied by default from bootstrap since we
use a combination of dropdown + simplebar for this element.
This doesn't match the expected structure of elements by
bootstrap since simplebar inserts elements of its own.
The style is same as in bootstrap v2.3.2 for
.dropdown-menu > li > a.
This reverts commit 5275d49f05
(effectively), which created more problems than it solves. #8484 is
not a bug: a newline can be included literally with no escaping within
POSIX quotes. Meanwhile, $"" is a bashism, and not even the correct
bashism: it translates strings using the LC_MESSAGES catalog. If the
user wants to do something complicated, they can consult the
documentation for their shell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The rabbitmq cron jobs exist in order to call rabbitmqctl as root and
write the output to files that nagios can consume, since nagios is not
allowed to run rabbitmqctl.
In systems which do not have nagios configured, these every-minute
cron jobs add non-insignificant load, to no effect. Move their
installation into `zulip_ops`. In doing so, also combine the cron.d
files into a single file; this allows us to `ensure => absent` the old
filenames, removing them from existing systems. Leave the resulting
combined cron.d file in `zulip`, since it is still of general utility
and note.
We call build_message_send_dict from check_message instead of
do_send_messages.
This is a prep commit for adding a new setting for handling
wildcard mentions in large streams.
We extract the loop for building message dict in
do_send_messages in a separate function named
build_message_send_dict.
This is a prep commit for moving the code for building
of message dict in check_message.
There is a bug where we send event for even
those messages which do not have embedded links
as we are using single set 'links_for_embed' to
check whether we have to send event for
embedded links or not.
This commit fixes the bug by adding 'links_for_embed'
in message dict itself and send the event only
if that message has embedded links.
There is a bug in invite flow, where the button text resets
to incorrect text after the invitation process is completed.
This bug is because we are using $().button("reset") to
reset the button text, but the data-reset-text attribute of
button is not changed when toggling between multi-use link
invite and email invites.
This bug can be fixed by setting data-reset-text attribute
in a way similar to the way we set data-loading-text attribute
on toggling between multiuse and normal invites.
But according to the bootstrap docs,"reset" method was removed
in v4.0 (https://getbootstrap.com/docs/4.0/migration/#buttons).
Though we are not using Bootstrap v4.0 as of now, but it is
good to remove such methods as it will help in future when we
would upgrade to later bootstrap versions.
So instead of fixing this by above mentioned patch, we are
fixing this by removing "reset" method and instead using simple
jquery to reset the text and enable the button after the invite
process is completed.
As explained in the previous commit, yamole preprocessed allOf with an
algorithm that is not standards compliant. We replicate that
algorithm, but importantly, we only use it for our own code and not
for building the openapi_core RequestValidator.
This improves the time taken by OpenAPISpec().check_reload() from
1.69s to 0.53s, nearly all of which is inside
openapi_core.create_spec.
Closes#10484. Significantly improves #16068.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
yamole preprocesses our schema by naïvely merging all the objects in
an allOf array together, but this fails to capture the meaning of
allOf according to the OpenAPI specification. allOf is supposed to be
a strict logical intersection of each subschema interpreted
independently. It does not combine their properties maps before
interpreting additionalProperties. So according to the old definition
of JsonSuccess, every response is invalid:
allOf:
- additionalProperties: false
properties:
result:
type: string
- required:
- result
- msg
properties:
msg:
type: string
because the first subschema disallowed msg and the second subschema
required msg.
To fix this, whenever we use allOf for schema “inheritence”, the base
schema must not specify additionalProperties, and the child schema
must explicitly list all properties recursively inherited from the
base schema in any subschema that uses additionalProperties.
Fixes#16109.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit introduces the UI model for the 'view in playground' feature.
The option is a 1-click UX if only one playground link has been configured
for the programming language in the code block. If multiple such playgounds
have been configured, we display a popover with the different playground
options.
The actual code extraction logic occurs here and we set the target href
combining the url_prefix and the extracted code for both these scenarios.
Fixes: #11618
In case of previews, we tweak the positioning a bit more
to the right.
The previous styling also had the focus-within action
which isn't needed here as hovering over the codeblock
is enough to display both the icons.
The Pygments language used is extracted from the data-attribute attached
to the outer `div` element. This option is displayed if the playground
mapping for that language can be found.
The UI model which does the actual code extraction and displaying the
popover is done in a future commit.
This is being hardcoded just for the prototype, post which we should add
support for realm admins to configure their own choices. The structure
here is similar to what we eventually want in the configuration API.
In c563cdba61 we imported the generated
pygments data from outside `/shared` folder. This had a couple of
problems:
* Using `require` was the wrong way to do the import in ES6 modules.
* Since we get the data from outside `/shared`, clients like
zulip-mobile would not receive it - this case had to be handeled.
Here, we fix the above problems by receiving the data when initializing
through fenced_code.initialize, and when the pygments data structure is
empty (for zulip-mobile) we fallback to the old header structure without
the data-code-language tag.
Also, this commit does a small refactor to improve the way we fetch
canonicalized_alias from pygments_data.
Tests amended.
In this commit I attempt to make the docs for
writing `test_events` a little more oriented
toward telling the developer that they need to
write a test in `test_events`; in other words,
the tone is a bit more of how-to-do-it vs.
explaining the innards.
So I now start with a concrete example.
Then much of the copy related to `verify_action`
is the same as before this commit, but I rework
it a bit to be a bit more step-by-step, and I refer
back to the concrete example.
Finally, I explain the schema checking step, which
is all new copy, although for that, I mostly
instruct the developer to read event_schema.py
itself.
This commit removes the unnecessary comment which was added in
9454683108, when we were using message.get() for keys which
were also passed as args in do_send_messages, but there are no
such keys in the current code.
This commit removes the unnecessary line of code to get
rendered_content from message dict sent by check_message
when it actually does not inlcude 'rendered_content' key.
This line was added in 9454683108, but now we do not send
rendered_content in the message dict as we render the message
in do_send_messages itself.
This adds back Chrome 72, Chrome 70, Firefox 68, Opera 68, Safari
12.1, and Samsung 9.2. Firefox 68 is still Debian stable’s default
browser. This probably has no practical effects with IE 11 still
listed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Chrome breakpoints and stack traces work fine with the eval-* options;
all that breaks is that stacktrace-gps can’t extract the source
snippets from the source map with eval-*.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
As of commit 1cdab5ae61 we use the
octopus image through Webpack, so the prefetch needs to be from the
same Webpack URL for it to do any good. We don’t want to waste more
bandwidth on this [AWESOME CRITICAL FEATURE] than we have to.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The failures saying incorrect password were caused due to
change in focus. Some actual code of ours calls focus on
the modal when opened but puppeteer was starting to type
before this occurs due to which the test was only able
to enter a part of string.
This was happening with change full name too but less
frequently as it's a relatively shorter string.
As a fix, we wait till the focus is on the modal and then
start typing.
The Formatting button that opens our Markdown help popover previously
had an "A" as its icon (the Font Awesome icon for font). This commit
changes the link to spell out "Help" to make it more discoverable.
Now that they are tab accessible, we should order them by importance.
Previously the order was:
1. Add emoji
2. Formatting
3. Attach files
4. Add video call
5. Preview
6. Drafts
This commit changes the order to:
1. Attach files
2. Preview
3. Add video call
4. Add emoji
5. Drafts
6. Formatting
The "Add emoji" button is moved back because emojis can be more
conveniently entered using the typeahead triggered with ":" or the
emoticon conversions.
A later commit alters `authenticate` of EmailAuthBackend to
add a store `needs_to_change_password` variable to session
which is useful to insist users on changing their weak password.
The tests start failing with that change because client.login()
runs `authenticate` without a `request` object. So, this commit
sends a request object with `request.session=self.client.session`
to self.client.login() in tests wherever needed.
Part of #16094.
Strings tagged with i18n were not being translated on the stats page.
This was because the translation data wasn't being sent to the front
end for this page. That logic will be required in any page with a
bundle containing i18n JavaScript.
We previously used to to redirect to config error page with
a different URL. This commit renders config error in the same
URL where configuration error is encountered. This way when
conifguration error is fixed the user can refresh to continue
normally or go back to login page from the link provided to
choose any other backend auth.
Also moved those URLs to dev_urls.py so that they can be easily
accessed to work on styling etc.
In tests, removed some of the asserts checking status code to be 200
as the function `assert_in_success_response` does that check.
Instead of prohibiting ‘return undefined’ (#8669), we require that a
function must return an explicit value always or never. This prevents
you from forgetting to return a value in some cases. It will also be
important for TypeScript, which distinguishes between undefined and
void.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
After migration to an ES6 module, `password_change_in_progress` would
no longer be mutable from outside the module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It would conflict with the stream_id variable after migration to an
ES6 module, and adds no real convenience over stream_sub().
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The configuration change made in 1c17583ad5 only allowed delivery to
those specific Zulip addresses. However, they also prevent the
mailserver from being used as an outgoing email relay from Zulip,
since all mail that passed through the mailserver (from any
originator) was required to have a `RCPT TO` that matched those
regexes.
Allow mail originating from `mynetworks` to have an arbitrary
addresses in `RCPT TO`.
We now no longer define any schemas in test_events--all
of them are in event_schema, which helps our tooling
cross-check schemas for openapi and node tests.
It happens that whether you add a reaction or remove
a reaction, we send the exact same fields, just using
a different op code.
This sort of symmetry is actually kind of rare, as
usually "add" events have more fields, and "remove" events
might just send an id of something to remove.
Our openapi schema treats these as two seperate events,
so we are more consistent with it, and it helps our
schema-checking tooling for node fixtures, too.
Note that we now have to exempt the two events from
our openapi checks, due to the is_mirror_dummy field
in the deprecated user block. We can decide how to
handle this later--one possibility is to just add it
as an optional field on the event_schema side.
Note that we use value_type for value instead of
bool, since properties can be non-bool things
like color, which we just don't test now. We
should test them.
We more than compensate for this by checking
the actual value of the value in
check_subscription_update.
There is a legacy format where we send
singular "message_id" instead of plural
"message_ids".
Then there are different fields for "private"
and "stream" message types.
Note that we make the schema for profile_data
slightly more realistic, but it doesn't actually get
exercised by our current tests (apart from
making sure it's a dict), since we don't have
profile data for our test realm.
We also don't have the optional fields for bots,
since our tests don't exercise that, nor
delivery_email.
So we exempt realm_user_add_event from openapi
checks for now.
When we try to match the openapi specs better, we
will probably want to add a few tests to test_events.
Obviously getting good coverage for adding users
would be nice for all these scenarios:
* delivery_email matters
* bots
* realm has profile fields
This is a prep commit for supporting "presence"
events, where the key of the dictionary is some
arbitrary string like "website" but the value
of the dictionary is another dictionary itself
with keys that are more like variable names.
This also forces us to create TupleType.
We exempt this from the openapi check,
since we haven't figured out how to model
tuples in openapi with the same precision
as event_schema (and it may be impossible).
Long term we just want to stop dealing in
tuples, of course.
StringDict is a data type for representing dictionaries where
all keys and values are strings. Add this data type to data_types.py
and edit other files so that this data type is put to use and tested.
(slightly tweaked by @showell to remove a comment and shorten
a var name now that we have a proper data type)
We also make our schema in event_schema reflect this,
which in turn makes us match the already accurate
openapi spec, so we no longer need to exempt four
types of events from our sanity checks.
We might want to rename the tool to something more
general now, since we are really reconciling three
things:
- node fixtures
- event_schema checkers for test_events
- openapi specs
The way we compare python and openapi schemas is
as follows:
- first convert openapi schemas to be build
from DictType, ListType, etc. with from_opeapi
- do a diff on the schemas
Most of the new code is just having the FooType
family of classes serialize themselves with schema().
Defining types with an object hierarchy
of type classes will allow us to build
functionality that was impossible (or
really janky) with the validators.py
approach of composing functions.
Most of the changes to event_schema.py
were automated search/replaces.
This patch doesn't really yet take
advantage of the new FooType classes,
but we will use it soon to audit our
openapi specs.
Use the validation of the tornado sharding config that
`stage_updated_sharding` does, by depending on it. This ensures that
we don't write out a supervisor or nginx config based on a
bad (e.g. non-sequential) list of tornado ports.
Fingerprinting the config is somewhat brittle -- it requires either
custom bootstrapping for old (fingerprint-less) configs, and may have
false-positives.
Since generating the config is lightweight, do so into the .tmp files,
and compare the output to the originals to determine if there are
changes to apply.
In order to both surface errors, as well as notify the user in case a
restart is necessary, we must run it twice. The `onlyif`
functionality cannot show configuration errors to the user, only
determine if the command runs or not. We thus run the command once,
judging errors as "interesting" enough to run the actual command,
whose failure will be verbose in Puppet and halt any steps that depend
on it.
Removing the `onlyif` would result in `stage_updated_sharding` showing
up in the output of every Puppet run, which obscures the important
messages it displays when an update to sharding is necessary.
Removing the `command` (e.g. making it an `echo`) would result in
removing the ability to report configuration errors. We thus have no
choice but to run it twice; this is thankfully low-overhead.
Even before GDPR changes, it was strange that we displayed
users differently for fork events vs. all other events.
After GDPR, we don't even get the `username` field any
more.
So now we simply use `display_name` if available, and then
we try `nickname`.
See https://developer.atlassian.com/cloud/bitbucket/bitbucket-api-changes-gdpr/
for more context.
We were trying to share the same format string between
the two different versions of bitbucket, but this only
creates confusion, as the two versions are only close
enough to be confusing.
The format string might be the same, but the semantics
are different, as well as the eventual outputs.
For example, the {username} piece here is simple in version
2, but in version 3 we append a url to the user's name.
Prefer using `assert_called_once` to protect against places where a
mock might be reused, and in so doing have been previously called,
thus making the second usage of `assert_called` not assert anything of
note.
Problems with the card itself should not be logged as errors -- while
perhaps notable in aggregate, they are not worthy of being logged to
Sentry, for instance.
Downgrade these to `info`; continue to log other problems at the
`error` level. This updates tests for this change, and in so doing
corrects a test that does not do its job, due to a missing
`reset_mock`.
This commit renames 'test_message_to_self' and
'test_api_message_to_self' tests to
'test_message_to_stream_by_name' and
'test_api_message_to_stream_by_name' to depict
the actual purpose of these tests.
user_profile will be None for web_public_guests here. Hence, for
settings (of which most be inaccessible by web public guest),
which require a user_profile, we either set an empty value for
them or set them to a default value. This will help render
the frontend or extend support to our clients without breaking
a lot of code.
Tweaked by tabbott to add many comments.
This commit adds a dev dependency on the pyre-check package, to
enable the running of Pysa (a python static analyzer for security) in
integration tests.
We now display the name of referrer instead of email in invites list
and clicking on the name opens the user popover.
This helps us to avoid showing fake emails when the email address
visibility is hidden.
Tweaked by tabbott to still look at both email and name for filtering.
We remove handle_bot_owner_profile function and we handle the opening
of popover of bot owner from a single click handler in popovers.js
using 'view_user_profile' class.
We also rename 'view_user_profile' class to 'view_full_user_profile'
for the button in popover, which is used to open full user profile.
This commit enables keyboard support for user info popovers for
navigating through popover options using up/down keys.
We add get_user_info_popover_items function, whose implementation
is different from other similar functions. Instead of using
popover_data.$tip we directly use $("div.user-info-popover")
because when we open the popover of bot owner from the bot
popover, the element which opens the popover is removed from
DOM and popover_data is undefined.
We should show normal popover instead of extended profile one for the bot
owner in bots section of organization settings.
A new function show_user_info_popover is added, as it makes sense to keep
it separated from the function used to open popover for sender of a
message, which uses the message from which the popover is opened.
This added function can further be used for showing popover for
"invited_by" in invites table.
This commit replaces the "Reply mentioning user" option with "Copy mention
syntax" for user info popovers that are not opened from a message.
Clicking on "Copy mention syntax" will copy the mention syntax of user to
clipboard.
This change is done because user popovers not opened from message are not
linked to any message.
We check for open popovers before overlays on pressing escape key
because we will be adding popovers in overlays for bot owners in
further commits also and we would want to close the popover only
on pressing escape key and not the overlay.
We rename data-bot-owner-id and data-owner-id, used to open user
profile of bot owners, to data-user-id such that we can make a
global click handler for all of them by making a separate class
in next commit.
We rename user_info_popover_handle_keyboard and get_user_info_popover_items
to user_info_popover_for_message_handle_keyboard and
get_user_info_popover_for_message items to differentiate it from functions
that will be added for bot-owner popovers.
These represent known errors in what the user submitted. This is
slightly complicated by UnsupportedWebhookEventType being an instance
of JsonableError.
allow_webhook_access may be true if the request allows webhook
requests, regardless of if it only used for a webhook integration.
Only actually log to the verbose webhook logger if it is explicitly a
webhook endpoint, as judged by `webhook_client_name`. This prevents
requests for `POST /api/v1/messages` from being logged to the webhook
logger if they mistakenly contain a `payload` argument.
This argument does not define if an endpoint "is a webhook"; it is set
for "/api/v1/messages", which is not really a webhook, but allows
access from webhooks.
If multiple filters match the same string, we run into an infinite
loop of converting string into urls. To fix it, we mark the matched
string as atomic after first conversion.
We raise MissingAuthenticationError now, which adds
`www_authenticate=session` header to the error response. This
stops modern web-browsers from displaying a login form everytime
a 401 response it sent to the client.
Previously, compose_ui.autosize_textarea didn't work while editing
messages in many cases (uploading files, typeaheads, keydown handling,
etc.).
Refactored the autosize_textarea function in compose_ui to work
while editing messages too and added appropriate argument for the
introduced function parameter at all occurences of the function
use.
Also, updated the corresponding test cases.
On uploading a few files from markdown_preview mode of compose box and
then switching back to edit mode, the compose box doesn't get resized.
It even doesn't allow to scroll through the content.
Fixed this by switching back to the edit mode everytime user uploads
some file in markdown_preview mode as there's no use of staying in
markdown_preview mode anyways after uploading a file as the preview
doesn't get updated.
Also, updated the corresponding test cases.
Fixes: #16296.
Having both of these is confusing; TORNADO_SERVER is used only when
there is one TORNADO_PORT. Its primary use is actually to be _unset_,
and signal that in-process handling is to be done.
Rename to USING_TORNADO, to parallel the existing USING_RABBITMQ, and
switch the places that used it for its contents to using
TORNADO_PORTS.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
This system can't update stats while the queue is idle, without using
threads for this, but at least we ensure to update the file after
consuming an event if more than MAX_SECONDS_BEFORE_UPDATE_STATS passed
since the last update, regardless of the number of iterations done so
far.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
This mimics the backend logic for adding the data-attribute -
to know what Pygments language was used to highlight the code
block - in locally echoed messages.
New test added checks our logic for canonicalizing pygments alias
(for both frontend and backend).
Other fixtures and tests amended.
We need this information in the frontend to:
* Display the 'view in playground' option for locally echoed messages.
* When we add a UI settings for realm admins to configure their
playground choices, we'll need to use these canonicalized aliases
for displaying the option.
Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.
Bumping major PROVISION_VERSION since folks need to provision in both
directions.
Tests amended.
In ae58ed5a7 we decided to echo back the text, when no Pygments lexer
matching that language was found. When we do so, we must take care to
HTML escape the lang before wrapping it in a data-code-language attribute.
Tweaked by tabbott to make clear the escaping is defensive.
This changes the success text of the `subscriber_list_add`
form to display the subscribed and already subscribed users
on success. We also display the user profile as a popover.
Previously we would only display the email ids of the already
subscribed users.
Formatting tweaked by tabbott.
We can compute the intended number of processes from the sharding
configuration. In doing so, also validate that all of the ports are
contiguous.
This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding. Having behaviour which
changes merely based on if an empty section exists is surprising.
This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.
After this commit, configuring sharding can be done by adding to
`zulip.conf`:
```
[tornado_sharding]
9800 = # default
9801 = other_realm
```
Followed by running `./scripts/refresh-sharding-and-restart`.
In development and test, we keep the Tornado port at 9993 and 9983,
respectively; this allows tests to run while a dev instance is
running.
In production, moving to port 9800 consistently removes an odd edge
case, when just one worker is on an entirely different port than if
two workers are used.
tornado.web.Application does not share any inheritance with Django at
all; it has a similar router interface, but tornado.web.Application is
not an instance of Django anything.
Refold the long lines that follow it.
bootstrap sets <code> to use `Monaco` font by default. We don't
want to use this font since some characters are not clearly
readable like `()` appearing as `0`.
Hence, we use Menlo font by default if available.
Since `Monaco` font is only installed in macOS by default, this
mostly affected mac users.
While urllib3 retries all connection errors, it only retries a subset
of read errors, since not all requests are safe to retry if they are
not idempotent, and the far side may have already processed them once.
By default, the only methods that are urllib3 retries read errors on
are GET, TRACE, DELETE, OPTIONS, HEAD, and PUT. However, all of the
requests into Tornado from Django are POST requests, which limits the
effectiveness of bb754e0902.
POST requests to `/api/v1/events/internal` are safe to retry; at worst,
they will result in another event queue, which is low cost and will be
GC'd in short order.
POST requests to `/notify_tornado` are _not_ safe to retry, but this
codepath is only used if USING_RABBITMQ is False, which only occurs
during testing.
Enable retries for read errors during all POSTs to Tornado, to better
handle Tornado restarts without 500's.
This was called in both if and else with the same argument.
I believe there's no reason for it to exist twice and having
it just once would be a bit cleaner.
webfonts-loader now defaults writeFiles to true, which makes spurious
copies of zulip-icons.{css,eot,svg,ttf,woff,woff2} for no reason.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When a fragment (i.e. section starting with `#`) is present in the URL
when landing the development login page, dev-login.js used to append
it to the appends it to the `formaction` attribute of all the input
tag present in the `dev_login.html`.
This made sense before 139cb8026f, which
adjusted the structure of how `next` was passed.
To fix this, we adjust the JavaScript login to set the `next` hidden
input instead.
fixes#16215
This makes our icon font smaller. We only reference its glyphs by
Unicode private-use codepoints and not by ligatures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
css-loader@4 broke @import statements referencing files with
extensions other than .css, unless those @import statements are
compiled away by another loader. Upstream is more interested in
arguing that such @import statements are semantically incorrect than
applying the one line fix.
https://github.com/webpack-contrib/css-loader/issues/1164
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Without an explicit port number, the `stdout_logfile` values for each
port are identical. Supervisor apparently decides that it will
de-conflict this by appending an arbitrary number to the end:
```
/var/log/zulip/tornado.log
/var/log/zulip/tornado.log.1
/var/log/zulip/tornado.log.10
/var/log/zulip/tornado.log.2
/var/log/zulip/tornado.log.3
/var/log/zulip/tornado.log.7
/var/log/zulip/tornado.log.8
/var/log/zulip/tornado.log.9
```
This is quite confusing, since most other files in `/var/log/zulip/`
use `.1` to mean logrotate was used. Also note that these are not all
sequential -- 4, 5, and 6 are mysteriously missing, though they were
used in previous restarts. This can make it extremely hard to debug
logs from a particular Tornado shard.
Give the logfiles a consistent name, and set them up to logrotate.
Making this include "zulip-tornado" makes it clearer in supervisor
logs. Without this, one only sees:
```
2020-09-14 03:43:13,788 INFO waiting for port-9807 to stop
2020-09-14 03:43:14,466 INFO stopped: port-9807 (exit status 1)
2020-09-14 03:43:14,469 INFO spawned: 'port-9807' with pid 24289
2020-09-14 03:43:15,470 INFO success: port-9807 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
```
Calling `render()` in a middleware before LocaleMiddleware has run
will pick up the most-recently-set locale. This may be from the
_previous_ request, since the current language is thread-local. This
results in the "Organization does not exist" page occasionally being
in not-English, depending on the preferences of the request which that
thread just finished serving.
Move HostDomainMiddleware below LocaleMiddleware; none of the earlier
middlewares call `render()`, so are safe. This will also allow the
"Organization does not exist" page to be localized based on the user's
browser preferences.
Unfortunately, it also means that the default LocaleMiddleware catches
the 404 from the HostDomainMiddlware and helpfully tries to check if
the failure is because the URL lacks a language component (e.g.
`/en/`) by turning it into a 304 to that new URL. We must subclass
the default LocaleMiddleware to remove this unwanted functionality.
Doing so exposes a two places in tests that relied (directly or
indirectly) upon the redirection: '/confirmation_key'
was redirected to '/en/confirmation_key', since the non-i18n version
did not exist; and requests to `/stats/realm/not_existing_realm/`
incorrectly were expecting a 302, not a 404.
This regression likely came in during f00ff1ef62, since prior to
that, the HostDomainMiddleware ran _after_ the rest of the request had
completed.
This commit moves docs for users/{user_id}/subscriptions/{stream_id}
enndpoint to be after users/me/subscriptions/muted_topics docs.
We are rearranging the docs because after adding the new patch
endpoint for users/{user_id}/subscriptions/{stream_id}, openapi_core
validator tries to match 'users/me/subscriptions/muted_topics'
with 'users/{user_id}/subscriptions/{stream_id}' path in zulip.yaml
and thus gives error while running tests.
This is a bug in 'openapi_core' as it does not follows OpenAPI specs
to match concrete paths before their templated counterparts. Thus,
this commit rearranges the docs such that openapi_core validator
tries to match muted_topics endpoint with the correct path in
zulip.yaml docs.
When converting fenced code markdown, we add the language (if specified)
in a data-attribute by tweaking the HTML generated. Doing so, allows the
frontend to make use of this attr to display view-in-playground option
for codeblocks.
We use pygments to get the lexer subclass name and use that instead of
directly using the language in the data-attribute. Doing so, helps us
map different language aliases (like `js` and `javascript`) into a common
variable (like `JavaScript`) - and avoids the client from dealing with
multiple tags corresponding to the same language.
The html structure for a message like this:
``` js
..content..
```
would now be:
<div class="codehilite" data-codehilite-language="JavaScript">
<pre>..content..</pre>
</div>
Tests and fixtures amended.
Its functionality was added to Django upstream in 2.1. Also remove
the SESSION_COOKIE_SAMESITE = 'Lax' setting since it’s the default.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`supervisorctl` starts and stops its arguments sequentially, in the
order they are passed[1]. Start them in the opposite order from the
order in which they were stopped -- this puts the dependencies first,
and the most core services (`zulip-django`) last.
While the only "dependency" here is currently thumbor, this sets us up
in case others are added later.
[1] https://github.com/Supervisor/supervisor/blob/master/supervisor/supervisorctl.py#L782
This supports running puppet to pick up new sharding changes, which
will warn of the need to finalize them via
`refresh-sharding-and-restart`, or simply running that directly.
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Fixes#16284.
Most of the work for this was done when we implemented correct
behavior for guest users, since they treat public streams like private
streams anyway.
The general method involves moving the messages to the new stream with
special care of UserMessage.
We delete UserMessages for subs who are losing access to the message.
For private streams with protected history, we also create UserMessage
elements for users who are not present in the old stream, since that's
important for those users to access the moved messages.
This undoes a small part of b8a2e6b5f8; namely, logs to
`zulip.zerver.webhooks`, which are all exceptions from webhooks except
UnsupportedWebhookEventType, should still be logged to the main error
loggers. This maintains the property that exceptions generating 500's
are all present in `errors.log`.
Previously, S3UploadBackend.delete_export_tarball failed to strip the
leading ‘/’ from the export path. This mistake is now caught by Moto
1.3.15. I expect it caused deletion failures in the real S3, although
I haven’t verified this.
We store export_path in the audit log with a leading ‘/’, but the
actual S3 keys do not have a leading ‘/’. Changing either system
would require a migration. So the new convention is that the
variables named ‘export_path’ have a leading ‘/’, while variables
named ‘path_id’ or ‘key’ do not.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, the GitLab webhook code, namely the `get_objects_assignee`
method first tried to get a single assignee and if that failed then it
looks for multiple assignees and then it would return the first
assignee that it found (there's actually a code smell here - a loop
which would always return on the first iteration).
Instead, this commit will change that behavior to first check for
multiple assignees first then for a single assignee if we can't find
multiple assignees. Ultimately it will return a list of all of the
assignees (however many that might be [0, n]). This method has then
aptly been renamed to `get_assignees`.
Finally, we tweked the code using this method to always use it's
output as an "assignees" parameter to templates (there's also an
assignee parameter which we want to avoid here for consistency).
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
For some reasons, some of the fixtures had the +x bit set, while
some didn't. What this commit does is make sure that no fixture
is marked as "executable" (for anyone).
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
The previous code only worked by accident and hyperlink 20.0.0 breaks
it.
>>> hyperlink.parse("example.com").replace(scheme="https")
DecodedURL(url=URL.from_text('https:example.com'))
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Django treats path("<name>") like re_path(r"(?P<name>[^/]+)") and
path("<path:name>") like re_path(r"(?P<name>.+)").
This is more readable and consistent than the mix of slightly
different regexes we had before, and fixes various bugs:
• The r'apps/(.*)$' regex was missing a start anchor ^, so it
incorrectly matched all URLs that included apps/ as a substring
anywhere.
• The r'accounts/login/(google)/$' regex was missing a start anchor ^,
so it incorrectly matched all URLs that ended with
accounts/login/google/.
• The type annotation of zerver.views.realm_export.delete_realm_export
takes export_id as an int, but it was previously passed as a string.
• The type annotation of zerver.views.users.avatar takes medium as a
bool, but it was previously passed as a string.
• The [0-9A-Za-z]+ pattern for uidb64 was missing the - and _
characters that can validly be part of a base64url encoded
string (although I think the id is actually a decimal integer here,
in which case only 012345ADEIMNOQTUYcgjkwxyz are present in its
base64url encoding).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The clipboard tooltip for the "copy and close" button was incorrectly staying visible after the
copy event. Fix this by explicitly hiding tooltips in the click handler.
Fixes#16328.
The specification says “any sibling elements of a $ref are ignored”,
so their presence, although not invalid, indicates a mistake. yamole
incorrectly merges these siblings into the referenced object, but we
should not rely on this nonstandard behavior.
https://swagger.io/docs/specification/using-ref/#sibling
Signed-off-by: Anders Kaseorg <anders@zulip.com>
$ref siblings are ignored according to the OpenAPI specification, and
the referenced definitions already have examples.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Replace default root logger with zulip.auth.apple for apple auth
in file zproject/backends.py and update the test cases
accordingly in file zerver/tests/test_auth_backends.py
Replaced mock.patch with assertLogs for testing log outputs
in file test_auth_backends.py.
This change requires adjusting
test_log_into_subdomain_when_email_is_none to use an explicit token
since that appears in the log output.
This clears it out of the data sent to Sentry, where it is duplicative
with the indexed metadata -- and potentially exposes PHI if Sentry's
"make this issue public" feature is used.
The value in the stats file can get outdated if the queue hasn't done
enough iterations to update the stats file for a while. The queue size
output by rabbitmqctl list_queues is more up to date, and empirically
tends to agree with the value in the stats file (when the stats file is
fresh).
The previous link was to "extended callable" types, which are
deprecated in favor of callback protocols. Unfortunately, defining a
protocol class can't express the typing -- we need some sort of
variadic generics[1]. Specifically, we wish to support hitting the
endpoint with additional parameters; thus, this protocol is
insufficient:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str) -> HttpResponse: ...
```
...since it prohibits additional parameters. And allowing extra
arguments:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str,
*args: object, **kwargs: object) -> HttpResponse: ...
```
...is similarly problematic, since the view handlers do not support
_arbitrary_ keyword arguments.
[1] https://github.com/python/typing/issues/193
`zulip.zerver.lib.webhooks.common` was very opaque previously,
especially since none of the logging was actually done from that
module.
Adjust to a more explicit logger name.
Any exception is an "unexpected event", which means talking about
having an "unexpected event logger" or "unexpected event exception" is
confusing. As the error message in `exceptions.py` already explains,
this is about an _unsupported_ event type.
This also switches the path that these exceptions are written to,
accordingly.
8e10ab282a moved UnexpectedWebhookEventType into
`zerver.lib.exceptions`, but left the import into
`zserver.lib.webhooks.common` so that webhooks could continue to
import the exception from there.
This clutters things and adds complexity; there is no compelling
reason that the exception's source of truth should not move alongside
all other exceptions.
The main race conditions, which actually happened in production was with
concurrent execution of deliver_email and clear_scheduled_emails.
clear_scheduled_emails could delete all email.users in the middle of
deliver_email execution, causing it to pass empty to_user_ids list to
send_email. We mitigate this by getting the list of user ids in a single
query and moving forward with that snapshot, not having to worry about
database data being mutated anymore.
clear_scheduled_emails had potential race conditions with concurrent
execution of itself due to not locking the appropriate rows upon
selecting them for the purpose of potentially deleting them. FOR UPDATE
locks need to be acquired to prevent simultaneous mutation.
Tested manually with some print+sleep debugging to make some races
happen.
fixes #zulip-2k (sentry)
There are three functional side effects:
• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.
• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.
• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.
(The base32 alphabet is A-Z 2-7. We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Improved responsiveness of message reactions tooltip .
Added css property for the reaction tooltip in ./static/js/click_handlers.js so that the tooltip doesn't exceed the sidebar .
Adressed a comment in #15364 .
Wait for disable_stream_notifications selector to be visible
before clicking as it could cause flakes if the test tries
to click without it being visible.
Added a stronger validation of waiting for text "Verona" to
appear but that didn't really seem to have worked though it
seemed like fixing the flake by passing ~600 runs.
So, change the puppeteer click to a click through evaluate
as we had experiences where page.click() didn't work sometimes.
Though this has passed 1000 runs on CI, I'm not very certain
if this fixed it as this test passed 1000 times with my previous
PR fixing the same flake.
Compose box placeholder text for streams currently updates when focus
is shifted to the text area.
With this change, it will also get updated when the stream name is
changed (it already updates if topic names are changed).
Currently, compose box placeholder text for PMs only gets updated
when the focus shifts to it.
With this change, the text is now also updated if recipients are
added or removed.
Fixes#15897.
Previously, onPillCreate function was called after the individual
pill object was created.
Now, we call it after creating and adding it to the pill container.
For web-public streams, clients can access full topic history
without being authenticated. They only need to additionally
send "streams:web-public" narrow with their request like all
the other web-public queries.
This verifies that we actually do enqueue a record when there is an
error on non-staging. With the previous commit, it verifies that that
data serializes correctly.
By default, the Django Sentry integration provides the email address
and username of pulled from the auth layer. This is potentially PII,
and not data that we wish to store. Enable sending user data at all,
by setting `send_default_pii=True`, but strip the username and
email (which are the same, in Zulip) before sending. Users will be
identified in Sentry only by their IP address, user ID, realm, and
role.
The return type of `ugettext_lazy('...')` (aliased as `_`) is a
promise, which is only forced into a string when it is dealt with in
string context. This `django.utils.functional.lazy.__proxy__` object
is not entirely transparent, however -- it cannot be serialized by
`orjson`, and `isinstance(x, str) == False`, which can lead to
surprising action-at-a-distance.
In the two places which will serialize the role value (either into
Zulip's own error reporting queue, or Sentry's), force the return
value. Failure to do this results in errors being dropped
mostly-silently, as they cannot be serialized and enqueued by the
error reporter logger, which has no recourse but to just log a
warning; see previous commit.
When we do this forcing, explicitly override the language to be the
realm default. Failure to provide this override would translate the
role into the role in the language of the _request_, yielding varying
results.
AdminNotifyHandler is used to notify admins of errors; it is a
critical piece of logic. Failures in reporting errors will compound,
since its `except Exception` clauses cannot generate logging at the
`error` or `exception` level, as that would be recursive. It must
settle for logging at the `warning` level, and hope that admins are
vigilant to the logging there.
Increase the chances of being notified of failures in this logger, by
bubbling up those exceptions to Sentry, which is an orthogonal
reporting stack.
When user requests for a realm that doesn't exists, we raise
a InvalidSubdomainError.
This reduces our effort at repeatedly ensuring realm is valid
in request in web-public queries.
If there are unsupported keys, we still log an error,
but we now also send a message to the stream. (This
is a good tradeoff for the github webhook, since users
can just turn off notifications if they find it spammy.
Also, we intend to support "repository" soon.)
This is a bit of an experiment to see how this plays
in the field:
* will customers notice the change?
* will Sentry reports look any different?
The main thing fixed here is that we weren't turning
on our keys into a list. And then I refined the message
a bit more, including sorting the keys.
I also avoid the unnecessary "else".
The EVENT_FUNCTION_MAPPER maps a string event name
to a function handler. Before this we circumvented
mypy checks with a call to get_body_function_based_on_type,
which specified Any as the type of our event function.
Now the types are rigorous.
This change was impossible without the recent commit
to introduce the Helper class.
The Helper class will soon grow, but the immediate
problem it solves is the need to jankily inspect
the parameters of our get_*_body function.
Most of the changes were handled by an ad hoc
munge.py script.
The substantive changes were adding the Helper
class and passing it in.
And then the linter discovered a place where
the optional include_title parameter wasn't used
(which is one of the reasons to avoid the janky
inspect-signature technique).
As a side note, none of the include_title parameters
needed a default value of False, as we always passed
in an explicit value.
We test cover both sides of include_title, which
you can verify by hard coding it to either True or
False (and seeing the relevant failures), although I
suspect most individual codepaths
only test one value, based on whether "topic" is in
the fixture or not.
Finally, I know Helper is not a great name, but I
intend to evolve the class a bit before deciding
whether a more descriptive name is helpful here.
(For example, an upcoming commit will add a
log_unexpected helper method.)
We get the header_event one level up the call
stack now, too.
It's somewhat annoying that we have our own
concept of "event" here, instead of just returning
our event handlers directly, or just calling them
directly, but it's a bit non-trivial to fix that
right away.
In passing, I remove the strange OR for "ping",
which is already a key in EVENT_FUNCTION_MAPPER.
See https://github.com/zulip/zulip/issues/16258 for
possible follow up here.
We now ignore the following two new pull_request
actions (as well as the three existing ones
from before):
approved
converted_to_draft
As the issue above indicates, we may want to actually
support "approved" if we can find somebody to work
on the webhook. (And then the issue goes a little
broader than what changed here.)
We consolidate the tests and remove the fixtures, which
just have a lot of noisy fields that we ignore. Also,
pull_request__request_review_removed was named improperly.
Before this the only way we took advantage
of the summary from UnexpectedWebhookEventType
was by looking at exc_info().
Now we just explicitly add it to the log
message, which also sets us up to call
log_exception_to_webhook_logger directly
with some sort of "summary" info
when we don't actually want a real
exception (for example, we might want to
report anomalous webhook data but still
continue the transaction).
A minor change in passing is that I move
the payload parameter lexically.
It was broken by commit aaedec1fdb which
moved it to tools/i18n without adjusting its relative path references,
and it contains a sketchy injectable os.system call that I’d like to
remove.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In 6653e19e3a we added
the convenient line to tell folks about the coverage
report. But if we failed coverage checks, we didn't
show the link.
Arguably we should just always show this, even if
tests fail, but that can also be potentially confusing.
The code to run single files was added
in c15695e514,
and it's just kinda strange code.
We already do a lot of file logic in Python
to check for line-coverage, so it's easier
to just have all the logic in Python.
This adds a new feature--you can now specify
the actual file:
./tools/test-js-with-node frontend_tests/node_tests/people.js
(This is helpful if you just want to use
shell autocomplete.)
Another minor change is that if you specify
individual files, we won't sort them. This is
important when you're trying to hunt down test
leaks.
Finally, we have a nicer message if we can't find
the file.
nyc was added in 29f04511c0
All the stuff after "&&" was actually passed to
node, because we didn't use shell=True, so the
"nyc report" command didn't run, and the ugly
finder.js code just skipped over all the final tokens.
Our isort configuration was almost Black-compatible, but we were
missing ensure_newline_before_comments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Because `util` is so late in the alphabet, this
leak never surfaced in practice, but I tried running
the node tests in reverse, and this leak came
up if you ran `util` before `stream_list`. I guess
it's nice that `stream_list` actually exercises
the difference between a dumb sort and an
Intl-aware sort.
It's possible that we should just assume that
Intl.Collator is always available at this point,
which would eliminate the need for this test.
This was converted automatically using a jscodeshift script followed
by running eslint and prettier to convert let -> const (whenever
applicable) and removing "use strict;".
There are lots of function that will be used before they are defined
when we migrate to ES6 from CommonJS, so disable this rule. Functions
are hoisted and processed by webpack so this is fine.
There is good reason to do this (explanation is bit long!). With the
TypeScript migration, and the require and ES6 migrations that come
with it, we use require instead of set_global which loads the entire
module. Suppose we have a util module, which is used by some other
module, say message_store, and util is being required in message_store
since it is removed from window. Then, if a test zrequires
message_store first, and then zrequires the util module qand mocks one
of its methods, it will not be mocked for the message_store
module. The reason is:
1. zrequire('message_store') leads to require('util').
2. zrequire('util') removes the util module from cache and it is
reloaded. Now the util module in message_store and the one in
the test will be different and any updates to it in tests won't
be reflected in the actual code.
Which can lead to confusion for folks writing tests. I'll mention this
can be avoided doing zrequire('util') first but...that is not ideal.
And, since there was one outlier test that relied on this behavior,
we add the namespace.reset_module function.
This version keeps the internal and external state of the ES6 module
in sync so its state is similar to a CommonJS module. Since we don't
need to ask rewire for the internal state of the module, the tests
will be same as they were before the TS/ES6/remove from window
migrations.
Some users setup zulip with trailing / at end, like 'https://meet.jit.si/
leading to extra / on clients while generating video chat link.
This commit removes trailing '/' if it exists to make it consistent. Manual
testing was done by generating jitsi url.
Fixes#16225
This uses our Webpack configuration to more accurately predict how our
imports will actually behave; for example, it automatically resolves
the .ts extension.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We lost the war against top level configuration files many moons ago.
This is what developers and tools expect. And it seems to be required
for eslint-import-resolver-webpack (there’s ostensibly a {"config":
"tools/webpack.config.ts"} option, but it doesn’t work correctly:
https://github.com/benmosher/eslint-plugin-import/issues/1861).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We eliminate optional parameters and replace `request_body`
with `payload`.
There is much less confusion if we just pass in `payload`,
and then we optionally re-format it if it's json.
For unclear reasons the original code was trying to
do `request_body = str(payload)` when `request_body`
was no longer being used.
The query to finds and marks all unread UserMessages in the stream as read
can be quite expensive, so we'll move that work to the deferred_work
queue and split it into batches.
Fixes#15770.
Fixes#16252.
icon* classes are used by bootstrap for displaying glyphicons.
We removed these classes in our custom version of bootstrap 2.1.1;
but since our reset to v2.3.2, they have been added again and hence
any classes starting with icon* in zulip will have to be renamed.
In 468c5b9a58 we changed the method of
getting the list of management commands. Using app_config.path has a
caveat in that the value depends on the path from which we're executing.
An example of things breaking can be reproduced by calling
/home/vagrant/zulip/tools/test-backend TestCommandsCanStart
This makes the app_config.path values to start with /home/vagrant/zulip,
but DEPLOY_ROOT in the dev environment is set to /srv/zulip.
/home/vagrant/zulip is a soft link to /srv/zulip, so it's a valid path
to call test-backend through, but it causes self.commands to end up
being an empty list. We fix this by converting app_config.path to the
real path.
We were not updating the trailing bookend on deactivation of stream
if the user was narrowed to deactivated stream and this commit fixes
this.
For subscribed streams, we just show the trailing bookend with
content as 'This stream has been deactivated' and hide the
Unsubscribe button.
For unsubscribed streams, we change the content of trailing bookend
to 'This stream has been deactivated' and hide the Subscribe button.
Fixes#15999.
Improves the display of error messages on registration page fixing
mis-positioning of error messages and overlapping with other text
in some cases.
Part of: #15750.
This completes the remaining work required to support
addition of all members of another stream.
This allows the creation of stream pills on pasting
the #streamname and copying it from the stream pill.
The user pills uses email ids instead.
And also allows creating stream pills when the user
hides the typeahead.
Tested by commenting out the "set_up_typeahead_on_pills"
line in `stream_edit.js`.
A `node_tests/stream_pill.js` file has been created
for the node tests and the other half of the coverage
check takes place in `node_tests/stream_edit.js`.
This fixes the regression introduced in the pervious
commit to regain the 100% line coverage in `user_pill.js`
as well as `stream_pill.js`.
The new `stream_edit.js` mainly tests for:
* The stream related queries of the typeahead in `user_pill.js`
* The "Add subscribers" event handlers.
* The event handler which displays the settings for a stream.
We update the pills typeahead logic to also include
stream results and pass the "stream" key in `opts`
to enable this option for the Add subscriber form.
This commit implements the feature of adding all the
subscribers of another stream in the "Add subscribers"
UI, with the help of a new "stream_pill.js` file.
We temporarily add `user_pill.js` to the EXEMPT_FILES
list as typeahead will be set up in `stream_edit.js`
file which does not have any dedicated tests file.
Work towards #15186.
We can safely remove the block added in commit
7d51b6a454
which checks whether the target is a JQuery object
and applies it, if it is not, because re-applying
the JQuery selector on a JQuery object returns the
same object.
We also refactor the related function calls to pass
the target instead of applying the JQuery selector
to it and finding the closest "subscription_settings"
class, as the same operation takes place in the next
line of the removed block. Thus this is redundant too.
The `show_subscription_settings` function is only called
from one place. And the first 2 lines of this function is
redundant as the `sub_settings` are passed to this function
but we obtain it's stream id and again convert it back to
`sub_settings` from it.
We display a centered spinner and hide the Submit / Cancel
buttons in the Move Topic modal similar to what is done in
the Deleting messages modal.
This commit also makes a change where we now close the modal
after success/failure response of the second request instead
of the first.
Rather than catching, checking action type, and possibly re-raising,
instead return None explicitly from `get_subject_and_body`, which
already signals for a blank success result. This collocates the logic
of the action types in one place, and removes the complexity of the
re-raise.
Sentry may get reported multiple exceptions stacks, in the case where
a `raise ...` was caught, and a new exception was `raise`d. In this
case, the `filename` is the most recent exception -- but the
exceptions are stored in the `exception` key in the order in which
they occurred. As such, taking the first value with a `stacktrace`
will result in showing the wrong line, or in no stack trace being
resolved at all.
Look from the last `exception` backwards, for matching stacks.
We don't modify bootstrap.js here but override its popover and
tooltip plugins. In future we will not import these plugins
via npm. We also copy all the popover code from bootstrap.css v2.1.1
to popovers.scss since all the code in bootstrap-tooltip.js is
based upon this css or vice versa.
Update THIRDPARTY info about bootstrap libraries.
There were 4 types of changes to bootstrap.js - bugfixes, file
moves, changes to typeahead plugin and changes to tooltip +
popover plugin.
Bugfixes were automatically fixed when upgrading to v2.3.2, file
moves are irrelevant to this upgrade and the plugins were
extracted into separate files.
46e562f - POPOVER
8779e55 - POPOVER
66c6423 - POPOVER
21ccf45 - POPOVER
cb9b526 - TYPEAHEAD EXTRACTED
3079cf8 - TYPEAHEAD
9ea4f50 - TYPEAHEAD
b961093 - TYPEAHEAD
0e2c509 - TYPEAHEAD
28589c5 - TYPEAHEAD
70a14d8 - TYPEAHEAD
0c42e4a - TYPEAHEAD
213b8ce - FIXED IN 2.3.2
0bac986 - TYPEAHEAD
0e3332d - FIXED IN V2.3.2
eaa777b - TYPEAHEAD
f944a8e - TYPEAHEAD
546ae10 - TYPEAHEAD
3bba0cc - FILE MOVED
b8794e1 - TYPEAHEAD
6217c1a - TYPEAHEAD
dc85fa7 - TYPEAHEAD
d329317 - TYPEAHEAD
b3ef776 - TYPEAHEAD
fcb3999 - TYPEAHEAD
0975cfa - TYPEAHEAD
fbed3e2 - TYPEAHEAD
0fa857d - POPOVER
68b890a - TYPEAHEAD
b5cadec - typeahead
441e429 - copyright
22ce2c0 Fixed In v2.3.2
d78d761- typeahead
bff933e- typeahead
ef585cf- typeahead
7e35369 - typeahead
8f1cee0 - Files moving around
1490ae1 - add file
Changes to bootstrap.css made by us after these are not relevant:
d7f9a21 - Reducing z-index of overlay doesn't make sense.
9b740df - some changes were added.
1143ed7 - changes in above commit were moved to a different file.
We don't want bootstrap-btn css from v3.1.1 overlapping with v2.3.2's
.btn css; so we remove it.
Commits that were skipped:
3142d74 - false typo fix. It add support for IE9.
ead73f3 - We retain Glyphicons since they don't make any difference.
3bba0cc - Moving code around.
This commit starts to bring back our changes to bootstrap
files. We try to move these changes to zulip.scss as much
as possible. We are starting with bootstrap.css.
Apply fix in commit 7a3a3be.
Changes before 7a3a3be that
were not to be added. These mostly involve moving files around
and hence are not relevant.
441e4295e2c49389ed4448f1cee01490ae1
We merge bootstrap-responsive.css into bootsrap.css since that is
how bootstrap distributes it from this version onwards.
bootstrap.js has a lot of changes to it which completely breaks
our typeaheads and popovers, so we will have to override these
plugins with our version of these plugins. In future versions
of bootstrap when we use npm, we can just choose not to
import them.
This commit clearly shows what changes we made to the bootstrap
library for a future reference.
bootstrap-btn.css was left out since it is from version v3.1.1. We
will integrate custom changes made by us into zulip when we upgrade
to v3.x.
We have also removed our license from these files.
Since ALL_HOTSPOTS is a global object, it is initialized
at the time the backend server is started. Hence, the
title and description is translated only once. Using
ugettext_lazy makes sure that the strings are translated
in each and every request according to the language
of the user.
Fixes#16224
The node package allow use to control xvfb apt package in puppeteer
tests. This help us create a fake display so we can run puppeteer in
headful (headless: false) mode, which is required to use the chrome
extension desktop capture API.
Now that all casper tests have been migrated to
puppeteer, there's no need for having casper
related things.
Removed the casperjs package and removed/replaced
casper in few places with puppeteer.
Only removed few of them which I'm confident
about. Also didn't make any changes in docs
as it would be easier to remove them while
adding puppeteer docs.
We were showing the incorrect error message when the user who
is trying to deactivate himself is the last owner. This commit
fixes this to show "Cannot deactivate the last organization owner"
instead of "Cannot deactivate the last organization administrator".
We had already removed the restriction for deactivating last admin
and added it for last owner, while adding the new owner role.
This commit fixes examples in "400" response for deactivating user
endpoints to have msg as "Cannot deactivate the last organization
owner" instead of "Cannot deactivate the last organization
administrator".
We had already removed the restriction on deactivating last admin
and added it for last owner, while adding owner role.
The `typing: stop` event did not have any tests in test_events
hence its documentation wasn't added. So add tests and relevant
documentation for the typing stop event. Also edit the documentation
of `typing: start` to include the fact that servers should use
their own timeout incase `stop` event event isn't received.
Fixes#16122.
596cf2580b ignored the loggers of all SuspiciousOperation subclasses,
but not SuspiciousOperation itself. Almost all locations raise one of
the more specific subclasses, with the exception of one location in
the session middleware[1].
Ignore the overall django.security.SuspiciousOperation logger as well.
[1] https://code.djangoproject.com/ticket/31962
We display the text of the consent message, and then continue with the
export, which will scroll the content off the screen. Allow the
administrator time to examine the contents of the message, and decide
whether to proceed based on that and the fraction of users that have
responded so far.
check_messages_test should have table set to zfilt
instead of zhome as we are narrowed to only starred
messages.
Found it as the test failed with the previous commit.
The next adds a few tests which heavily rely on
check_messages_sent. There were some weird errors,
this fixed those. I think the errors were due to
us navigating multiple times and this function
not waiting for the messages to become visible.
These `waitForSelector`s appear just after page loads.
Though they worked most of the times, a few clicks weren't
getting registered because of these selectors not appearing
and thus causing flakes as the modal takes time to appear.
Adding visible: true asserts that it's visible and not just present.
In few rare cases the click on display settings section wasn't working
which was causing the test to stay in the "your account" settings section.
This lead to a waitFor fail.
The screenshot in this failed CircleCI build suggests the above.
https://app.circleci.com/pipelines/github/chdinesh1089/zulip/525/workflows/cd77e269-6a3e-4283-b765-d1c4584ccf35/jobs/1807/artifacts
This and the previous commit along with the changes to prevent
logging out of user on changing password were tested by running
this test 1200 times and all of them passed!
This handles a rare race condition that occurs when the session hash
is not updated by the backend during the password change process.
This mostly occurs in puppeteer tests, but could occur to a user.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.
Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.
Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
Improve OpenAPI documentation of /zulip-outgoing-webhook by moving
data and making appropriate additions from its couterpart in the
/outgoing-webhook docs. Then remove the redundant documentation
from the doc and add command to render OpenAPI documetation. Also
add a test to outgoing_webhooks_interface.py to ensure that OpenAPI
documentation is correct.
Fixes#16203.
In list_render.js, [...list] requires list to be an array, and
widget.set_sorting_function(...opts.init_sort) requires init_sort to
be an array.
This allows the Node tests to pass in Babel strict mode. We currently
use loose mode for performance, and so we should test in loose mode as
well; but we must never depend on loose mode for correctness, since
individual Babel transformations may stop being applied as our browser
support baseline improves.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This lets the backend tests pass if zilencer has been (manually)
removed from EXTRA_INSTALLED_APPS, by skipping the tests that require
it. test-backend complains that some URLs are untested in this case:
ERROR: Some URLs are untested! Here's the list of untested URLs:
api/v1/users/me/android_gcm_reg_id
api/v1/users/me/apns_device_token
team/
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It's never safe to access the mock RemoteZulipServer object; this
caused exceptions on every request in production for any server with
ZILENCER_ENABLED=False.
This rule is a bit marginal, in that we've only seen this mistake
once, but it is really subtle and took a while for translators to
notice it, so seems worth linting for anyway.
This change fixes a translation bug that prevented a string in the
message retention dropdown options from getting translation.
Removing the space properly matched the string with the translated
strings.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.
Fixes: #1587.
Having this option in preview feels rather odd. The code here
would either be pasted from elsewhere (in which case it would
be in the clipboard already) or it could be copied from the
writebox just as easily.
Clicking on the copy-to-clipboard button triggers the clipboard.js
API to dynamically set the text to be copied. This text is the
actual code content from the sibling <code> element (extracted
though jQuery text() method).
The html structure would now look like:
<div class="codehilite">
<pre>
<button> The copy button </button>
<span></span>
<code>......</code>
</pre>
</div>
Additionally, this preserves the original code formatting of
the codeblock during copy-paste.
Tests amended.
Fixes: #15208
Extracting a section for presence endpoints and using path() rather
than re_path() results in a much cleaner implementation of this
concept.
This eliminates the last case where test_openapi couldn't correctly
match an endpoint documentation with the OpenAPI definitions for it.
This renames 'group_id' to 'user_group_id' in the api docs to remove
the naming mismatch between the url config and the docs and eventually
remove the 'user_groups' endpoints from 'pending_endpoints' in
test_openapi.py.
'user_groups' endpoints are currently under 'pending_endpoints' in
test_openapi.py (even after being documented except one), due to the
'user_group_id' and 'group_id' parameter name mismatch in the
url config and the view functions.
This commit includes 'path_only=True' for 'user_group_id' parameter in
views to avoid the failure of 'test_openapi_arguments', in
test_openapi.py, which excludes the path parameters. This is a prep
commit for renaming 'group_id' to 'user_group_id' in the documentation
and removing the 'user_groups' endpoints from 'pending_endpoints'.
This queue had a race condition with creation of another Timer while
maybe_send_batched_emails is still doing its work, which may cause
two or more threads to be running maybe_send_batched_emails
at the same time, mutating the shared data simultaneously.
Another less likely potential race condition was that
maybe_send_batched_emails after sending out its email, can call
ensure_timer(). If the consume function is run simultaneously
in the main thread, it will call ensure_timer() too, which,
given unfortunate timings, might lead to both calls setting a new Timer.
We add locking to the queue to avoid such race conditions.
Tested manually, by print debugging with the following setup:
1. Making handle_missedmessage_emails sleep 2 seconds for each email,
and changed BATCH_DURATION to 1s to make the queue start working
right after launching.
2. Putting a bunch of events in the queue.
3. ./manage.py process_queue --queue_name missedmessage_emails
4. Once maybe_send_batched_emails is called and while it's processing
the events, I pushed more events to the queue. That triggers the
consume() function and ensure_timer().
Before implementing the locking mechanism, this causes two threads
to run maybe_send_batched_emails at the same time, mutating each other's
shared data, causing a traceback such as
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "/srv/zulip/zerver/worker/queue_processors.py", line 507, in maybe_send_batched_emails
del self.events_by_recipient[user_profile_id]
KeyError: '5'
With the locking mechanism, things get handled as expected, and
ensure_timer() exits if it can't obtain the lock due to
maybe_send_batched_emails still working.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit hides the "Save" and "Cancel" buttons
after the first click and shows a spinner until a
successful / failed response is received.
We do not allow sending any other message edit
requests during this time frame, similar to how
our inline topic edit ui works.
Fixes#16143.
This commit removes the part which mentions specifying the property field
type in new feauture tutorial as it is no longer required to specify the
type.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.
Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.
Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
If you look at line number 1121 (new) of commit 14c0a387cf,
I seem to have accidently set the description for a status
200 response to "Bad Request" instead of "Success" which
is what it really is. It's basically an ugly typo (maybe
due to hastily copy-pasting the template).
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
I noticed RateLimitTests.test_hit_ratelimits fails when run as an
individual test, but never when run after other tests. That's due to the
first API request in a run of tests taking a long time, as detailed in
the comment on the change to the setUp method.
Django always sets request.user to a UserProfile or AnonymousUser
instance, so it's better to mimic that in the tests where we pass a
dummy request objects for rate limiter testing purposes.
The data is now stored in memory if things are happening inside tornado.
That aside, there is no reason for a comment on a rate_limit_user call
to talk about low level implementation details of that function.
I can find no evidence of it being possible to get an Exception when
accessing request.user or for it to be falsy. Django should always set
request.user to either a UserProfile (if logged in) or AnonymousUser
instance. Thus, this seems to be dead code that's handling cases that
can't happen.
`zproject/settings.py` itself is mostly-empty now. Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
It's possible that this is a new name for the "due"
field, but it's not totally clear.
In the exception we saw in the field:
payload['action']['data']['old']['dueComplete'] = False
payload['action']['data']['card']['dueComplete'] = True
We remove the fixture for create_check_item, which
has been bit-rotting for as long as we have ignored
this type of card data.
Our new test is more powerful, in the sense that it
shows we successfully ignore all fixtures of this
type.
If we want to handle this, we'll just need to get
new, representative fixture data from trello.
Commit c4254497b2
curiously had get_body() round tripping its data
through json load and dump.
I have seen this done for pretty-printing reasons,
but it doesn't apply here.
And if you're doing it for validation reasons,
you only need to do half the work, as my commit
here demonstrates.
We arguably don't even need the fail-fast code
here, since our fixtures are linted to be proper
json, I believe, plus downstream code probably
gives reasonably easy-to-diagnose symptoms.
For most cases you don't need to override `get_body`,
and for non-trivial cases, there's really no set pattern.
(It would be nice if we didn't default to json extensions
and just forced folks to be explicit about file extensions,
which would remove a whole class of `get_body` overrides.)
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.
Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.
I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).
In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
We tighten up the mypy types here. And then
once we know that expected_message and expected_topic
are never None, we don't have call the do_test_message
and do_test_topic helpers any more, so we eliminate
them, too.
Finally, we don't return a message, since no tests
use the message currently.
If we're not passing in expected_topic or expected_message
to check_webhook, it's better to just call send_webhook_payload,
since we'll want to explicitly check our messages
anyway.
This preps us to always require those fields for
check_webhook, which can prevent insidious testing no-ops.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit. I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.
I also clean up the code a bit for the tests that
have two new messages arriving.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.
And then the other two callers are already
checking topics, so they might as well check
stream names, too.
We also have the two stream-oriented callers
make their own call to "subscribe". And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.
Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper. This is partly to satisfy mypy, but
it's also a good sanity check.
This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.
In passing I also remove "bytes" from the Union type for
"payload" parameter.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.
Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.
We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used. So
for now I just made sure the docstrings of the two
sibling functions reference each other.
The "EXPECTED_" prefix and "_EVENTS" suffix
usually provided more noise than signal.
We also use module constants to avoid the "self."
noise. It also makes it a bit more clear which
constants actually have to be in the class (e.g.
"FIXTURE_DIR_NAME") to do their job.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:
def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
if expected_message is not None:
self.assertEqual(msg.content, expected_message)
Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
We had optional parameters for expected_topic and
expected_message, which are trivial to eliminate,
since the integration is really simple.
And we were doing strange things trying to reset
class variables at the end of tests. Now we just
set them explicitly in the tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored. So the tests weren't
testing anything.
Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".
I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
PostgreSQL packages for Ubuntu run "initdb" without specifying locale
on installation. It means that the default template
database (template1) is created by the system default locale. If the
system default locale is non UTF-8 compatible encoding such as
en_US.ISO-8859-15, "zulip" database is also created non UTF-8
compatible encoding such as LATIN9.
You can reproduce this case by running the following script:
apt update
apt install -y locales
locale-gen en_US.ISO-8859-15
update-locale LANG=en_US.ISO-8859-15 LANGUAGE=en_US:
apt install -y wget
wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
tar xf zulip-server-latest.tar.gz
zulip-server-*/scripts/setup/install \
--hostname=zulip-test.example.com \
--email=zulip-test-admin@example.com \
--self-signed-cert
scripts/setup/install is failed with the following error:
+ ./manage.py migrate --noinput
Operations to perform:
Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying zerver.0001_initial...Traceback (most recent call last):
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
return wrapper_execute(self, super().execute, query, vars)
File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
return action(sql, params)
psycopg2.errors.UntranslatableCharacter: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
CONTEXT: line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 50, in <module>
execute_from_command_line(sys.argv)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
post_migrate_state = executor.migrate(
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 105, in database_forwards
self._run_sql(schema_editor, self.sql)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 130, in _run_sql
schema_editor.execute(statement, params=None)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
cursor.execute(sql, params)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
return wrapper_execute(self, super().execute, query, vars)
File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
return action(sql, params)
django.db.utils.DataError: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
CONTEXT: line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"
This flake occurs because the Verona dropdown menu couldn't be
clicked, in rare cases, because puppeteer would click it too
quickly before it appears and then fails. To fix this we wait for it
to fully appear and then click it. Around 1000 runs
passed without a failure.
The error the flake caused was:
TimeoutError: waiting for selector
"#org-submit-notifications[data-status="unsaved"]" ...
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later. Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.
Only look for images which have a defined `src` attribute on them. We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's. It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
c2f9227892 switched from a long hard-coded list in `TEST_FUNCTIONS` to
using a new `@openapi_test_function` decorator; update the
documentation about writing API docs accordingly.
Per [1], the sentry API returns frames sorted from oldest to newest.
As such, matching against the first filename that matches is most
likely not the right frame.
Match against the last frame with the guilty filename.
[1] https://develop.sentry.dev/sdk/event-payloads/stacktrace/
The original commit was broken here:
b553507412
The intention was to run the same loop for all
settings, but instead, we did a funny loop of
just resetting schema_checker, and then we only
actually tested the last value of the loop.
The dispatch test here really only cares that values
get passed on.
Note that the dispatch code ignores the email field, because
we only send subscription/update events to the user
whose subscription has changed.
This fixes a bug with the original frontend-side implementation for
has: filters, where it would incorrectly not match content in cases
where the message's nesting structure did not have an outer tag.
Bug was introduced in 02ea52fc18.
Fixes#16118.
This commit adds "role" field to the Subscription objects passed to
clients. This is important preparation for being able to work on the
frontend for this feature.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.
Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
This restores the Tab + Enter shortcut to send.
We are floating the send button to the right so that it still looks like
before. Instead of moving the button we could have also given every
message control button a tabindex, but these would be cumbersome to
maintain.
Tweaked by tabbott to add a comment recording the reasoning behind
the somewhat unusual CSS here.
Part of #15910.
Previously the emoji picker, the formatting help, the button to attach
files, the video call button, the Drafts button and the Press Enter to
send checkbox were all inaccessible from the keyboard.
This does break the Tab + Enter workflow for sending messages, which is
fixed in the next commit by moving the Send button to be the first
element after the textarea.
Part of #15910.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
Since bug https://bugs.python.org/issue3445 was resolved in Python
3.3, we can avoid the use of assigned=available_attrs(view_func) in
wraps decorator (which we were only using because we'd copied code
that handled that from Django).
Also available_attrs is now depreciated from Django 3.0 onwards.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.
We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
This function now matches the copy in zerver/lib/actions.py.
This is the same migration as
b250e42f61c525029bd2b3bbb8f4ea93ece62072; orjson enforces that we
don't use integers as keys in JSON dictionaries.
Apparently, we were incorrectly using constants for title/description
rather than the nice non-constant values from og:title and
og:description in our meta tags.
The dispatch for presence is a trivial one-liner,
so the test just makes sure three important parameters
get passed along.
We will eventually want to use the fixtures data in
other presence-related tests, but for now the only
goal is to make it pass the schema checks.
This is a prep commit for changing the bots list page to show normal
user popover instead of extended profile one. This is added so that any
open popovers are closed while switching panels in settings overlay.
This change was not needed previously because we were using modal for
showing extended user profile. Now as we would be adding popover, we
would need this change to close the open popovers while switching
panels in settings overlay.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream
The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.
We have also added a new exception class StreamAdministratorRequired.
This commit adds role field to the Subscription class. Currently,
there are two option of roles - STREAM_ADMINISTRATOR and MEMBER.
We also add a property 'is_stream_admin' for checking whether the
user is stream admin or not.
Via API, users can now access messages which are in web-public
streams without any authentication.
If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.
Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
django.security.DisallowedHost is only one of a set of exceptions that
are "SuspiciousOperation" exceptions; all return a 400 to the user
when they bubble up[1]; all of them are uninteresting to Sentry.
While they may, in bulk, show a mis-configuration of some sort of the
application, such a failure should be detected via the increase in
400's, not via these, which are uninteresting individually.
While all of these are subclasses of SuspiciousOperation, we enumerate
them explicitly for a number of reasons:
- There is no one logger we can ignore that captures all of them.
Each of the errors uses its own logger, and django does not supply
a `django.security` logger that all of them feed into.
- Nor can we catch this by examining the exception object. The
SuspiciousOperation exception is raised too early in the stack for
us to catch the exception by way of middleware and check
`isinstance`. But at the Sentry level, in `add_context`, it is no
longer an exception but a log entry, and as such we have no
`isinstance` that can be applied; we only know the logger name.
- Finally, there is the semantic argument that while we have decided
to ignore this set of security warnings, we _may_ wish to log new
ones that may be added at some point in the future. It is better
to opt into those ignores than to blanket ignore all messages from
the security logger.
This moves the DisallowedHost `ignore_logger` to be adjacent to its
kin, and not on the middleware that may trigger it. Consistency is
more important than locality in this case.
Of these, the DisallowedHost logger if left as the only one that is
explicitly ignored in the LOGGING configuration in
`computed_settings.py`; it is by far the most frequent, and the least
likely to be malicious or impactful (unlike, say, RequestDataTooBig).
[1] https://docs.djangoproject.com/en/3.0/ref/exceptions/#suspiciousoperation
The OS upgrade paths which go through 2.1 do not call
`upgrade-zulip-stage-2` with `--audit-fts-indexes` because that flag
was added in 3.0.
Add an explicit step to do this audit after the 3.0 upgrade. Stating
it as another command to run, rather than attempting to tell them
to add it to the `upgrade-zulip` call that we're linking to seems
easiest, since that does not dictate if they should upgrade to a
release or from the tip of git.
We do not include a step describing this for the Trusty -> Xenial
upgrade, because the last step already chains into Xenial -> Bionic,
which itself describes auditing the indexes.
Fixes#15877.
Only Zulip 3.0 and above support the `--audit-fts-indexes` option to
`upgrade-zulip-stage-2`; saying "same as Bionic to Focal" on other
other steps, which are for Zulip 2.1 or 2.0, will result in errors.
Provide the full text of the updated `upgrade-zulip-stage-2` call in
step 5 for all non-3.0 upgrades. For Trusty to Xenial and Stretch to
Buster, we do not say "Same as Xenial to Bionic" , because it is
likely that readers do not notice that step does not read "Same as
Bionic to Focal."
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.
(ujson raised ValueError directly, as did json in Python 2.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created. Appropriate changes have been made in
the docs and test_users.py.
Fixes#16072.
These escapes are valid YAML 1.2 (for JSON compatibility) but not
valid YAML 1.1, which means they don’t work with the faster
yaml.CSafeLoader that we’d like to transition to.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
datetime objects are not ordinarily JSON serializable. While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways. So we want to fix the post-processing
code to do its job.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Also add helper functions needed.
`select_item_via_typeahead` has been ported from casper
and is exactly same in puppeteer to as I couldn't find
any better way for that purpose.
I had a misconception with hidden and visible options
and thought `hidden: false` was same as `visible: true`
and other way too.
But `hidden: false` or `visible: false` does nothing
more than checking if the selector exists.
Also, to mention, `visible: false`'s were fixed in
33e19fa7d1
It is more suited for `process_request`, since it should stop
execution of the request if the domain is invalid. This code was
likely added as a process_response (in ea39fb2556) because there was
already a process_response at the time (added 7e786d5426, and no
longer necessary since dce6b4a40f).
It quiets an unnecessary warning when logging in at a non-existent
realm.
This stops performing unnecessary work when we are going to throw it
away and return a 404. The edge case to this is if the request
_creates_ a realm, and is made using the URL of the new realm; this
change would prevent the request before it occurs. While this does
arise in tests, the tests do not reflect reality -- real requests to
/accounts/register/ are made via POST to the same (default) realm,
redirected there from `confirm-preregistrationuser`. The tests are
adjusted to reflect real behavior.
Tweaked by tabbott to add a block comment in HostDomainMiddleware.
This redirect was never effective -- because of the
HostDomainMiddleware, all requests to invalid domains have their
actual results thrown away, and replaced by an "Invalid realm" 404.
These lines are nonetheless _covered_ by coverage, because they do
run; the redirect is simply ineffective. This can be seen by the test
that was added with them, in c8edbae21c, actually testing the contents
for the invalid realm wording, not the "find your accounts" wording.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.
Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`. Adjusts tests to match.
consume_time_seconds wasn't properly defined at the beginning, so when
a BaseException that isn't a subclass of Exception is thrown, the
finally: block could be entered with it still undefined.
By defaults, `requests` has no timeout on requests, which can lead to
waiting indefinitely. Add a half-second timeout on these; this is
applied _inside_ each retry, not overall -- that is, with retries any
of these functions may take a total of 1.5s.
Use the `no_proxy` proxy, which explicitly disables proxy usage for
particular hosts. This is a slightly cleaner solution than ignoring
all of the environment, as removing proxies is specifically what we
are attempting to accomplish.
The change in #2764 provided a better error message on one of the
three calls into Tornado, but left the other two with the old error
message. `raise_for_status` was used on two out of three.
Use a custom HTTPAdapter to apply this pattern to all requests from
Django to Tornado.
We need to replace 'visible: false' with 'hidden: true' to wait
for elements to get hidden. Using 'visible: false' just checks
whether the selector exists or not and does not check whether
the element is hidden or not.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.
This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.
Fixes#16066.
There are three exceptions in Python3 which are descended from
BaseException, but not Exception: GeneratorExit, KeyboardInterrupt,
and SystemExit. None of these are suitable to be sent to Sentry.
For example, SystemExit is raised by `sys.exit`; in that sense, it is
never "uncaught" because we chose to cause it explicitly.
Use the suggested form[1] for ignoring specific classes of exceptions.
[1] https://github.com/getsentry/sentry-python/issues/149#issuecomment-434448781
This requires a few redundant runtime isinstance
checks, but the extra assertions arguably make
the code more readable, and isinstance checks
are extremely negligible.
This will let PyYAML link against LibYAML when PyYAML is next
installed. Due to virtualenv-clone, that won’t happen until the next
Python package removal anyway, so we don’t bother bumping
PROVISION_VERSION.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
datetime objects are not ordinarily JSON serializable. While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways. So we want to do this explicitly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since our Webpack config passes pre-minified JS files to
script-loader, they can’t be used as modules. Use the normal
unminified version, letting Webpack minify it and give us source maps.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
JSON keys must be strings, and orjson enforces this. Mypy didn’t
catch the mismatched type of profiles_by_user_id because it doesn’t
understand CustomProfileFieldValue.field_id.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It doesn’t end well. Or sometimes it doesn’t end (OverflowError:
Maximum recursion level reached).
Introduced by commits ccdf52fef6 and
94d2de8b4a (#15601).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is a prep commit for changing the bots list page to show
normal user popover instead of extended profile one.
This is added so that any open popovers are closed, if one
tries to close the overlay.
This was not needed previously because we were using modal for
showing extended user profile. Now as we would be adding popover,
we would need this to close the open popovers before closing overlay.
This commit renames the show_user_info_popover function to
show_user_info_popover_for_message, as it is used to open
the popover for users which are essentially related to a
particular message, like message sender and mentioned user.
This section referenced the old Puppet path for
puppet/zulip/templates/nginx/upstreams.conf.template.erb, and overall
felt focused on naming files that developers never look at.
We also just make the test express what's actually
happening in the code; we just pass the entire
"exports" section of the event to the settings code
and let it do its thing.
We should send PATCH request for changing stream description only if
it is actually changed, there is no need to send request to backend
if the description is not changed.
We shoudl only send PATCH request to API for stream rename only if
stream name is actually changed.
Previously, when trying to save the stream name without actually
changing it, backend returned 400 with error as "Stream already
has that name". Ideally, we should not make PATCH request if name
is not changed and it should just close the edit widget.
This commit solves this bug.
I’m going to assume that this is not intended to be an optimization
for “WekBit” and can, in fact, be deleted.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit abbd8a7f45 (#13112) should have
removed the nonexistent user-drag property rather than the
Webkit-specific -webkit-user-drag property.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit changes change_stream_privacy function to only
send the values of changed settings to backend.
We also avoid sending PATCH request if none of the settings
in stream privacy modal are changed.
This change also fixes the bug in changing stream permissions
for realms with limited plans.
Fixes#16024.
Our intent throughout the codebase is to treat email
case-insensitively.
The only codepath affected by this bug is remote_user_sso, as that's the
only one that currently passes potentially both a user_profile and
ExternalAuthDataDict when creating the ExternalAuthResult. That's why we
add a test specifically for that codepath.
We follow the naming convention.
I also arbitrarily assign the "op" of
"add" to the attachment event, even
though we don't meaningfully test it.
The situation with attachment from the
dispatch test point of view is that
we just want to test that the one line
of code that calls into attachments_ui
(for all three ops) does get dispatched
correctly. We eventually want to get
deeper coverage there, but attachments_ui
wasn't written in the most test-friendly
way. I think it might actually be easy
to fix up attachments_ui to make it a
bit easier to test, but it's out of the
scope of my current PR.
The benefit here is check-node-fixtures
now gives a more concrete plan for
moving schemas to event_schema.py.
We extract test_realm_emojis, and we make
the name of the event more explicit (adding
the __update suffix).
We also add the "op" of "update" here, which
is sort of a quirk of the api, since we don't
actually have alternatives like add/remove,
and therefore the current frontend code doesn't
look at the "op", and thus the original tests
never had to provide a correct value for it.
To increase code reusability and reduce code redundancy, we move data
structures which occur multiple times in the OpenAPI documentation to
the `schemas` section. Note that this a pure data movement commit
without any changes to the data beyond removing over-specific
descriptions (E.g. that suggest the user group was just created).
(Future commits will use these)
We move this function from `user_pill.js` to `pill_typeahead.js`.
The function has also been renamed to `set_up`.
The move was made because there are plans to update the pills
typeahead (i.e. to include user-groups/streams in the results).
Thus this function should not belong in `user_pill.js`.
This commit allows skipping over any disabled tabs
that are in the middle when using the left or right
arrow keys.
We also add `enable_tab` to the `components` API.
".stream-info-title" selector is used to hide both
"#add_new_stream_title" and "#stream_settings_title"
classes. This will be helpful when we add new html
elements to display in the title area..
`clear_edit_panel` can be removed as the next line to
where it is called makes it redundant, we only need
to de-select the stream row, as done in this commit.
Since the subscriptions container contains multiple toggler components,
it is helpful to know that the function's tab key returns all the active
tabs in the page (currently there are 2). Thus `tab` is changed to `tabs`.
Also, which togglers tab data is being used has been made more specific.
After the latest message in a stream is deleted, we should update
the max_message_id in the stream.
Removed false comment in message_util.get_messages_in_topic
this method only takes 2ms for 10,000 messages loaded locally.
Fixes#15992.
If the last message of the topic was deleted, we update the stored
message_id in the topic history so that the topic order in topic_list
is updated correctly.
When messages weren't locally echoed,
`wait_for_full_processes_message` fails to assert that the message
is being sent. It was figured out by tabbott that the messages
aren't locally echoed because of content no loading completely.
So, this commit changes the `log_in` function to wait till
the selector '#zhome .message_row' is visible which indicates
that the cotent is loaded.
Removed the waitForNavigation in the `log_in` as it would
become a redundant check after this change.
Also removed this same check present in 03-compose.js which
becomes reduntant as we already do it in `log_in`.
This test changes user password causing all subsequent
tests to fail. Since rechanging the password would be
a redudant test/task or having manually entering the
new password in every test after this aren't good ideas,
this commit makes the settings test run in the end by
renaming it be numbered 16. It is assumed that we'll
end up having 16 tests seeing the number of tests in
casper and considering 03 and 02 from casper are being
combined as 02-message-basics.js. Though 03 of casper
has not yet been added in message-basics test it will
soon be added.
It only cancel previous runs on forks or for pull request builds
and does not run for zulip/zulip repo pushes. It finishes pretty
quickly (under 1 minute) so it fine to have it as a workflow
rather than to add a new step under every single job to cancel it's
previous runs.
The only downside to this is that GitHub creates a notification for
the cancelled job just like it does for failed jobs!!!
This commit adds EMAIL_PORT setting for explicitly specifying the
port of SMTP provider in dev_settings.py.
We also change email_backends.send_email_smtp to pass EMAIL_PORT
along with EMAIL_HOST to smtplib.SMTP.
After this change, we will not need to include the port along with
host in EMAIL_HOST.
Also updated the email.md docs accordingly for this change.
Previously there was a documented_events set which provided for partial
OpenAPI documentation while documentation was still going on. But since
the documentation is complete now, remove it.
This giant commit completes basic OpenAPI documentation for all events
in Zulip's real-time events API.
Further work will be required in the near future to make
/api/get_events usable.
With many edits by tabbott for wording and correctness (especially
around which clients receive events, and their purpose).
The variant `update_message` events have this extra sender field not
present in normal update_message events; this field has no purpose, so
we remove it.
Apparently, `update_message` events unexpectedly contained what were
intended to be internal data structures about which users were
mentioned in a given message.
The bug has been present and accumulating new data structures for
years.
Fixing this should improve the performance of handling update_message
events as well as cleaning up this API's interface.
This was discovered by our automated API documentation schema checking
tooling detecting these unexpected elements in these event
definitions; that same logic should prevent future bugs like this from
being introduced in the future.
Use `ujson.loads(ujson.dumps())` wrapper on events sent for OpenAPI
testing so that all tuples are converted into arrays as tuples aren't
valid in JSON.
As part of issue #15344, the error report emails add the user role
information. This commit adds the user role information to be used
by sentry as well.
To make it easier to check if there is user information to be used
in the error report emails, we create a user object inside report.
Now, to check if we have the user's full name, email, etc, we just
need to do report['user']['user_full_name'] rather than check
each information one by one, because if the value of one key in
the report is different than None, all the others will be as well.
The hash keys were missing hash for package.json and yarn.lock
because they were not present since we don't do a full checkout
in this job. We fix this by sending over those files and generating
hashes from them.
I usally verify these cache keys by clicking the Restore <cache>
step dropdown menu and then clicking the Run ... dropdown menu again
to see the generated hash.
The combination of `--force --noop` is potentially confusing, but
currently `--noop` makes no sense without `--force`, as it will prompt
and then not make changes.
Make `--noop` skip the prompt as well.
ES and TypeScript modules are strict by default and don’t need this
directive. ESLint will remind us to add it to new CommonJS files and
remove it from ES and TypeScript modules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 114cc1ec25 (#15949) introduced a
subtle bug because sortablejs provides both a CJS module and an ES
module that expose different interfaces to CJS require() under
Webpack. This difference will disappear when we convert
settings_profile_fields to an ES module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Clients that close their socket to nginx suddenly also cause nginx to close
its connection to uwsgi. When uwsgi finishes computing the response,
it thus tries to write to a closed socket, and generates either
IOError or SIGPIPE failures.
Since these are caused by the _client_ closing the connection
suddenly, they are not actionable by the server. At particularly high
volumes, this could represent some sort of server-side failure;
however, this is better detected by examining status codes at the
loadbalancer. nginx uses the error code 499 for this occurrence:
https://httpstatuses.com/499
Stop uwsgi from generating this family of exception entirely, using
configuration for uwsgi[1]; it documents these errors as "(annoying),"
hinting at their general utility."
[1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#ignore-sigpipe
Per the API documentation[1], the following codes all correspond to
HTTP 404:
- `34`: **Sorry, that page does not exist.** The specified resource
was not found.
- `144`: **No status found with that ID.** The requested Tweet ID is
not found (if it existed, it was probably deleted)
- `421`: **This Tweet is no longer available.** The Tweet cannot be
retrieved. This may be for a number of reasons.
- `422`: **This Tweet is no longer available because it violated the
Twitter Rules.** The Tweet is not available in the API.
Treat all of these identically.
[1] https://developer.twitter.com/en/docs/basics/response-codes
Since the title of a merge request can often change, it shouldn't be a
part of the topic that we send the message to. Otherwise things would
get messy and confusing.
But at the same time we don't want to make this mandatory. So we add
a new boolean GET parameter that can toggle whether or not the topic
should include the MR title (`use_merge_request_title`).
Fixes#15951.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
We already have single-key shortcuts for all message controls but Zulip
should be usable from the keyboard without having to learn a bunch of
Zulip-specific keyboard shortcuts.
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base.
We purged tab_list in 1267caf5009118875f47fdafe312880af08024e1.
This commit purges tab_bar, it includes:
- A blanket search and replace of tab_bar with message_view_header.
- Splitting a single line comment in
tab_bar.js / message_view_header.js.
- The renaming of tab_bar.js to message_view_header.js.
- The renaming of tab_bar.hbs to message_view_header.hbs.
- A blanket search and replace of tab_data with
message_view_header_data.
- Replacing the single occurrence of tabbar with message_view_header
(it was within a comment.)
This commit upgrades 0015_clear_duplicate_counts migration to remove
duplicate count in StreamCount, UserCount, InstallationCount as well.
Fixes https://github.com/zulip/docker-zulip/issues/266
For all buttons in the compose box, `href="#"` is replaced
by "tabindex=0" so that the buttons are still focusable.
This change also fixes a bug that caused the Formatting
button to redirect to All messages.
The S3 data export tool's upload code path uses this nice boto
callback feature for showing a progress bar, which is nice for the
management command. It's spammy/broken in production and the backend
tests, so we change percent_callback to be a parameter passed in so
that it can only be used in the contexts where it makes sense.
There were a lots of flakes in CI recently because typeahead didn't
appear when Enter was pressed and real emails are not accepted as
valid inputs. To fix this we wait for typeahead to appear and then
click that instead of Enter. We also use delay option to type the
email (100ms delay between keypresses) since without we'd also get
flakes.
Re-enable puppeteer test in CI after this fix too.
For mysterious reasons, this avoids the following message printed by
webpack on a cold cache after upgrading postcss-nested from 4.2.1 to
4.2.2:
Ignoring local source map at "/srv/zulip/<no source>" as resource is missing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reverts commit c3779338c6 (part
of #14638), which incorrectly depended on commits from the future,
with the effect of either halting the flow of entropic time in an
irresolvable temporal paradox, summoning extradimensional beings to
rain destruction on the galaxy, or failing CI.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.
This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
Modifies filter_stream_authorization so that web-public streams are
added in the list of authorized streams that a guest user can
subscribe.
This commit is part of PR #14638 that aims to allow guest users
to browse and subscribe to web-public streams.
In this commit, we grant guest users access to stream history,
send message and common stream data of web-public streams.
This is part of PR #14638 that aims to allow guest users to
browse and subscribe to web-public streams.
This modification allows guest users to have access to web-public
streams subscribers, even if they aren't subscribed or never
subscribed to that stream.
This commit is part of PR #14638 that aims to allow guest users to
browser and subscribe to web-public streams.
Now, gather_subscriptions include web-public streams in the 3 sets
of streams that it returns, subscribed, unsubscribed and never
subscribed.
This is part of PR #14638 that aims to allow guest users to browse and
subscribe to web-public streams.
This change makes the flow more coherent by instead of checking,
in the last condition, if the user isn't authorized to access that
stream, check if they are, as it is done in the other checks. Only
if all the conditions are false, which means that the user doesn't
have access to that stream, the stream is added to the
unauthorized_streams list.
Previously the title for all pages of the user and API documentation was
just "Zulip", which does not only bad for UX but also for accessibility.
We were already extracting the title from the Markdown for the og:title
tag, so we just need to set the <title> tag.
Since our documentation fetches pages with Ajax if you have JavaScript
enabled, we also need to save the titles in the article cache.
Part of #15948.
Not having a focus outline is very bad for accessibilty.
Browsers have it by default but we completely disabled it for links in
the sidebar in 9955580251.
Showing the outline when selecting a page in the sidebar can be
distracting, so we hide the outline for the highlighted sidebar link.
Since every focusable element however should have a focus outline, we
make the highlighted link unfocusable by setting tabindex=-1 (which also
makes sense since the link to the current page doesn't do anything
anyway).
Part of #15948.
😛 should be the most general version, which is the one
with open eyes. Other apps do the same and it also means that :P, which
is converted to 😛 is rendered like the emoticon.
Fixes#15970.
This is used rarely enough that it’s easier to document how to use it
as a non-global than to document the horrifying things that might go
wrong as a global.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Increasing the uwsgi listen backlog is intended to allow it to handle
higher connection rates during server restart, when many clients may
be trying to connect. The kernel, in turn, needs to have a
proportionally increased somaxconn soas to not refuse the connection.
Set somaxconn to 2x the uwsgi backlog, but no lower than the
default (128).
Card descriptions aren't dates, and calling prettify_date on them results in removing upper case T characters, replacing uppercase Z characters with " UTC", etc. in descriptions when they appear in Zulip.
This was pretty clearly just a copy/paste mistake (these functions are very closely parallel to the *_due_date_* functions above, which do work on dates and call prettify_date).
Also add a Draft object-to-dictionary conversion method.
The following commits will provide an API around this
model using which our clients can sync drafts across each
other (if they so wish too). As of making this commit, we
haven't finalized exactly how our clients will use this.
See https://chat.zulip.org/#narrow/stream/2-general/topic/drafts
For some of the discussion around this model and in general,
around this feature.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Unlike the other Python datetime to Unix timestamp conversion
function (`datetime_to_timestamp`), `datetime_to_precise_timestamp`
won't drop the microseconds.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.
Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
Uses git release as this version 3.4.0 is not released to pypi.
This is required for removing some overriden functions of
apple auth backend class AppleAuthBackend.
With the update we also make following changes:
* Fix full name being populated as "None None".
c5c74f27dd that's included in update assigns first_name and last_name
to None when no name is provided by apple. Due to this our
code is filling return_data['full_name'] to 'None None'.
This commit fixes it by making first and last name strings empty.
* Remove decode_id_token override.
Python social auth merged the PR we sent including the changes
we made to decode_id_token function. So, now there is no
necessity for the override.
* Add _AUDIENCE setting in computed_settings.py.
`decode_id_token` is dependent on this setting.
Improved markup of help-text.
Showing Email as plain-text instead of disabled input.
Changed page heading to 'Create your organization' in realm creation form
and 'Create your account' in normal signup form.
Grouped org settings and user settings with fieldsets.
Reduced space between Password field and Password strength bar.
Also, updated the corresponding test cases.
Partially Fixes: #15750.
We will store list of stream ids to sort streams instead of names.
We have added a compare_function for sorting the list of stream_ids
by comparing stream names.
This change helps us to remove a couple of get_sub calls and using
stream ids instead of name also helps in avoiding bugs caused due
to live update on renaming of stream.
We add a function subscribed_stream_ids which returns an array
of stream ids of all subscribed streams.
This is a prep commit for changing the logic for sorting streams
to store stream ids instead of names.
We should not allow every function who wants to narrow to All
messages to come up with their own method to do so. This
commit makes existing such functions use hashchange library to
do so.
Remove click event on All message button, it already contains
an <a> tag which navigates correctly.
We always use hashchange.go_to_location method now to open the
info_overlay, this makes sure that the url hash are reliable and
hotkeys don't get confused if an overlay is open or not.
We don't want to change hash to "" (this also doesn't navigates
us to 'All messages' view, hence the bug was not noticed.) on
exit of info_overlay.
Three reasons:
1. The sliding was disorienting.
2. The collapsing disallowed searching for other pages with Ctrl+F.
3. The collapsing mechanism wasn't accessible (not usable with the
keyboard / no ARIA tags).
Tweaked by tabbott to center the left sidebar on the selected page.
Part of #15948.
This is a preparatory commit that exports user_sidebar_popped function
so that it can be used in hotkey.js for keyboard navigation support in
popover in right sidebar.
This commit is a preparotory commit to add support of keyboard
navigation by enabling movement using arrow keys and clicking of items
using enter key. So popover_items_handle_keyboard function so that it
can be resused other places.
This commit is preparatory commit to adding support for keyboard
navigation by focusing om first menu item of all of our popovers. So
exporting focus_first_popover_item so that it can be reused in other
places.
Document all events of `type`=stream i.e all `op`s. Also document some other
events.
Tweaked by tabbott to clarify some documentation details (especially
around who receives events).
Previously when hovering over a selected topic in the left sidebar
a barely different hover color was employed (and overriden in the dark
mode). This resulted in a small UX issue because after selecting a topic
in the sidebar it should immediately be colored as such (and not just
after moving the mouse cursor away).
Previously the left sidebar used a darker hover background than the
right sidebar, presumably to stand out more when hovering over an active
filter (which have a blue background in the light theme). This can be
more elegantly solved by using a transparent hover background.
There were two problems preventing the autofocus:
* The focus was triggered at the wrong time.
* transition: all; affected the visibility, making browsers
abort the focus since the input was still hidden.
Commit a9ca5f603b (#15863) incorrectly
converted these too; indexing a jQuery object gives you a DOM element.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The idea behind doing this is that we would rather let the code error
out rather than add to the logs. It's webhook code usually never uses
the logging module so this section of legacy code needed to be changed
or removed.
Assists PR #15942.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Note that require("moment") and require("moment-timezone") resolve to
the same thing, but the latter adds timezone support as a side effect.
So I went with the latter in every file where .tz is used.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Edit the function `validate_against_openapi_schema` and add some
helper functions to allow for validation of documented events.
Also add OpenAPI response validation in `verify_action` as it is
called in a large number of `/events` tests.
I'm not sure how this got added; it seems to have happened in a visual
redesign of the /features page. Certainly the claim should only have
been added after the work it described was done, and it has not.
This lets us test the recursion bug behavior of this logging handler
without resulting in `logging.error` output being printed to the
console in the event that the test passes.
The comments within update_message_backend function of views/message_edit.py
indicates 4 types of permissions which all edit a message. The 4th of these
indicates that a message is editable if the realm allows topic edits. This
was previously missing from the docs and is now added.
The parameter Stream.date_created is now sent down to the clients
for both:
- client.get_streams()
- client.list_subscriptions()
API docs updated for stream and subscriptions.
Fixes#15410
Use the default configuration, which catches Error logging and
exceptions. This is placed in `computed_settings.py` to match the
suggested configuration from Sentry[1], which places it in `settings.py`
to ensure it is consistently loaded early enough.
It is placed behind a check for SENTRY_DSN soas to not incur the
additional overhead of importing the `sentry_sdk` modules if Sentry is
not configured.
[1] https://docs.sentry.io/platforms/python/django/
This commit adds a handlebar template for the View source/Edit message
button in message controls in the message view.
This change also fixes the broken html titles that were added in
commit fdbab54614, and adds proper
internationalization for the title text.
This is done to decouple our message view related update events
from MessageListData as there are plans to create multiple
MessageListData objects. Instead we update the `stored_messages`
which tracks the complete data for all messages.
This just lets us temporarily assign a value
to a field.
Differences with the "override" scheme:
* override only works on globals
* override (when passed in via run_test) will
just automatically clean up at the end of
the function
Some parameters such as `to` and `topic` have been intentionally
undocumentecd hence fail request validation. So mark tests which
fail due to this accordingly.
Change the condition for allowing failed validation to the condition
that `if the test fails, response status code begins with 4`. Also
add `intentionally_undocumented` argument in `validate_request` for
allowing passing of tests which return `200` responses but fail
validation due to some intentionally undocumented feature in
OpenAPI specification.
This makes sure out fixture data for node tests
is realistic, according to the schemas in
zerver/lib/event_schema.py.
Note that we are still in the process of extracting
schemas from test_events.py -> event_schema.py,
so the checks here are somewhat incomplete as of
now.
One nice thing is that the program will tell us
what checkers are missing, so this can motivate
us to move more checkers to event_schema.py.
I considered just making this happen as part of
tools/test-js-with-node, but it's convenient to
run by itself. Also, it currently requires
Django (although we could fix that), which makes
it just expensive enough that I wouldn't want
to always run it before the node tests.
This is a pretty straightforward conversion.
The bulk of the diff is just changing emoji.js
to ES6 syntax.
There is one little todo that can be deferred
to the next commit--we are now set up to have
markdown.js require emoji.js directly, since
it is no longer on `window`.
We now show before/after, and we don't complicate
our other test that runs in a big loop.
And we take advantage of function injection to
not have to hack into the "real" emoji_codes
structure.
Note that we're simulating the missing emojis
at a slightly higher level, but we already had test
coverage that emoji.get_emoji_name returns
undefined for unknown codepoints.
The main thing here is that we check that the
actual data got put into our data structures.
(In general we want to move away from stubbing
data modules; any place where we stub data modules
is a relic of earlier days, where we were just
trying to set the bar for 100% line coverage,
even though some of the original coverage was
quite shallow.)
I also use real stubs instead of noops for
the calls out to UI-oriented modules.
In passing I tweak some comments in the actual
dispatch code.
This makes it so that the authoritative holder
of all emoji data is emoji.js, and all our
UI components that need emoji data consistently
pull data from emoji.js as needed.
Or to put it another way, we no longer need the
dispatch module to know that emoji_picker is
coupled to emoji precisely by the active_realm_emojis
data; it can now make fewer assumptions.
With update_emojis, it is pretty easy
to set up the tests with reasonable
data without reaching into internal
data structures.
Also, we can begin the process of
sharing the same data with our
dispatch tests (upcoming).
Fixes#15904.
settings is supposed to be a proper OneLogin_Saml2_Settings object,
rather than an empty dictionary. This bug wasn't easy to spot because
the codepath that causes this to demonstrate runs only if the
SAMLResponse contains encrypted assertions.
Added order_by("id") clause in query for RealmAuditLog
for consistent output.
It was causing zerver.tests.test_audit_log.TestRealmAuditLog
to fail due to order mismatch.
We want to undo overrides in reverse order,
which is important if you override the
same name more than once in the same
function.
Until today the code basically prevented
us from ever using the original implementation
of a name we stubbed, and most of them start
as undefined due to their parent modules
starting with `set_global`.
But I do want this proper, and I introduced
a tiny pitfall today.
There was only one place where we weren't
overriding a function, and the use case there
was fairly unique.
Knowing that we're dealing with only functions
will simplify override and allow us to add
features like detecting spurious stubs.
This forces us to more explicitly document at the
top of the file what dependencies we are stubbing,
plus it's less magical.
Also, we may want to do occasional audits of
set_global to clean up places where we mock
things like stream_data, which are probably just
easier to use the real version of now that we
have cleaner APIs to set up stream data.
The modules most affected by this change are our
dispatch-oriented tests--basically, all the
modules that test handling of Zulip events
plus hotkey.js.
Before we were making it impossible to reuse
the function again (so we were preventing
leaks), but it's fine to just restore the
original function, especially now that some
of our tests have grown bigger.
Commit a9ca5f603b (#15863) incorrectly
converted this; window is quite obviously a DOM element, not a jQuery
element.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Clock time checks lead to tests that nondeterministically fail when
the CI container is super slow, and there's no good reason this test
in particular needs to do that sort of test in addition to our
standard database query count check (which is already does).
Now when you are reading a single test, you can
explicitly see that the event and service handler
are tied to your bot, which is our test bot
for outgoing webhooks.
Decorating an entire test with a mock makes it
hard to ascertain where the actual mock behavior
is expected to happen, plus it clutters up
the parameter list.
In fact, we remove a dubious re-assertion here that
a mock was called. The assertion that a mock was
called was true, but it was misleading to think
the code right before it had invoked the mock.
Unlike stream_stats, I'm not aware of any of these having been used in
the last few years, and it's basically just really bad subsets of the
data in /activity, which also doesn't require shell access to use.
These haven't had real work or usage, AFAIK, since 2013.
This is an easy prep step to help out phase
out page_params.realm_emoji.
All callers pass in what's effectively
page_params.realm_emoji. (The dispatch
code does it indirectly.)
The "event log" in question was never useful in our test systems (and
hasn't been used for anything real since 2014). I'm not sure how we
ended up with in the CI configuration.
The previous steps for standing up a new host were somewhat manual.
This further scripts the process, by using the AWS CLI to start the
instance, and pass it a "user data" script to provision itself upon
boot. This results in a hands-off provisioning process which
completes in 5min.
Additional settings are required for `~/.zulip-install-server.conf`.
It is not suited for all roles, as it assumes one instance type and
security group value. Additionally, not all of the post-provision
process is currently automated -- Nagios SSH key verification, for
instance, is still a manual step. There are also additional steps for
database or frontend servers. Regardless, this is a move toward
automated provisioning.
Rishi never had a zulip.com email id. Giving preference to
zulip.com email id resulted in GitHub marking Rishi's commit
as anonymous. So we give preference to zulipchat.com instead,
which is linked to his GitHub account.
Including anon=1 in API requests will retrieve all contributors
of the repo. If there is no asscoiated GitHub account present for
the commits then the email and name of the author mentioned in
commit messages is returned.
All the steps are same from circleci except two steps:
1. The 'Add permissions ...' step is Actions specific as explained
in comments.
2. The step that used upload-artifacts is Actions verison of
presist_to_workspace.
Finally, I should note the duplication in this and zulip-ci
workflow. There are three reason this is not a problem:
1. It will be messy to mush this into zulip-ci workflow only for
benefit of un-duplicating the env and cache restore steps.
2. We needs this on its own workflow if we want to only run it
when production related dependencies are updated.
3. I don't see us updating the duplicated steps between both
workflow. Circle CI config is prefect example for this; nothing
is changed except for adding or updating steps which are not
duplicated.
This change makes it so if focal backend job fails the bionic
backend and frontend jobs keeps running. Previously, it failed both
of the jobs if one failed. This is expected since typically matrix
is used to run sames tests on multiple versions and such but our use
case is bit more than that.
Previously, we copied them to /tmp and from there we specified those
assets we copied in circleci config in presist_to_workspace step.
Copying it to a directory allows us to get rid of list in circleci
config and GitHub Actions's upload artifact (their version of
presist to workspace) doesn't allow us to specify indivivual files
so only is this cleaner but required.
The stream schema is used in two locations so move it to the
components section. Also the `is_default` key returned by `/streams`
is not returned by `/events`. So handle it separately.
The `Messages` schema present in `#/components/schemas` was a
combination of all keys possible in any message object used in Zulip.
Edit it so that the original `Messages` contains just the keys present
in the `message` event. Also make another schema `GetMessages` which
adds a few other keys which are received when using `GET /messages`.
The message object in the `/zulip-outgoing-webhook` has also been
modified and corrected.
The `subscriptions` has use in multiple endpoints and hence instead
of redefining it at every point move it to the the components section
for easier reuse.
We also have the caller pass in the property name for an
additional sanity check.
Note that we don't yet handle the possibility of extra_data;
that will be a subsequent commit.
Also, the stream_id fields aren't in Realm.property_types,
so we specify their types in the checker.
This a pretty big commit, but I really wanted it
to be atomic.
All realm_user/update events look the same from
the top:
_check_realm_user_update = check_events_dict(
required_keys=[
("type", equals("realm_user")),
("op", equals("update")),
("person", _check_realm_user_person),
]
)
And then we have a bunch of fields for person that
are optional, and we usually only send user_id plus
one other field, with the exception of avatar-related
events:
_check_realm_user_person = check_dict_only(
required_keys=[
# vertical formatting
("user_id", check_int),
],
optional_keys=[
("avatar_source", check_string),
("avatar_url", check_none_or(check_string)),
("avatar_url_medium", check_none_or(check_string)),
("avatar_version", check_int),
("bot_owner_id", check_int),
("custom_profile_field", _check_custom_profile_field),
("delivery_email", check_string),
("full_name", check_string),
("role", check_int_in(UserProfile.ROLE_TYPES)),
("email", check_string),
("user_id", check_int),
("timezone", check_string),
],
)
I would start the code review by just skimming the changes
to event_schema.py, to get the big picture of the complexity
here. Basically the schema is just the combined superset of
all the individual schemas that we remove from test_events.
Then I would read test_events.py.
The simplest diffs are basically of this form:
- schema_checker = check_events_dict([
- ('type', equals('realm_user')),
- ('op', equals('update')),
- ('person', check_dict_only([
- ('role', check_int_in(UserProfile.ROLE_TYPES)),
- ('user_id', check_int),
- ])),
- ])
# ...
- schema_checker('events[0]', events[0])
+ check_realm_user_update('events[0]', events[0], {'role'})
Instead of a custom schema checker, we use the "superset"
schema checker, but then we pass in the set of fields that we
expect to be there. Note that 'user_id' is always there.
So most of the heavy lifting happens in this new function
in event_schema.py:
def check_realm_user_update(
var_name: str, event: Dict[str, Any], optional_fields: Set[str],
) -> None:
_check_realm_user_update(var_name, event)
keys = set(event["person"].keys()) - {"user_id"}
assert optional_fields == keys
But we still do some more custom checks in test_events.py.
custom profile fields: check keys of custom_profile_field
def test_custom_profile_field_data_events(self) -> None:
+ self.assertEqual(
+ events[0]['person']['custom_profile_field'].keys(),
+ {"id", "value", "rendered_value"}
+ )
+ check_realm_user_update('events[0]', events[0], {"custom_profile_field"})
+ self.assertEqual(
+ events[0]['person']['custom_profile_field'].keys(),
+ {"id", "value"}
+ )
avatar fields: check more specific types, since the superset
schema has check_none_or(check_string)
def test_change_avatar_fields(self) -> None:
+ check_realm_user_update('events[0]', events[0], avatar_fields)
+ assert isinstance(events[0]['person']['avatar_url'], str)
+ assert isinstance(events[0]['person']['avatar_url_medium'], str)
+ check_realm_user_update('events[0]', events[0], avatar_fields)
+ self.assertEqual(events[0]['person']['avatar_url'], None)
+ self.assertEqual(events[0]['person']['avatar_url_medium'], None)
Also note that avatar_fields is a set of four fields that
are set in event_schema.
full name: no extra work!
def test_change_full_name(self) -> None:
- schema_checker('events[0]', events[0])
+ check_realm_user_update('events[0]', events[0], {'full_name'})
test_change_user_delivery_email_email_address_visibilty_admins:
no extra work for delivery_email
check avatar fields more directly
roles (several examples) -- actually check the specific role
def test_change_realm_authentication_methods(self) -> None:
- schema_checker('events[0]', events[0])
+ check_realm_user_update('events[0]', events[0], {'role'})
+ self.assertEqual(events[0]['person']['role'], role)
bot_owner_id: no extra work!
- change_bot_owner_checker_user('events[1]', events[1])
+ check_realm_user_update('events[1]', events[1], {"bot_owner_id"})
- change_bot_owner_checker_user('events[1]', events[1])
+ check_realm_user_update('events[1]', events[1], {"bot_owner_id"})
- change_bot_owner_checker_user('events[1]', events[1])
+ check_realm_user_update('events[1]', events[1], {"bot_owner_id"})
timezone: no extra work!
- timezone_schema_checker('events[1]', events[1])
+ check_realm_user_update('events[1]', events[1], {"email", "timezone"})
The status_element parameter is optional, and the other caller in
stream_popover.js does not provide it. This fixes a regression in
commit e6a66063a9 (#15868).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
As of commit 87e72ac8e2 (#15267), we
need to be an owner for some of the tested functionality, not just an
administrator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 7c0fa3aefc (#15734) added sample
alert words to the test database, so the Casper test can no longer
assume its alert word is the only one.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is a prep commit which passes the `update_func` and `source`
data through an object. This will be helpful as there are plans
to pass furthur information to the function (i.e. whether we should
allow creating pills from streams and/or user-groups).
Obviously, this file will soon grow--this
was the easiest way to start without introducing
noise into other commits.
It will soon be structurally similar
to frontend_tests/node_tests/lib/events.js--I
have some ideas there. But this should also
help for things like API docs.
We add the ability to supply optional_keys,
and we don't mutate the list of required
keys that gets passed into us.
We also enforce that there is a "type"
field.
(We will use optional_keys soon.)
This change makes our handling of youtube-url previews consistent
with how we handle our inline images. This allows the previews to
render next to the paragraph that links to the youtube video.
Follow-up to PR #15773.
In particular importing gitter data leads to having accounts with these
noreply github emails. We generally only want users to have emails that
we can actually send messages to, so we'll keep the old behavior of
disallowing sign up with such an email address. However, if an account
of this type already exists, we should allow the user to have access to
it.
jQuery’s $(callback) already checks document.readyState to decide
whether to run the callback immediately (that’s like, jQuery’s entire
value proposition). We probably don’t need ready callbacks at all
anymore thanks to <script defer>, but that’s a larger change.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
$.fn.typeahead, on the other hand, returns the jQuery object back (not
the Typeahead object, which also happens to have a select method), so
this should be converted.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit rewrites the way addresses are collected. If
the header with the address is not an AddressHeader (for instance,
Delivered-To and Envelope-To), we take its string representation.
Fixes: #15864 ("Error in email_mirror - _UnstructuredHeader has no attribute addresses").
Commit a9ca5f603b (#15863) incorrectly
converted these. e.target is a DOM element, not a jQuery element;
likewise for the elem parameter of activate_element.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Listen to change events from the checkbox and pay attention to its
actual value, rather than simulating it by toggling booleans on click
events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Zulip converts :) to the 1F642 Unicode emoji and promotes the same emoji
in the popular section of the emoji picker.
Previously Zulip has labeled 1F642 as "slight smile". While that name
conforms to the Unicode standard (which describes the code point as
SLIGHTLY SMILING FACE), it didn't match our use case of the emoji.
If a user types :) or selects the first smile in the emoji picker they
probably mean to express a regular "smile" and not a "slight smile",
which raises the question why they are only smiling slightly.
This commit relabels 1F642 as 😄 and our previous 😄 263A as
:smiling_face:. Note that 263A looks different in our three supported
emoji sets, so it is not suited to be our "default smile".
This change does not require a migration since our emoji system stores
both unicode points and names and handles name changes transparently.
Previously, image upload widget delete text CSS class name was
`settings-page-upload-text`.
We can change the CSS class name to `image-upload-text`
so that the name can be more generic.
Previously, image upload widget delete text CSS class name was
`settings-page-delete-text`.
We can change the CSS class name to `image-delete-text`
so that the name can be more generic.
Previously, image upload widget delete button CSS class name was
`settings-page-delete-button`.
We can change the CSS class name to `image-delete-button`
so that the name can be more generic.
Previously, image upload widget CSS class name was
`avatar-icon-logo-settings` it is not relevant to the widget so
we can change the CSS class name to `image_upload_widget`
so that the name can be more generic.
These checkboxes will now be more consistent
in design as we have in other part of the UI.
e.preventDefault() is added inside the
stream_is_muted_clicked function will disable
the default checkbox and make sure click event
come from only <span> part of the checkbox.
ERROR_BOT setting is not None during testing, so running
test_report_error without making errors stream was causing exception.
This commit make a stream name errors thus removes exception and error
log spam caused by it in ./tools/test-backend output.
This commit verify that error logging while testing data export in
test_notify_realm_export_on_failure using assertLogs so that the logs
do not spam test output.
This commit tests if error logs are logged when an error occurs during
testing of check_send_webhook_fixture_message using assertlogs. Using
assertlogs ensure logs are not printed as spam in test output.
This commit verify warning logs while testing validate_api_key and
profile is incoming webhook but is_webhook is not set to True.
Verification is done using assertLogs so that logs does not cause spam
by printing in the test output.
This commit verify error logs printed during testing of log_and report
function using assertLogs without printing it in test output and hence
avoiding spam.
Previously the private_message_recipient input remained focused after
closing the composebox with Escape. On Firefox this resulted in it
gobbling up all further keyboard shortcuts until you clicked
somewhere. On Chromium this bug didn't occur because it automatically
blurs hidden inputs.
Introduced in 3a1bf04a56.
Fixes#15849.
This commit re-adds the integration for canarytokens.org, now separate
from the primary Thinkst integration.
Signed-off-by: David Wood <david@davidtw.co>
This commit fixes the Thinkst Canary integration which - based on the
schema in upstream documentation - incorrectly assumed that some fields
would always be sent, which meant that the integration would fail. In
addition, this commit adjusts support for canarytokens to only support
the canarytoken schema with Thinkst Canaries (not Thinkst's
canarytokens.org).
Signed-off-by: David Wood <david@davidtw.co>
On calling `loading.make_indicator` for the second
time or more no spinner is being displayed.
This bug can be viewed on visiting a `near: 1` narrow
and the spinner for the newer messages is displayed
only once (i.e. the first time it is rendered), while
the logo is displayed every time.
This happens because `loading.destroy_indicator` sets
the css of that container to display: "none". This can
be removed as we are emptying the container just above.
Introduced in 953d475274.
Zulip Desktop version 5.3.0 only supported adding custom certificates
inside the application. Starting in version 5.4.0, it also supports
reading from the system certificate store; in the next release (likely
5.5.0) the support for the internal store will be removed.
Document the change, and add explicit instructions on how to add
certificates into the system store on each of the operating systems.
Co-authored-by: Manav Mehta <tmanavmehta@gmail.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A few major themes here:
- We remove short_name from UserProfile
and add the appropriate migration.
- We remove short_name from various
cache-related lists of fields.
- We allow import tools to continue to
write short_name to their export files,
and then we simply ignore the field
at import time.
- We change functions like do_create_user,
create_user_profile, etc.
- We keep short_name in the /json/bots
API. (It actually gets turned into
an email.)
- We don't modify our LDAP code much
here.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.
An upcoming commit will eliminate it from the
database.
This fixes up some complex helpers that may
have had some value before f-strings come along,
but they mostly obscured the logic for
people reading the tests.
We still keep really simple helpers for the
common cases, but there are no optional
parameters for them.
One goal of this fix is to remove the
short_name concept, and we just explicitly
set senders everywhere we need them.
We also now have each test just explicitly set
its reaction_type.
For cases where we have custom message ids
or senders, we just inline the simple call
to api_post.
We generally want to avoid having two sibling test
suites depend on each other, unless there's a real
compelling reason to share code. (And if there is
code to share, we can usually promote it to either
test_helpers or ZulipTestCase, as I did here.)
This commit is also prep for the next commit, where
I try to simplify all of the helpers in EmojiReactionBase.
Especially now that we have f-strings, it is usually
better to just call api_post explicitly than to
obscure the mechanism with thin wrappers around
api_post. Our url schemes are pretty stable, so it's
unlikely that the helpers are actually gonna prevent
future busywork.
It's not clear to me why these passed mypy
before, given this:
def assert_realm_values(f: Callable[[Realm], Any], ...
But this is clearly more accurate.
This issue isn't something a system administrator needs to take action
on -- it's a likely minor logic bug around organization
administrators moving topics between streams.
As a result, it shouldn't send error emails to administrators.
It's unclear what the purpose of this logic was, but testing confirms
that the text color is as expected without this in the day theme (so
it's likely a relic of an old design) and removing it fixes the hover
text being overridden to white in the night theme.
This is a hacky fix to avoid spoiler content leaking in emails. The
general idea here is to tell people to open Zulip to view the actual
message in full.
We create a mini-markdown parser here that strips away the fence content
that has the 'spoiler' tag for the text emails.
Our handling of html emails is much better in comparison where we can
use lxml to parse the spoiler blocks.
We hide the spoiler content in browser/desktop notifications.
Note: its not worth adding zjquery tests for this bit of code because
the tests do not operate on the actual data and are likely to get stale
if we change the syntax for spoilers.
We include tests for the new implementation to avoid churning the
codebase too much so this can be easily reverted when we are able to
re-enable the feature.
This handler adds a neat little effect whereby hovering over the
clickable region to open the navbar triggers the search_icon hover
effect and is a neat little visual cue about what happens onClick.
The previous implementation was slightly messy because it fetched the
color and applied it via ".css(". This commit cleans it up by creating
and using the class "search_icon_hover_highlight" instead. We also
make the selectors more specific, ensuring they target children of
"#tab_bar", this was so because it was reasonable to expect someone to
define eg `search_closed` elsewhere and we wanted to prevent bugs when
that happened.
In 9046fc1032 we updated the navbar.html
file so that our css selectors did not override each other and cause
annoying problems.
Unfortunately this caused a regression in night mode where the
search_icon didn't have the correct hover effect.
This fixes the regression by adding the selectors.
The page_params.timezone feature is perhaps a misfeature, but
importantly it's not what is used to display the time in the message
feed (it's mainly used to show others your timezone).
Given that reality, we shouldn't use it for a feature whose whole
purpose is to display the time using the same timezone we use in the
message feed.
Fixes#15790.
We were not passing any arguments to needs_subscribe_warning
when testing the function itself.
This commit changes the code to pass the user-id and stream-id
to needs_subscribe_warning. We also remove the stubs for
get_by_user_id and is_user_subscribed and do these tests by
calling the original functions, because passing the arguments
(user-id and stream-id) only makes sense if we use original
functions for them rather than stubs.
Delete stored topic data in `recent_senders` and `recent_topics`
about the message's topics and re-render them. The process is similar
to topic editing. See `recent_senders.process_topic_edit` for
logical details.
The tests had a bunch of different ways to create
users; now we are consistent. (This is a bit of
a prep step, too, to allow us to easily clean
Hamlet's existing words before each test.)
The prior version clobbered all flags, which means
we had unrealistic values for is_private.
Now we only touch the unread flag, which
also means when we go next to create alert words,
those will now work.
In 9648e64d23, we added a clear outline
around focused link elements in popovers.
This was a good fix for popovers, but it's distracting for the
experience of clicking links in the sidebars and message feed.
We could certainly do better with the handling here, but using the raw
string that the user gave us is okayish for now.
Proper formatting of timestamps requires handling locales and timezones
of the receiver as well which is a larger project.
We now do something sensible for spoilers in notifications. A message
like:
```spoiler Luke's father is
Vader. Don't tell anyone else.
```
would be rendered as:
Luke's father is (...)
We have changed our all instances of list_render to use
simplebar and thus, we will now use simplebar container
to track scroll event for all the lists created by
list_render.
This fixes the bug of new subscribers not rendering on
scrolling at the end of subscriber list in stream settings
and similar bug in some other lists also.
This commit also removes scroll_util.get_list_scrolling_container
function as this is no longer used.
Fixes#15637.
This commit fixes the dropdown_list_widget to use simplebar for
scrolling.
It was not being used because data-simplebar should not be inserted
to the element being rerendered. This commit adds a new element
wrapping 'dropdown-list-body' which was being rerendered and
data-simplebar is added to that new element 'dropdown-list-wrapper'.
Also, there should always be a max-height property on data-simplebar
element and it is also added in this commit.
There is also a change to set margin of 10px only on the first div
element and which is direct children of organization-settings-parent
element. This is correct because we only want margin to add some space
between the heading of subsection and the first setting of that
subsection. Previously, the margin was being added to first div of all
the other child containers also and this was adding unnecessary margin
to the first div element of different simplebar containers.
We do not need to use list_render for displaying list of streams specific
notification settings, as this is not scrollable and we do not provide
option to sort or filter this list as well.
After this change, all our list_render instances will be using simplebar
and we can change the code accordingly to fix the behaviour of scrolling.
We remove the action column and show action buttons next to topic
after unread count (if present). This save us a lot of extra space
on small window sizes.
For a:focus Bootstrap sets the following rules:
outline: thin dotted #333;
outline: 5px auto -webkit-focus-ring-color;
Firefox does not know -webkit-focus-ring-color and falls back to the
previous rule, making the outline invisible in darkmode.
Chromium has a bug[1] that makes outline: auto invisible when focussing
elements programmatically (which we do for the up & down arrow keys).
[1]: https://bugs.chromium.org/p/chromium/issues/detail?id=1105822Fixes#15768.
jQuery's fadeOut() sets display: none using inline CSS.
This was overriden by .alert-notification since it used !important
to override the display: block set in loading.js. Removing the latter
allows us to remove the !important, and doesn't seem to break anything.
Fixes#15759.
Prior to PostgreSQL 12, the `recovery_target_timeline` setting is only
valid in a `recovery.conf` file, as that file has its own
configuration parser. As such, including it in `postgresql.conf`
results in an error, and PostgreSQL will fail to start.
Remove the setting, reverting bff3b540b1. This fixes PostgreSQL 9.5,
9.6, 10, and 11; while the setting is not an error in a PostgreSQL 12
configuration file, it is unnecessary since `latest` is the default.
7d4a370a57 attempted to move the replication check to on the
PostgreSQL hosts. While it updated the _check_ to assume it was
running and talking to a local PostgreSQL instance, the configuration
and installation for the check were not updated. As such, the check
ran on the nagios host for each DB host, and produced no output.
Start distributing the check to all apopdb hosts, and configure nagios
to use the SSH tunnel to get there.
OpenAPISpec is our main class for accessing OpenAPI objects and
has the capability of creating the OpenAPI objects only once, thus
saving time. Since the openapi-core request validator object is going
to be accessed for considerable time during testing, hence add it to
the class for faster testing.
We extract OptionalContent and RequiredContent since some endpoints
require it and other don't.
In an ideal world, we'd have a better way to express these small
variants.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/users/{user_id}` can match
with `users/me`. So change the position of`/user/{user_id}` after all
such possible matches to avoid errors.
See https://github.com/p1c2u/openapi-core/issues/226 for details.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/messages/{message_id}`
can match with `messages/matches_narrow`. So change the position of
of `message/{message_id}` after all such possible matches to avoid
errors.
See https://github.com/p1c2u/openapi-core/issues/226 for details.
This reverts commit 63643c9d9d.
As the commit mentions, it makes a UI change for legacy search which
has largely been considered a regression. We've been running with
this reverted in zulip.com essentially since it was first merged.
Apparently, our scrollbar logic crashed with an invalid URL fragment
(hash), which resulted in initialization not completing and thus the
logic failing.
In my view the root issue here is that we're not doing a good job of
catching JavaScript exceptions in portico pages.
Fixes#15706.
wal-g was used in `puppet/zulip` by env-wal-g, but only installed in
`puppet/zulip_ops`.
Merge all of the dependencies of doing backups using wal-g (wal-g
installation, the pg_backup_and_purge job, the nagios plugin that
verifies it happens) into a common base class in `puppet/zulip`, since
it is generally useful.
In 42f20e81be I fixed an edge case but
also accidentally made clicking on reactions open the compose box.
This commit adds back the e.stopPropagation(); and explicitly hides the
emoji picker popover, to address the inconsistency fixed in the previous
commit.
Previously, we were experiencing a bug that caused the left border of
the searchbox/tab_bar to disappear when the searchbox was opened. This
bug was a result of the following changes:
- 4cdd7aed2b accidentally added this line
as right: 2; instead of right: 2px;
- 46c966576d fixed this line to be
right: 2px; but caused the regression.
This commit fixes the bug by deleting this line.
If the push_notification for the UserMessage is already active,
we don't send any push notification to the user. This may
happen due to race conditions.
Added and fixed test cases affected by this.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
