The pyldap fork was merged back into python-ldap, and released as
python-ldap 3.0.0; `pyldap` is now just a wrapper package that depends
on python-ldap.
Fixes#8912.
This commit adds support for '+' and 'Z' for "zoom in" and '-' and 'z'
for "zoom out" shortcut keys in the lightbox image viewer for Pan and
Zoom.
Fixes: #8689.
Implement few optimizations for reading admin's bot dicts from database
for a constants number of requests:
- add models.get_user_profiles_by_ids() for reading bots profiles
by single query from database
- add models.get_services_for_bots() for reading services for bots
by single query from database
- add bot_config.get_bot_configs() for reading config data for bots
by single query from database
Fixes#8838
This fixes a bug where the endpoint for editing bot users would allow
an organization administrator to edit the full name of a bot user.
A combination of this an another recently fixed bug made it possible
for this process to set a `bot_owner` for a non-bot user; so we also
include a migration to fix that for any users that might have had our
model invariants corrupted in that way.
Apparently, since 1948cb6a89, we've been
sending requests by an administrator to change a user's name to the
/json/bots endpoint, which would end up changing the "bot owner" of
these objects to some random user.
We fix this by re-splitting the views code.
This was a user-reported bug and a very subtle and painful one
to track down.
Previously, if payload['push']['changes'][i]['closed'] was True,
we assumed that a branch was removed. Looking at whether `closed`
was set to True or not was our way to tell whether a push removed
a branch or not.
However, this is wrong! `closed` being set to True can also mean
that the pull request associated with the branch was approved but
the branch itself was not deleted. According to the BitBucket docs,
the correct way to see if a branch is deleted is to check if `new`
is null.
This bug was leading to KeyErrors about not being able to find
the `commits` key, which shouldn't happen anymore!
'processing_emojis' check is added in the 'import_uploads'
function, so that the emoji files present in the to be imported
data file can be uploaded.
The procedure of saving emoji files in slack importer is same as
saving attachments and avatars, and the import has the similar
procedure too.
Change 'get_user_data' function to a more general function
to get data from the slack api using legacy tokens.
Also, change the error handling as upon invalid token,
the response is 200, but the response has an error
field in it.
For eg. Go to the following link with invalid token:
https://slack.com/api/emoji.list?token=xoxp-249056023425
Remove allocation ID function from slack import script. All the IDs
count will start from 0. Hence the ID List returned
by the allocation function is of no use, and we remove its implementation.
(example: get_total_messages_and_attachments function is of no use anymore,
hence we remove it)
In importing avatars, we use the implementation where the 'avatar_path'
is seperately calculated using realm and user ID and then the content
of the path provided in the avatar's 'records.json' are copied to this
'avatar_path'.
Similary, here for the uploads, 's3_file_name' is seperately calculated
using the realm ID and uploaded file name and then the content of the
path provided in upload's 'records.json' are copied to this 's3_file_name'.
'recipient_field' is added as a bool variable in the function
'update_id_map' to update the recipient foreign keys.
Recipient Foreign Key is equal to the UserProfile ID, if the
type is 1, and the same is equal to Stream ID, if the type is 2.
Hence a check is added in the 'update_id_map' field for this.
All the objects with realm ID as the foreign keys need to
be remapped with updated with the allocated ID.
Also the ID of the realm object itself is updated with the allocated
ID.
The 'id_field' bool variable is added to the function just to check
if the field is the ID of that object, and not the foreign key relation.
For foreign key field names, a "_id" has to be added after the field name,
however we don't need that for the ID field of the object.
Fixes#8853.
In certain cases, the browser is not able to look up the message.
Include the recipient data for the message in the delete_message event,
so look up of those attributes by the browser isn't required.
Replace mark_message_as_read with process_read_messages_event as the
latter function is only correct for marking a message as having been
read by this browser.
This should eliminate the occasional problems that some installations
have had with RabbitMQ trying to load-balance requests between
127.0.0.1 and ::1 (the ipv6 version of localhost).
(It is a preliminary change for deduplication of org settings template.)
This adds org settings labels as a context to admin templates so that
they can be used as a context variables in admin templates.
The reason we did this in JS code because of translation issue when
passed (as a context in `partial` handlebars helper) directly within
template.
(It is a preliminary change for deduplication of org settings template.)
This is done because of some settings like organization-settings
has ids which match the pattern of having a prefix `id_`
before the property name.
For those settings which don't have any prefix, there will be no
effect.
Here obsolete `t` in the label is removed since we do
label translations in JS files.
This splits "Language and notifications" section into "Default user
settings" and "Notifications".
With this, we can easily add other default user settings in the
same place.
In our new system for updating realm settings, we don't need to create
separate functions to update the input elements for each feature.
Most of the work is done automatically by
`settings_org.sync_realm_settings`.
We are having a same code in `render_notifications_stream_ui`
and `render_signup_notifications_stream_ui` functions aside from
the HTML element. So this commit will remove the duplicate code in
`render_signup_notifications_stream_ui` and make use of
`render_notifications_stream_ui`.
Fixes#8886.
On IE11, ClipboardData isn't defined; one can instead access it with
`window.clipboardData`, but that doesn't support text/html, so this
code path couldn't do anything special anyway.
So we instead just let the default paste handler run on IE11.
Fixes#8850.
This reverts commit bcdd12773e.
We need to do some improvements in handling FetchStatus for initial
narrows before this will be safe to deploy in production.
Still some errors (e.g. "Hover over a message to replace the message's
timestamp with its message actions" is no longer correct), and a lot of
other improvements to be made, but just doing a quick fix of a few things I
noticed.
There are several ways we open help for keyboard shortcuts,
markdown help, and search operators.
- from the gear menu
- from the compose box
- from the search box
- hitting ? for keyboard help
- arrowing/clicking through the tabs
This just moves the relevant code into a module and changes a
bunch of one-line calls in various places.
This adds some helpers to avoid some duplication, and we also
now track the selected idx directly, since it's all under our
control.
The main addition is `select_tab`, which we now use for some
things that used to simulate clicks.
This PR solves some of the parity issues in the emoticon translation
logic. I was unable to find a way of matching only one of the
lookaround groups, so we still have some inconsistency (see
testcase). The approach of having another check while converting just
for this seemed like an inefficient way, so I've left that last change
as it is.
Now that we have support for displaying custom profile fields, this
adds administrator-level support for creating them.
Tweaked by tabbott to fix a few small bugs and clean up the commit message.
Fixes#1760.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value. It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
Since the test database is in part controlled by the Zulip settings
files for testing, these settings files should be included in the list
of files that require populate_db to be rerun.
This issue was found due to changes to internal bots.
This was causing a rather confusing test flake in
test_stream_error_pm_to_bot_owner. What was happening was that if
this test (which used that code path) ran within 5 minutes of the
populate_db run, it would fail.
This initializes the bot_creation_policy_values after the page
is loaded.
Previously we initialize these values in `settings.js` when settings page
is loaded at least once, so if we open two tabs, one(1) in which we
haven't opened the settings page yet and if in another tab (2) we
update the `bot_creation_policy` value, then because of the event
which calls `settings_bots.update_bot_permissions_ui` causes exception
in (1) because `bot_creation_policy_values` isn't initialized yet.
Fixes: #8852.
This fixes a minor bug in which the value of `input` element of
`realm_waiting_period_threshold` don't get updated because in
`set_create_stream_permission_dropdwon` we don't change the
value of the input.
So, this minor refactor handles this more carefully.
This moves `sync_realm_settings` function out of `_setup` so that
we can call `settings_org.sync_realm_settings` without opening
settings page at least once.
This also fixes a minor bug in which if we have two tabs opened and
in one we haven't opened settings page at least once and in
another we change an org setting, then we get an exception in
the former tab because of the event as `sync_realm_settings`
isn't defined yet.
Changing the selected value of dropdown by `val` has more advantage
over `attr` especially in the case when dropdown value is chnaged
multiple times like due to events and discarding changes.
Though it is applicable to other elements but in this commit it is just
used for `id_realm_create_stream_permission` dropdown.
There are many question about this at stackoverflow:
https://stackoverflow.com/a/22093618/7418550https://stackoverflow.com/a/4837162/7418550
Since we don't have any string needed to be translated in
property_types we can move it outside `_setup` function to
get to the safe side if they needed to get accessed before
`_setup` is called.
Usually, to debug a small change, you have to remove some tests from JSON
because of lack of support for comments in JSON. This commit allows to
ignore some tests by setting `"ignore" : true` in the bugdown fixtures.
Also, since this is only for while developing, the complete test suite will
throw an error if we leave an 'ignored' test in a commit.
This is a preliminary commit which includes settings' (checkboxes)
label as a context for rendering template.
The reason we did this in JS code because of translation issue when
passed (as a context in `partial` handlebars helper) directly within
template.
Currently this is done for notifications' settings.
(There will be no UI change)
This is basically a simple fix, where we consistently set
`flags` to an empty array when we pass it around. The history
here is that we had kind of a nasty bug from setting it to
`None`, which only showed up in the somewhat obscure circumstance
of somebody subscribing to all stream events in our API.
Fixes#7921
This is the analog of 7b2c9223e7 for the
emoji cache; the only difference is that the existing code was working
correctly. It's still worth changing for improved robustness.
We saw issues with /srv/zulip_npm_cache being cleaned incorrectly by
this tool in production (more correctly, we noticed broken symlinks to
those directories, even from the current deployment). Print-debugging
showed that indeed older deployments were being ignored, because the
logic for `get_caches_in_use` was totally broken (this was sorta
masked because we also keep the last week's deployments).
The specific bug here turned out to be that we weren't passing the
`production` argument to generate_sha1sum_node_modules, but the
broader problem is that this logic isn't robust to changes in the
hashing algorithm.
Fix this by replacing the broken logic for trying to compute the
correct hash for that deployment with just checking the symlink inside
the deployment to let it self-report.
We can't easily do this same change for clean-venv-cache, because we
use multiple virtualenvs there. But a similar change could be useful
for the emoji cache as well.
Fixes#8116.
Note from tabbott: This is a somewhat surprising feature to be adding
this late in Zulip's development, but the model we've had of what
narrows are easy to access via clicking around has meant that it was
fairly difficult to get into a narrow that didn't include the very
latest messages in that narrow.
Fixes#3465.
If individual messages arrive before we get the latest
messages from the server, they can create gaps in rendering,
and would often be offscreen anyway, so we just ignore them.
This wraps each subsection in notification settings in a parent div.
This is done just to make the code more readable and clean.
There will be no UI change.
Also switches the default behaviour of the code to not translate the
emoticons. Earlier, the code was testing-aware, and used to translate
when there was no user profile data available(assuming that as a testing
environment).
The main testing of the translate emoticons code is in the
node_tests/emoji.js file. This code just checks if the setting
to enable/disable the emoticon translation is being honored.
Earlier, we used to convert all occurrences of an emoticon on the
frontend. That behavior has been altered to do conversions only
when the emoticon has some terminal symbols around them, and not
any alphabet or number. Also adds tests for emoji conversions for
the above logic.
Fixes#8585.
With this we have the same way to save changes done in org profile
subsection, i.e. show "Save" button beside header of subsection,
add "Discard changes" button for org profile subsection and
show "Save" and "Discard" button only when needed.
Also, there is so much code which become obsolete which is removed
in this commit.
Based on extensive manual testing with print-debugg (the exact
situation here was highly reproducible), in the absence of this line
here or slightly above here, Chrome 64 will consistently trigger an
extra scroll-forward-by-12000-pixels size downward scrolling event
immediately after it finishes rendering the 5th batch of ~100 messages
one gets from hitting the End key in `near:1` narrows.
I don't understand clearly why this change would protect against such
a Chrome bug, but my best guess is that Chrome was doing some sort of
incorrect optimization, and querying the scrollTop was forcing it to
come to a clear conclusion about the scrolling position before
appending more content.
But runs with the scrollTop() line not present in that function show a
scrollTop of around 25K in `append()` just before the call to
`render()`, and 37K at the end; while runs with this scrollTop line
always show 25K both before and after, so it does seem to work.
This number is way too high, because of a recent regression. Adding a
test here lets us prevent similar regressions in the future and
provides an easy way to be sure if we've fixed the issue.
Previously, when unnarrowing, we were calling this on the wrong
selector (this was missed years ago when we refactored Zulip to use
divs rather than table rows in the main message feed).
Noticed while debugging #5312.
Previously, if you started out with a very small window, loaded the
Zulip webapp, and then resized the window to be larger, you'd get the
condensed "[More]" links on essentially every message. This failure
mode was most visible with multi-protocol Electron apps like Rambox
that would sloppily start with a tiny window and then resize it when
loading Zulip.
The Rambox experience was essential to our being able to track this
down; once we knew what was happening, the fix was simply to
re-compute the condense state on resize.
Commit message rewritten by tabbott to explain the debugging and
context involved here, since this was one of our longest-lived mystery
bugs.
Fixes#5312.
This is apparently installed by the perl package; I hadn't even known
it existed. We of course want to use the sha1sum command from
coreutils.
Fixes#8836.
This commit is similar to the prior commit, in that we are
more disciplined about setting up handlers. We set them up right
as the widgets get rendered, and the handlers only delegate
up to the container div (id="stream_creation").
We now wire up the handlers that correspond to elements in
the 'new_stream_users' template when we render that template,
rather than at startup time.
We also delegate the events only up to #people_to_add, rather
than all the way up to body/document.
Added support for passing a filename without `.js` suffix.
This then fixed the issue of no complaints for invalid test
files. Now, throws an error for invalid test files.
Fixes#8579.
This cleans repeating code in error callback in settings.
We made a generic function in `ui_report.js` which require two
arguments `xhr` and `btn`; we preferred `btn` over `row` as argument
because a row may have more than one buttons.
Fixes: #8788.
This tests the initialize() function for now.
It goes deep on this:
* uses "mostly real" message lists
* asserts on fetch parameters
It stubs out many modules that aren't really central to
the logic of fetching. In particular, when messages are
processed, we notify things like the buddy list that messages
have been added.
An integration with RSS can be much easier to set up with Zapier.
Since the current RSS integration predates Zapier, we should at
least mention that there is an easier way to set it up via
Zapier!
This replaces the previous logic of triggering change() event.
(Also, when we trigger the change() event whole path for detecting
changes in a subsection are triggered which isn't good.)
This makes use of `settings_ui.disable_sub_setting_onchange` for
handling dependent `id_realm_disallow_disposable_email_addresses`
checkbox when `id_realm_restricted_to_domain` is changed.
On discarding changes made for `realm_create_stream_permission` always
"by_admin_user_with_custom_time" get selected because
`create_stream_by_admins_only` isn't a valid page_param.
This reverts success callback extension for `do_settings_change` function
because it seems it is better to make the request directly rather
than calling `do_settings_change`.
And hence `error` callback extension is also removed for the same
reason, but a error_continuation is added to do additional tasks when
errors happened.
For forms that are built early in setting up the settings panel,
we don't want to attach multiple submit handlers every time we
go into the gear menu, so we use "off" to clear any old handlers.
We also attach handlers directly to the form, instead of
using delegation up to the container div.
We only have one possible email hint, so there's no reason
to create one for each stream row, especially since we don't
clean them out when we close stream settings.
webhook-errors.log file is cluttered with Stream.DoesNotExist
errors, which hides the errors that we actually need to see. So,
since check_message already sends the bot_owner a PM if the webhook
bot tries to send a message to a non-existent stream, we can ignore
such exceptions.
We don't have any consumers for this event after removing
some obsolete code related to subscribe buttons.
Handling this event reliably consumed about 75% of the time
spent in _post_process_dom_messages, and maybe a percentage
point or two of overall rendering, so this will be a minor
speedup.
In 18e43895ff we replaced
stream subscribe buttons with stream links. The new feature
has been well tested and well received for over a year now,
so it's safe to remove the older feature at this point.
Older sites will have super old messages that still have the
rendered markup; this commit does not attempt to address those
situations. Most likely, clicking on an old button in the old
message will either do nothing or look like a message reply.
This cuts out about 11 calls to `git describe`. In a nice fast LXC
container following our instructions for development on a Linux host,
this might save "only" about 1.5s; in a dev environment on a Windows
host, the savings have been clocked at 49s, presumably due to an
extremely slow filesystem in the VM.
The tests weren't doing much with this codepath as they were, and
there isn't a lot of value to be gained by testing it anyway; it's
totally non-critical and rarely changes.
[Commit message rewritten by greg.]
This option "shows errors for missing return statements on some
execution paths." according to the mypy docs.
It doesn't currently bring up anything, after cleaning the cache.
Also, we remove unnecessary tips regarding modification settings.
Remove 'Only organization administrators can edit these settings'.
Remove 'Anyone in this organization can add user groups'.
Add function in user-groups.py for getting member ids
for a group.
Update view to enforce checks for modifying user-groups.
Only admins and user group members can modify user-groups.
This feature isn't really ready yet -- the relevance isn't good, so
the emails aren't a great experience. More work needed; pending that,
just don't send them.
There's already a per-realm setting, which doesn't have a control in
the org settings UI but does suppress it in the per-user settings UI.
Piggyback on that to suppress that UI control when the feature is
disabled at the server level too.
Also cut a comment that hasn't really made sense since the logic was
changed months ago -- the comment originally explained why we sent
digests on Tuesday, Wednesday, and Thursday, and doesn't correspond to
why we dialled back to weekly on Tuesdays.
The digest emails have little in common with the email mirror, beyond
that they both involve email. Give their tests their own file, with a
corresponding name, so it's easy to find this code's tests.
We filter out hidden comments out of Issue descriptions but this
breaks when description is null (which is unusual). So this commit
just checks to see if the description is None and if so, not to
filter anything out.
If an Android token has been used to connect a given device with
multiple Zulip servers, and then is expired, we would 500 in trying to
remove the Zulip-side registration for it, because the code assumed
there was only one such registration. If a token is no longer valid,
it's invalid for all servers, so the correct fix is to just remove them all.
Before this commit when we press enter in the text-input field the
first subsection get saved because a click event is triggered for
the first save button (of first subsection) irrespective of the
location of text input field in the whole form which is expected
as a default behavior of the forms.
The simple fix is to make the button of type "button" and override
action of "enter" in an input field.
More info at https://stackoverflow.com/a/12914700/7418550 and
https://stackoverflow.com/a/7060762/7418550
This makes each subsection of org-permissions independent from the
perspective of saving changes.
All the behavior we have for org-settings are also ultimately
reflected here as well like individual "Save" button for each
subsection, "Discard" button for discarding changes done in a
subsection and appearance of this button only when required.
In the new function, property values are handled more carefully i.e.
we first check whether property_name refers to a property values
which we can't get from the value of input elements like we have in org
permissions section for properties like realm_add_emoji_by_admins_only
where we have used dropdown and hence we get a string value but we
expect a true/false (check/uncheck) value.
Also, it is better to trigger a `change` event after changing a value
because we have properties which are dependent on the values of another
settings. Previously we trigger `click` event for checkboxes but
there exist some settings other than checkboxes like dropdown
for realm_create_stream_permission where realm_waiting_period_threshold
only appears after selecting custom values for threshold value from
dropdown.
Applies the logic to allow community members to edit topics
of others' messages if this setting is True. Otherwise,
only administrators can update the topic of others' messages.
This logic includes a 24-hour time limit for community topic editing.
This commit asserts that parse_user_agent never returns None. The
RegEx will match any string, so that `match` is never None. This
brings test coverage of lib/user_agent.py to 100%. Changes were also
made in test/test_decorators.py and views/compatibility.py to reflect
that parse_user_agent cannot return None.
Improves: #7089.
Fixes: #8779.
In this refactor property values are handled more carefully i.e.
we first check whether property_name refers to a property values
which we can't get from the input elements like we do have in org
permissions section for properties like realm_add_emoji_by_admins_only.
Small refactor in property_value_element_refers is to prevent many
return statements on further addition of property names.
This changes failed status element to use class
`.admin-realm-failed-change-status` rather than id so that we can use
the same code in `save_organization_settings()` in future to refer to
failed-status element of that section.
To keep click handler for "save" button clean, we extracted the
the `if` statements where we complete request data for certain fields
which aren't mentioned in `property_types`.
We restructured template to make org-permissions-template have more
separated subsections and so that we can easily apply event handlers
and selectors to their corresponding subsections (there will
be no change in UI).
(This is similar to 8b54b08)
This is nicer than the "For manual testing ..." comment. :-)
Also as a proper setting we can have it control some logging I
added locally while testing my recent changes to pika logging.
Details in comment. Together with a few previous commits, this should
completely eliminate sending error mail to admins when the RabbitMQ
server is simply restarted and comes back up normally.
There were two instances of `ensure_stream` being called and assigned to
a variable with the variable not being used elsewhere. pyflakes picked
up on this (where it didn't in the previous version likely due to tuple
unpacking), so the the variable assignment has been replaced with a call
to `ensure_stream`.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.
The changes weren't significant enough to add any tests or do any
additional manual testing.
The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.
Fixes: #2088.
To ensure that we have some basic data for custom profile settings,
in the `populate_db` data set, remove `options['test_suite']` check
for adding intial custom profile data.
This change mostly de-duplicates code, but it also prevents
some unnecessary backfills if you're in the home view and
happen to scroll back while the idle loop is right in the middle
of a backfill fetch, or if the idle loop has taken you back
to the oldest message already. This is a consequence of
do_backfill() calling msg_list.fetch_status functions.
Everything else should work pretty much the same, since
do_backfill() computes anchor essentially the same way we
originally doing it in the onIdle() function.
This slightly changes the order of operations of what we do in
the `cont` callback, so that we update fetch status earlier.
The order is somewhat arbitrary here, but we generally want to
update data structures first.
The "all" list and "home" list are basically kept in sync, and
the former is a superset of the latter.
Whenever we are backfilling, we want to backfill "all", so we use
it as the anchor, even though home_msg_list is the message list
we are actually rendering.
It's possible that this won't work with some versions of the
third-party backend, but tabbott has tested carefully that it does
work correctly with the Apache basic auth backend in our test
environment.
In this commit we start to support redirects to urls supplied as a
'next' param for the following two backends:
* GoogleOAuth2 based backend.
* GitHubAuthBackend.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.
Fixes: #6977.
Trying to replace the string on newer versions of vagrant fails with a
'RuntimeError: can't modify frozen String' error. On newer versions of Vagrant,
the server URL is correct already.
Update perfect-scrollbar to fix stutter space-scrolling in #8544. Also
reworked deprecated `element.perfectScrollbar` to `new
PerfectScrollbar(element)`. Lastly, updated provision version and
changed node module path to new path.
This also refactors perfect-scrollbar in help.js to work with updated
version of perfect-scrollbar. Because the update also changed
perfect-scrollbar's css selectors for all scrollbars in zulip, we
update those too.
Fixes#8544.
Algorithm for copying messages from serveral topics was changed:
- if there are selected messages from more than 1 recipient block
then the recipient bar headers are copied;
- If there are multiple messages from only one recipient block
then recipient bar header is not copied.
Fixes#7217.
Also adds a full suite of Casper tests for the copy-paste functionality.
The original implementation of this migration had a highly unfortunate
bug that would result in it deleting all reactions to realm emoji on
the server; we missed this in review, so essentially all historical
realm emoji reactions on chat.zulip.org were lost :(.
We both correct the problem, and also add logging of the deleted rows
that would help should anything be deleted erroneously.
In some environments, these exceptions happen regularly on upgrading
Zulip. They're harmless because we reconnect, so avoid complaining
noisily about them.
Because the base class's __init__ calls `_connect`, when we set the
value after that call has already returned, our new value only takes
effect if the first connection fails and we have to reconnect.
Make it take effect from the beginning.
This parameter isn't used anywhere. A good thing, because if it were,
the code would immediately raise an exception -- `self._on_open_cbs`
hasn't been initialized yet when we first call `_connect`, from the
base class's `__init__`.
So, just cut it. If we later need something like this, it's easy to
add a working version then.
This has a small stylistic change. The load_more function
is recursive, and we now early-exit when the recursion is
finished (i.e. when we've found the newest row).
We disabled the original "colorized HTML edit-history" feature way
back in 2013 in c51056ff8e.
That original feature involved showing what had been edited inline in
message bodies, so one could easily see what had been changed.
That old feature has since been replaced with the "view edit history"
menu option, and we're unlikely to ever want the old feature back.
So, we can just remove its code. There's a few supporting variables
that were created to help implement this; we can clean those up and
simplify the `update_message` code now that this feature is fully
removed.
This commit fixes currupted yarn lockfile, which generated this warning:
```
warning Lockfile has incorrect entry for "d3-queue@2". Ignoring it.
warning Lockfile has incorrect entry for "inherits@2". Ignoring it.
warning Lockfile has incorrect entry for "request@2". Ignoring it.
warning Lockfile has incorrect entry for "rimraf@2". Ignoring it.
```
This will reset the changes done in a subsection to the current ones
in page_params.
It will only appear when there are some user-provided changes to be reset.
With this "Save" button is only shown when there are changes in a
subsection. This means if we changed a setting and reverted it back to
original ones, then, "Save" button will get disappear. Hence, we're shown
"Save" when there are some property changes to send to the server.
This makes each subsection(like "Message feed") independent of changes
done in any other subsection and the save button of each subsection
saves the changes done in that subsection only.
If another user subscribed to or unsubscribed from a stream while the
current user was not subscribed, we previously would attempt to
rerender the subscriber counts for that stream, even though they
weren't rendered at all in the first place; this would trigger
blueslip errors from the expectOne() check in this function.
Fixes#8720.
In stream settings, if user add subscriber to unsubscribed public
stream from `Add` input widget it gives lots of blueslip warnings,
cause user isn't subscribed to public stream.
Fix this by changing condition to `sub.can_access_subscriber` from
`sub.subscribed` in blueslip warning, cause user can access
subscribers in such cases even if not subscribed to stream.
Tweaked by tabbott to make the node tests pass.
This checks for whether local storage is enabled and if it
is not enabled, then it will skip checking the value of
"dontAskForNotifications" key in the local storage.
This should eliminate an occasional JavaScript traceback we were
seeing in production.
Fixes#8721.
For a personal build, the teamcity webhook still sends a private
message using check_send_private_message since a personal build
should never trigger a public notification.
For a non-personal build, check_send_webhook_message is used,
which can either send a PM or a stream message based on whether
a stream is specified in the webhook URL or not.
We now only give users two options, to specify a stream and receive
public notifications for their goals, or to leave it out and receive
PMs and thus, keep their goals private. This simplifies the docs!
The original code featured a broken selector for .message-count. Instead
of using the right selector, the color selectors are removed altogether
for better contrast in dark mode.
This closes all the hotspots which aren't included in the event.hotspots.
This means we treat all the hotspots in event.hotspots as the hotspot
which need to be shown.
Fixes: #8690.
This applies only on a server open for anyone to create a realm.
Moreover, if the server admins have granted any given realm a
max_invites greater than the default, that realm is exempt too.
This makes this value much easier for a server admin to change than it
was when embedded directly in the code. (Note this entire mechanism
already only applies on a server open for anyone to create a realm.)
Doing this also means getting the default out of the database.
Instead, we make the column nullable, and when it's NULL in the
database, treat that as whatever the current default is. This better
matches anyway the likely model where there are a few realms with
specially-set values, and everything else should be treated uniformly.
The migration contains a `RenameField` step, which sounds scary
operationally -- but it really does mean just the *field*, in
the model within the Python code. The underlying column's name
doesn't change.
This fixes an unpleasant regression in
f5edeb01ae, where we stopped correctly
filtering users who have an open browser session that's idle. These
users are tagged as "UserPresence.IDLE" with an current timestamp in
the database, and should be treated as idle for presence purposes.
As a result, if you had an open Zulip browser session, you incorrectly
wouldn't get missed-message emails for PMs and mentions before this fix.
This fixes a regression in 93678e89cd
and a4979410f9, where the webhooks using
authenticated_rest_api_view were migrated to a new model that didn't
include setting a custom Client string for the webhook.
When restoring these webhooks' client strings, we also fix places
where the client string was not capitalized the same was as the
product's name.
This commit migrates all of our webhooks to use
check_send_webhook_message, except the following:
beeminder: Rishi wanted to wait on this one.
teamcity: This one is slightly more work.
yo: This one is PM-only. I am still trying to decide whether we
should have a force_private argument or something in
check_send_webhook_message.
facebook: No point in migrating this, will be removed as part of
#8433.
slack: Slightly more work too with the `channel_to_topics` feature.
Warrants a longer discussion.
This changes the followup_day2 emails delay from one day later to two days
later if it is getting delivered on any working days(i.e. Mon - Fri).
For Thursday it is compromised to next day as it would be too late to
postponed to Monday and for Friday it should be Monday.
At last actually, emails should send one hour before the above calculated so
that user can catch them when they are dealing with these kinds of stuff.
Fixes: #7078.
These changes are in one commit, since the previous typing of check_url
does not match the centralized strict definition (object/Any vs Text),
actually already used elsewhere in validator.py, and also had a different
API.
check_url is updated here to match the API of the other check_* functions,
ie. val is an object (not Text) & returns Optional[str]. It also now checks
the value is text explicitly at run-time, which was only type-checked
previously. Tests are updated accordingly.
Currently, when other private stream subscriber add realm admin to
stream, new copy private stream is created in realm admin's streams.
Which resulted in error, cause there are two similar stream element
in stream settings.
If new subscriber is added to private stream, we first send them
stream `create` event, cause private stream are not visible until
user don't get subscribed at least once. But realm admins can now
always access private stream, so when realm admin is subscribed to
stream, realm admin get stream `create` event even if stream already
exist in on realm admin client side.
Fix this by extracting realm admins from stream `create` event on
`add` subscription operation and sending private stream `create`
event to all realm admins on stream creation operation.
Fixes#8695
Remove the functions call for updating stream settings UI in
frontend, cause we are already handling this in the subscription
add and remove event we get after successful operation.
We now use the found_oldest flag from the server to determine
whether we should load older messages.
This requires us to change a few places in the code where we
pass the whole payload around now instead of just the messages.
Actually, many of the "cont" callbacks don't even look at the
data, so this change wasn't as invasive as I might have
predicted.
These are the straightforward ones.
Note that there is a line in zerver.lib.test_classes.build_webhook_url
that lost test coverage. That's because most of our tests test using
stream messages so the webhook URLs being tested always have a query
parameter. So the line that accounts for there being no query
parameters never gets called, which is fine, but we should still
keep it.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
in the webhook URL, it uses that topic as the subject of the
stream message.
Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!
Fixes#8607.
We solved the problem the TODO raised by using a different type
annotation syntax, and I'm not sure whether that refactor would
actually improve the code.
The previous system would crash with some files (because for some
reason the comment count was 1 but there was no "initial comment") and
also the file comment and file name were sorta redundant.
The 'make_new_dir' bool value was used to create a new directory
every time True is passed. Now that avatars and uploads directory
are being created seperately, we don't need this anymore.
We now wait to start typing notifications until everything else
is initialized. This prevents traceback where things like compose
pills have not been initialized.
If an emoji that was deleted was the only realm emoji, or more
generally if all realm emoji were deleted, then we would just leave
the reaction unchanged, with an `emoji_code` that is now corrupt.
Instead, treat this case the same as if only this emoji was deleted
while others remain.
The domain name is being set in the helper function
'slack_workspace_to_realm', but it should be set in the main function
'do_convert_data', as we need it in other child functions of
'do_convert_data'.
This code was originally written when we were using the old South
system, and hasn't been used in a few years. It probably doesn't
work, and thus only serves to clutter the codebase.
Clicking the cancel button removes all the changes and the user
group returns back to the original state. Saved button is showed
once the changes are saved on blur.
Add do_not_blur func to not save changes when blur event's origin
is one of name/description/pill input from the current user group.
Changes in any of name/desc/members are saved together on blur from
any of the input field given do_not_blur is false.
This restructures the organization-settings-admin template into
different subsections. With this, we can easily limit the scope of
event handlers and selectors within their subsection to which
are applied.
There will be no change in UI because of this.
To populate data for requests limited to only selected input elements,
we have to extend `populate_data_for_request` for objects other
than `property_types`, e.g. now we can collect data only for user identity
by passing org_permissions.user_identity as changing_property_types.
This splits property types according to sections in
"Organization settings" viz. Organization settings, Organization
permissions, and Organization profile. Where org. settings and
org. permissions are further categorized according to the
subsections.
These changes are made to save changes for each subsection independently.
This extends `do_settings_change` function for parameters like
`success` and `error` functions which will replace the default ones,
for functions that need to be excuted in success callbacks and for
desired success message.
All these parameters are optional.
Many declarations were previously annotated with
Callable[..., HttpResponse]; this is equivalent to ViewFuncT, so here we
switch to it.
To enable this migration, the WrappedViewFuncT alias is removed; this is
equivalent to the simple & legible Callable[[ViewFuncT], ViewFuncT], so
for relatively no space change, a clearer return type is possible.
Originally was going to centralize this in zerver/lib/request.pyi, but this
file is not visible at run-time, being only a stub. The matching request.py
file seemed inappropriate, as it doesn't actually use ViewFuncT.
This prevents accidental multiple click requests due to which we get
some errors like "This field cannot be blank" (though we have successfully
added the desired filter).
Fixes: #8699.
Namely, annotate as best as possible, and add notes to indicate preference,
if QuerySet develops generic typing.
Note that the return values of functions with annotations changed in this
commit are used elsewhere as QuerySets, so the Sequence[T] approach used
for some functions in models.py is not applicable.
Other functions took the form of returning Sequence[T] when the QuerySet
functionality is unused beyond the function, with T being the objects
filtered for in the function body; this commit follows that practice for the
one remaining python2 comment-annotated function, completing the transition
of models.py to py3.5 function annotations.
A note is also added to another function regarding a need to return a
QuerySet, and ideally a QuerySet[T] in line with the other functions, as and
when QuerySet becomes annotated as a generic.
We now consistently set our query limits so that we get at
least `num_after` rows such that id > anchor. (Obviously, the
caveat is that if there aren't enough rows that fulfill the
query, we'll return the full set of rows, but that may be less
than `num_after`.) Likewise for `num_before`.
Before this change, we would sometimes return one too few rows
for narrow queries.
Now, we're still a bit broken, but in a more consistent way. If
we have a query that does not match the anchor row (which could
be true even for a non-narrow query), but which does match lots
of rows after the anchor, we'll return `num_after + 1` rows
on the right hand side, whether or not the query has narrow
parameters.
The off-by-one semantics here have probably been moot all along,
since our windows are approximate to begin with. If we set
num_after to 100, its just a rough performance optimization to
begin with, so it doesn't matter whether we return 99 or 101 rows,
as long as we set the anchor correctly on the subsequent query.
We will make the results more rigorous in a follow up commit.
We do not update user_group_name_dict when name or description is edited,
which may lead to errors if get_user_group_from_name is called, so to fix
that we update user_group_name_dict also.
Note that the "Save" button has no text in the Taiga webhook
setup UI. There is a small floppy disk symbol for saving which
is visible right beside the form fields. So I simply said,
"Save the form".
We start to force downloads for the attachment files. We do this
for all files except images or pdf's. We would like images or pdf's
to open up in browser itself.
Tweaked by tabbott for comment clarity and correctness.
This makes it convenient to mention a stream you're not subscribed to,
which can be useful for communicating about where a topic is
discussed, for example.
Fixes: #5757.
This commit switches our emoji infrastructure to use 256 color indexed
64px spritesheets. Earlier we were using non-indexed 32px spritesheets
which were blurry on high dpi displays. These indexed spritesheets not
only provide a crispier display but are also smaller in size.
This commit also removes the `emoji-datasource` package as a dependency
as all the data is now sourced from individual datasource packages.
Fixes: #7862.
The edited group iswas going to the bottom, it is not staying at its
position, so to fix this we sort by ids of the groups to maintain
proper relative order.
(Original commit also fixed another issue, but Tim replaced that with
the previous commit for a better solution).
Fixes#8692.
This fixes the user groups UI to follow the Zulip standard mechanisms
(using the appropriate server_events system to update all browsers
properly). It also, as a side effect, fixes#8692, since it
eliminates the weird behavior of trying to re-insert a user group
after reformatting it in the frontend.
Thanks to Tarun Kumar for preliminary work on this.
Even if realm admin can access unsubscribed private stream subscribers,
admin can't add subscriber to stream. Hide the option to add subs to
stream in UI.
I noticed that our nice feature to show JavaScript errors collected by
blueslip in view of users was hidden in the background if you had one
of our settings screens open.
Further investigation determined the same was true for our other
panels.js alerts (e.g. the one for get_events failing, etc.). This is
easily fixed by just raising the z-index of the alerts panels; I
tested and a variety of our possible alerts look good on top of stream
settings, so no need to do anything special here.
Fixes#8576.
This will allow realm admins to remove others from private stream to
which the realm administrator is not subscribed; this is important for
managing those streams, because previously nobody could remove users
from private streams that didn't have any realm administrators
subscribed.
This will allow realm admins to access subscribers of unsubscribed
private stream. This is a preparatory commit for letting realm admins
remove those users.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
This generic function isolates the before/after logic that really
is independent of Message and doesn't need to clutter up
`get_messages_backend`. Also, introducing a new namespace
reduces some shadowing/mutation with variables like `query`.
It's a pure code move, with some very minor renaming (e.g.
inner_msg_id_col -> id_col).
If anchor is 0, there is no sense doing a before_query.
Likewise, if anchor is `LARGER_THAN_MAX_MESSAGE_ID`, there is
no sense doing an after_query.
We introduce variables called `need_before_query` and
`need_after_query` to enforce those conditions.
This also adds some comments explaining the fallthrough case
where neither query makes sense.
If use_first_unread_anchor is set and we don't have any unread
messages, then our anchor is effectively "positive infinity" and
we can streamline queries.
In the past we'd have clauses like `message_id <= 999999999999999`
in the query that were harmless but crufty.
We want to say `if num_after > 0` when we expect num_after to be
a positive integer. We don't want any confusion that we will
execute the blocks for values of -7 or None.
On uploading new avatar, user avatar in settings page doesn't
get updated. Cause we have set `src` attribute of wrong html
element instead of image element in user settings page.
Fixes#8680.
This makes the textarea responsive by making the width 100% and
the max-width 500px so that it doesn't get *too* wide.
Please close#8511 when this is merged.
Fixes: #8504.
Also adds a custom rule to eslint. Since the recommended way of extending
eslint is to create plugins as standalone npm packages, the separate rule
is published as 'eslint-plugins-empty-returns'.
Fixes#8669.
We also delete a couple helper functions that were only used there.
This management command was primarily used before we had a UI for
creating outgoing webhook bots.
In theory, we should be able to delete this, since first, if there are
no users in the organization, we'll end up with an equivalent value
(an empty collection of users), and second, it shouldn't be possible
for an active Zulip realm to have 0 active users in it anyway.
But the way we construct the database query in query_for_ids is such
that it's necessary to avoid a 500.
Main exceptions are scripts/tools/puppet & tests.
Other current exclusions:
- api_test_helpers.py (avoid changing test code);
- cache.py, due to comments in file;
- models.py, due to failure on QuerySet[Message];
- stream_subscription.py, due to failure on QuerySet[Subscription];
- tornado/descriptors.py;
- views/streams.py, due to remaining FuncKwargPair issue;
- zthumbor/, since thumbor is in python2.
Tweaked by tabbott to partially document the stauts in comments.
Firefox stores the last state of the page in its back/forward cache
in memory and uses that for quickly rendering the page. Since our page's
last state was 'faded-out', the content wasn't visible when the browser
rendered the page from it's bfcache.
Fixes#7907.
If we're entering a topic narrow, and the compose topic is the same as
narrowed topic, then leave the compose box open.
This is important if you open compose in a topic narrow, click on a
narrow, and then use the back button to return to that original
narrow. Before this change, we'd close the compose box for no reason.
Fixes#6510.
Apparently, we did essentially all the work to support showing full
topic history to newly subscribed users from a data flow perspective,
but didn't actually enable this feature by having the topic history
endpoint grant access to historical topics. This fixes that gap.
I'm not altogether happy with how the code and tests read for this
feature; the code itself has more duplication than I'd like, and the
tests do too, but it works.
This commit refactors the bugdown to perform a lookup only on active
realm emojis. This was needed because once we migrate realm emojis
to be addressed by `id` rather than name, it will be costly to
perform a lookup on all the realm emojis.
We no longer accept URLs while creating emoji; so this management
command was probably left out while migrating realm emoji
infrastructure to upload backend.
We could fix this to work properly today, but the command was
originally written in a context when Zulip didn't have a UI for
managing realm emoji at all. Now that we do have such a UI, it
doesn't have a compelling use case, and work on migrating the realm
emoji schema demonstrates that this does have a maintenance cost.
So, we simply remove this command.
Previously, stream subscribers were not rendering correctly. Due to
following reasons:
- Subscribers list didn't get initialized
- Subscriber members template add subscriber-list-table DOM element
conditionally, which doesn't handle case if user have access to
subscribers later.
Fix this by cleaning subscriber member template to hide the
elements, instead of conditionally adding DOM elements and
initialize subscribers list even if user can't access subscribers.
Previously, when creating an invite-only stream, the subscriber counts
were not being rendered properly, in that the "create" event for the
stream had the user not yet subscribed (so can_add_subscribers was
false), and then the rerendering we did when the user susbcribed just
tried to update the number, not actually rerender the thing.
When user click on unsubscribe button, to unsubscribe from private stream
immediately remove unsubscribed stream from all subscribed stream list.
We don't remove public stream immediately on unsubscription, cause
user may want to undo/subscribe back to stream.
If user unsubscribe from private stream, then user can't
access unsubscribed stream members.
After subscription removal, immediately render subscription members
template in case of user can't access stream subscriber.
If user has disabled message content in missed email notifications,
we shouldn't send any informations about missed messages i.e. sender,
stream, message text, etc. to email servers.
We are already hiding this informations in email templates, but we
shoudln't expose any information about message content if user
has disabled this setting.
We now include whether the message was a private or group private
message; this is particularly important with the new setting to
disable including any message content in these emails (since in that
case, one doesn't know anything about the message types).
If new private stream is created by realm admin without realm admin
subscribed to it, then it doesn't automatically add created stream to
realm admin's stream list. We have to reload the browser to get newly
created stream in stream list. Cause private stream creation event is
only sent to the subscribed users to private stream, so even if realm
admin is acting user, they don't get creation event.
We should send private stream creation event to realm admin users along
with subscribed user to stream, as realm admins can access unsubscribed
private streams.
Tweaked by tabbott to fix various typos and clean up the code.
Currently, our stream creation do not add stream list to all
stream list. Only subscription-add event add stream list to
stream list. If public stream is created without acting user
subscribed to stream, then newly created stream is not listed
to all stream list.
On stream creation add sub to all stream list table.
Only allow realm admins to create stream without subscribing
themselves, restrict non realm admin users.
This new restriction is only for the UI; we don't actually block it at
the backend/API level (one could always create and then remove oneself
anyway).
Restrict users(even realm admins) from creating stream with zero
subscribers only in UI.
In backend, if subscribers are zero, we automatically subscribe
current user to stream.
Add div element to inform user whether stream is successfully
created or not. This will help to inform user in case if current
user is not subscribed to created stream.
This migrates what were effectively data update functions to be called
from the main stream_events handlers, instead of being called from the
view-update code in subs.js.
To re-render stream members on subscription add or remove event,
we are accessing undefined value "active_stream.stream_id" rather
than "active_stream.id", which resulted in falsey value and our
stream members were not updated unconditionally.
For public stream, always show stream preview link.
For private stream, only show stream preview link if user is currently
subscribed or previously subscribed to private stream.
We now attach a fetch_status to message lists, so that they
can track their fetch status individually. When you go
back in a narrow and get all the older messages, we turn
off future fetches.
The narrow.js code no longer needs to orchestrate anything
here. The "home" message list won't have as many redundant
fetches after this commit, because we don't need to reset
flags every time we do `narrow.deactivate`.
And then actual narrows get a new message list every time
you narrow, so their fetch status gets reset implicitly
as part of constructing the MessageList object.
We are going to remove message_fetch.reset_for_narrow() soon,
but its callers probably still want to hide any scrolling
indicators, and we just let them do that directly.
This helps us consolidate the fetching constants without
having to export them. It will also remove some
responsibility for narrow.js to track fetching state.
Till now, we had been storing realm emoji's name in emoji code field
in reactions' model. This commit migrates it to store realm emoji's id.
It is a part of effort to migrate realm emojis to be referenced by their
id and not by name.
This removes `create-stream-dropdown` class from the template and css
because it is causing irregularity in size of other permissions dropdown
because of `right-margin` and removing it makes all the dropdown
regularly sized.
Rewritten in significant part by tabbott to actually be correct.
One particularly nasty thing the original webhook integration did is
do `current_time = time.time()` at the top of the `view.py` function
-- that means that code ran at import time, not runtime.
Currently, we shows data-no-recipients placeholder text even if
there are recipients selected. It's not visible cause it's value
is override by data-some-recipients helper text in case if there
are recipients selected.
Show data-no-recipients placeholder text only when `input` element
is first child of `pill-container`, which means there are no `pills`
selected by user yet.
"incorrect" here means rejected by a bot's validate_config() method.
A common scenario for this is validating API keys before the bot is
created. If validate_config() fails, the bot will not be created.
The former "the-bot-is-being-created" indicator was useless,
since it only edited the value attribute of the "create-bot" button;
that attribute only sets the text of a button when it is initially created.
This commit replaces the old code with a spinning loading indicator,
like the one used for editing bots.
This effectively changes a condition from `=== 0` to `<= 0`,
but that probably doesn't matter (and would only help). More
importantly, it just makes the code more consistent.
This mostly moves code from ui.js.
We change the arguments to `message_fetch.load_more_messages()`
to be `opts` with callbacks for `show_loading` and `hide_loading`.
We also defer starting the scroll handler until `message_fetch.js`
has been initialized.
Adds realm_bot delete event. On bot ownership change, add event is
sent to the bot_owner(if not admin) and delete event to the
previous bot owner(if not admin). For admin, update event is sent.
if the test fails, the 'output_dir' would not be deleted and
hence it would give an error when we run the tests next time,
as 'do_convert_data' expects an empty 'output_dir'.
Also the unzipped data file should be removed if the test fails
at 'do_convert_data'.
To prevent specifying notifications individually for global
updates, automate handle_global_notification_updates using
settings_notifications.notification_settings where we have
the keys of all the types of notifications.
If we have bugs in our compose code, we don't want them to
cause other strange side effects during reload. (We recently
had a test deployment that was a bit buggy, and the reload
problem was tough to chase down due to the misleading
symptoms.)
@brockwhittaker wrote the original prototype for having
pills in the recipient box when users compose PMs (either
1:1 or huddle). The prototype was test deloyed on our
main realm for several weeks.
This commit includes all the original CSS and HTML from
the prototype.
After some things changed with the codebase after the initial
test deployment, I made the following changes:
* In prior commits I refactored out a module called
`user_pill.js` that implemented some common functions
against a more streamlined version of `input_pill.js`,
and this commit largely integrates with that.
* I made changes in a prior commit to handle Zephyr
semantics (emails don't get validated) and tested
this commit with zephyr.
* I fixed a reload bug by extracting code out to
`compose_pm_pill.js` and re-ordering some
calls to `initialize`.
There are still two flaws related to un-pill-ified text in the
input:
* We could be more aggressive about trying to pill-ify
emails when you blur or tab away.
* We only look at the pills when you send the message,
instead of complaining about the un-pill-ified text.
(Some folks may consider that a feature, but it's
probably surprising to others.)
The main point of this change is to streamline the core
code for input pills, and we use also modify user groups.
The main change to input_pill.js is that you now
configure a function called `create_item_from_text`, and
that can return an arbitrary object, and it just needs
a field called `display_value`.
Other changes:
* You now call `input.create(opts)` to create the
widget.
* There is no longer a cache, because we can
write smarter code in typeahead `source` functions
that exclude ids up front.
* There is no value/optinalKey complexity, because
the calling code can supply arbitrary objects and
do their own external data management on the pill
items.
* We eliminate `prependPill`.
* We eliminate `data`, `keys`, and `values`, and just
have `items`.
This split notification settings to separate settings for streams,
PM/mention and other notifications settings.
Previously we have to treat them combined, but with this refactor,
we can treat them separately for future changes.
Fixes#8587.
Until page gets load, table lists are empty. Which results in
showing empty-data-text("No data match to search query") in
setting page parallel to loading spinner.
Hide table-list and show loading spniner until
setting page and table-list gets load.
In org settings, loading spinners are not visible, currently
because their size is very small.
Fix this, by increasing width and height of spinners and
adding spinner text, to make spinners more visible.
Fixes#8502
The summary already has a qualifier that basically says it shouldn't
matter for most people -- making it simultaneously the most
complicated bullet there, and among the least likely to matter.
And in fact, this requirement shouldn't matter for *anyone* when first
experimenting with Zulip. If certbot won't work in a given admin's
environment, and the available ways to get a cert aren't convenient,
they can always let the installer generate a self-signed cert to get
going, and circle back to the issue later.
So, make that option clear in the main requirements text, and then
just cut the whole bullet from the summary.
This further reduces the wall of text on the install instructions.
Simultaneously it lightens up the pressure on this summary to be quite
so terse; expand a couple of items into multiple bullets (yet with
fewer words!) for greater readability.
Now down to just 4 steps!
This version tries to prioritize: just two items that we really want
all admins to look at even if they aren't already mentally committed
to running a big production service and reading all the docs.
Namely, the two required in order to really try out Zulip effectively
with one's colleagues.
The screenshots weren't doing much good without being embedded in the
text... and in fact, looking at them for I think the first time,
they're badly out of date with the app. So cut them.
We might add screenshots later, but on the other hand if we do a good
job with the forms themselves, they should be superfluous.
This further shortens the wall of text inside the instructions.
Note that thanks to embedded reST, we now have the power to provide
custom anchors at section headings! Which in particular means this
link won't break if we later tweak the wording of this heading.
This helps shorten the wall of text between the start of the
instructions and the end. Conversely, now that there are fewer
followup steps, the troubleshooting section at the end isn't so
far away to point.
The installation isn't really complete here, and wasn't even when this
was the only success case; the instructions we're giving are for the
next step in the installation.
These instructions don't say what to do in an actual use case for this
option, but decent instructions there will require having a concrete
use case in front of us and designing the flow for it. At this stage,
just say where we are in the normal flow, and an admin who's chosen to
go off that flow can figure out how they want to vary it from there.
This flips the experimental `--express` option to be the default.
We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.
The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words. I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.
Also tweak the "production" test suite to match.
Kind of unfortunate because the `sudo` interface for running a command
is objectively better -- a list of arguments, rather than a string to
be re-parsed by the shell. But some bare-bones machine images lack
`sudo`, so this makes things a bit more portable.
The messages were first being read and passed to the helper
functions channel wise.
This function makes a list of all the messages in the all the channels
beforehand which would be used to pass in the helper functions.
There's probably follow-up work to do here to eliminate these
completely, but this dramatically shrinks the ~1 minute race window
that was previously present between import and test function being
called.
Running this on additional machines would be redundant; additionally,
the FillState checker cron job runs only on cron systems, so this will
crash on other app frontends.
While this is a different system than I'd written up in #8004, I think
this is a better solution to the general problem of cron jobs to run
on just one server.
Fixes#8004.
The problem that prompted this PR is that I couldn't select the API
key of a bot in chrome to copy it, but I could on Firefox.
Using the `auto` proerty value to override the `user-select`
property of a parent does not seem to be compatible
with all browsers.
This commit:
* Restructures the doc to use a numbered-step format.
Note that there are no screenshots. I signed up for a
Fabric/Crashlyics account but you have to link an Android/iOS app
to even get to the settings panel, which seemed like too much work
just to get a screenshot.
However, the way we can verify (somewhat) the correctness of the
last step is that it is a paraphrase of the first paragraph of
Fabric's Webhook docs, which can be found here:
https://docs.fabric.io/apple/crashlytics/custom-web-hooks.html
This commit modifies the text to:
1. Removes unnecessary screenshots.
2. Use the numbered-style format.
3. I also removed the instructions for generating an access token.
I took a look at Dropbox's docs and you shouldn't need that
for a webhook setup. The whole point behind a webhook is that
one can get by without using OAuth.
4. Rearranges the instructions to only contain 4 steps. For
uncomplicated instructions, that seems to be the ideal number.
In `recipient-row` template, if conditions to add/hide/show edit
icon for message topic is incorrect.
In some cases, we only want to just hide the edit icon, but icon
should be in DOM, cause in future if organization settings are
changed we want to show edit icon in message row.
If user can edit topic of message, surely add edit icon element to
DOM regardless of user is allowed to edit or not. If user is
allowed to edit then show edit icon otherwise hide edit icon element.
`update_message_topic_editing_pencil` function update edit
icon for message topic in message row header.
- Remove erroneous `meta.loaded` check.
- Add proper check of message editing permission in realm
before updating message-topic-edit-icon in function.
- Update `always-visible-topic-edit` element in function
simultaneously.
Rename `toggle_allow_message_editing_pencil` to
`update_message_topic_editing_pencil` in `settings_org.js`.
As this function update the pencil icon for updating message
topic in message row header.
We fallback to using data-user-email attrib for mention messages
from the times when we use to expose user emails to frontend.
This is only required for cases where we are dealing with anything
that isn't rendered dynamically (like the messages that were already
sent and stored long time ago). Hopefully we won't be needing this
kind of fallback logic in more places so I am putting off efforts
to try to extract fallback logic for common use. Also this is like
this because it will probably be tricky to extract out a common
fallback logic in this case because of different situations involved.
Fixes: #8588.
This provides a slightly clearer interface, allowing us to remove the
unnecessary split of the code for the clone_file_input concept across
multiple modules (we now just clone it on-demand).
This makes it easier for users to tell how Zulip ended up with an
avatar for them without them uploading one: through the Gravatar
service used across many Wordpress blogs.
Fixes#8225.
This field has been unused by clients for some time, and isn't great
for our public archive feature plans (where we'll not want to be
including email addresses in messages).
In stream deactivation modal, make "stream_name" a template variable,
rather than patching stream name to modal header in javascript.
Add tests for deactivation stream modal.
This replaces the cumbersome system we had for giving users feedback
on settings state changes in the display settings UI.
We expect this new system to be what we will attempt to migrate other
settings widgets to match over the coming weeks and months. It also
provides the opportunity to significant refactor away a lot of the
code duplication in settings_display.js.
Thanks to Brock Whittaker for redoing the styling and improving the
code simplicity.
Fixes#7622.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.
Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.
Add tests for emoticon translation, a settings UI, and a /help/ page
as well.
Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.
Fixes#1768.
Private streams were not included in stream suggestions for default streams
in org settings.
Remove function, which exclude private streams from stream suggestions
for default streams.
These two classes are tricky to test, and nocoverage-ing them
allows us to mark queue_processors.py as fully covered. We
still want to cover these two workers at some point, but for
now, it's nice to enforce full coverage for any future changes
to queue_processors.py.
Fixes (sort of) #6542.
This sets up a new test class with a simple
test, mostly for increasing coverage. The class
should in the future be extended to properly
verify the handle_feedback() logic.
We already check in get_service_bot_events() if a bot is mentioned,
and then only pass on the call to the bot handler if it is. The
commit removes the additional check in the embedded bot queue
processor simply because it is impossible to obtain test coverage
for it (there is no meaningful way to trigger the content of the
if-clause, because there will never be a message reaching the bot
without @-mentioning it.
To alleviate the danger of a potential regression, the check is not
removed completely, but rather replaced by an assert statement.
This appends a space when text is inserted at the end of a message
using `compose_ui.insert_syntax_and_focus`. This is definitely what
users expect when using this feature.
Fixes: #8569.
Previously, when a user updated the config data of an
embedded bot, only the updated fields were dispatched
back to the client. Dispatching all config data fields
makes the client updating code less brittle.
This fixes the real-time sync for the user groups membership data
structure to work like these work in the rest of Zulip; now, edits
made in one browser are immediately and correctly reflected in other
browsers.
Webhook functions wrapped by the decorator:
@authenticated_api_view(is_webhook=True)
now log payloads that cause exceptions to webhook-errors.log.
Note that authenticated_api_view is only used by webhooks/github
and not anywhere else.
We have some settings which are inter-dependent. If one setting
is checked or unchecked, it's dependent-sub-setting get disabled
or enabled. i.e. If user unchecked setting allow-message-editing
then message-editing-time-limit setting should get disabled in UI.
Add generic function to change disable attribute of sub settings
on checked or unchecked event of main setting in `settings_ui.js`.
Now, the fixtures.json file:
* Has all keys sorted alphabetically.
* Has a space after every `:`.
The file was generated using json.dumps with the appropriate
formatting parameters.
Note that this is one of the few fixtures that isn't tested against
a running server. But it still makes sense to move it to fixtures.json so
that it is rendered with indenting and a space after every `:` by
the api_code_example extension.
It turns out the consistent `.prop()` approach I switched to a few
commits back didn't actually work. Instead, we establish consistency
by always using `.attr`, which does.
Usually, I'd go back and fix the older commits, but in this case it
feels not worth it.
Rather than having a toggle function that just flips the state, it's
more correct to have the state just be set to what it should be.
Also, we clean up the use of .attr() for a thing better accessed via
.prop().
We should probably rename the function at this point, but we save that
for a future commit.
During a slack import, we don't have medium-size avatars already
available in the export data set (and possibly also with a normal
import/export?). The medium size avatar can be created by the
'ensure_medium_avatar_image' function, which checks if the medium
image exists, and if it doesn't, it creates the image.
This commit was substantially edited by tabbott to get rid of an
undefined variable bug, avoid initializing the upload backend classes
in a loop, and add some TODO notes on things that could be improved
later.
slack avatar urls have the format:
'https://ca.slack-edge.com/<team_id>-<user_id>-<avatar_hash>-<size>'
For any url of this form, if the user hasn't uploaded an image,
Slack uses default gravatar, but we don't have a way of knowing if Slack
has used the uploaded image or the custom gravatar
eg: https://ca.slack-edge.com/T5YFFM2QY-U6006P1CN-gd41c3c33cbe-512.
Hence, avatar_source should be mapped to 'U'.
This should help prevent against bugs where we accidentally introduce
use of sudo somewhere in the production installer or upgrade code path
(these used to happen all the time), which doesn't work on production
systems that don't have sudo setup.
Apparently, a bug in my refactor in
5edbcb87fd meant that "King L" would end
up matching "King Hamlet", because we weren't requiring a match at the
start of the word for the last word of a multi-word query.
Thanks to Greg Price for the report.
When our code raises an exception and Django converts it to a 500
response (in django.core.handlers.exception.handle_uncaught_exception),
it attaches the request to the log record, and we use this in our
AdminNotifyHandler to include data like the user and the URL path
in the error email sent to admins.
On this line, when our code raises an exception but we've decided (in
`TagRequests`) to format any errors as JSON errors, we suppress the
exception so we have to generate the log record ourselves. Attach the
request here, just like Django does when we let it do the job.
This still isn't an awesome solution, in that there are lots of other
places where we call `logging.error` or `logging.exception` while
inside a request; this just covers one of them. This is one of the
most common, though, so it's a start.
This restructures this fairly complicated function to a much cleaner
implementation, with fewer unnecessary variables and a cleaner flow.
While we're at it, we document the function.
For "#word text" (and similar situations, like "@word text" and
":word text"), we should only show the autocomplete for entries
where word matches the full first word of something being
completed (and similarly for multi-word phrases).
Fixes#8279
models.py should only contain thin wrapper functions. Furthermore,
this move allows us to remove the circular imports. The two moved
functions are interdependent and are thus moved in one commit.
compose_action.respond_to_message and compose_action.start starts new
compose box so we should check whether we are already composing
or not. So, behavior, when we are composing, is that the user we want
to mention is added to the same compose-text without the changing stream
and topic name.
Use compose_ui.insert_syntax_and_focus() when we need to insert text
inline-ly followed by the focus to compose textarea because it does
this job more smartly(it take cares of spaces).
No need to have separate click handler for user_popovers and
message_info_popovers as the same user-id can be extracted similarly
from the target of both the click events.
Another refactor, `sender_info_popover` was confusing as it doesn't
fix into the context of rightbar user popovers so changed it to
`info_popover_actions` since that section of popovers contains popover
actions.
Revert c8f034e9a "queue: Remove missedmessage_email_senders code."
As the comment in the code says, it ensures a smooth upgrade path
from 1.7.x; we can delete it in master after 1.8.0 is released.
The removal commit was merged early due to a communication failure.
This didn't really make much sense - it presented a quickstart summary
of the instructions as an item in the list of differences from the
Ubuntu case, plus that quickstart only incorporated one of the two
actual differences so it didn't work anyway. Given the networking
stuff, an actual quickstart isn't in the cards, so just cut that.
Also fix some small things while we're here.
Previously, this function executed the same test as
test_bots.py/test_create_embedded_bot_with_incorrect_service_name().
Now, instead of testing to add an embedded bot with an incorrect service
name, we test messaging an embedded bot with an incorrect service
name.
If your cursor is in the middle of a word when you upload
an image, the code will now properly put spaces in the markdown
around the attachment link.
Fixes: #7212.
This works simimlar to the "n" key for next topics.
This commit does a few things:
* It wires up the hotkey to an existing function
that could change narrows.
* It adds documentation.
* It adds logic to make sure the compose box does
not open.
@showell helped a bit with the wording of comments here.
Fixes#4874
This requires updating one of the tests for the group_pm_with feature
in test_narrow to use the new style of tautology generated by SQLAlchemy.
Thanks to Sinwar for investigating this.
Fixes#8381.
We create a node unit test,
with 'muting' and 'stream_data' modules as dependencies,
to test the logic in notifications.message_is_notifiable.
Part of #2945
GitLab recently changed their event name for build notifications
from "Build Hook" to "Job Hook". Instead of just supporting the
latter, we should support both just in case people are running
older versions of GitLab.
The check for the channel ('general' and 'random') must be added before
'build_defaultstream' function is called and then the id is incremented.
Otherwise, the id appended at the end of second defaultstream object, which would be
greater than the total number of defaultstream objects would crash at
'defaultstream_id_list[defaultstream_id]' which is a paramater of 'build_defaultstream'.
Added tests to prevent the same.
This is necessary for mobile apps to do the right thing when only
RemoteUserBackend is enabled, namely, directly redirect to the
third-party SSO auth site as soon as the user enters the server URL
(no need to display a login form, since it'll be useless).
This makes it easier to work on features that depend on messages
having been sent in the past (E.g. the date parts of recipient bars).
The new feature only works with --threads=1; since with the ~100
default messages that populate_db generates, the multi-threaded
feature shouldn't have significant performance impact (and it would be
tricky to make increasing timestamps work with the multi-threaded
model), it's reasonable to just set the default number of threads to 1
for now and have this timestamp-spreading feature only supported with
--threads=1.
Fixes#8277.
At some point, GitLab decided to change the name of the event for
CI notifications from "Build Hook" to "Job Hook" and we started
running into errors in webhook-logger.log.
This commit:
* Removes the unnecessary screenshot. The UI is intuitive enough
and standalone instructions should suffice.
* Rearranges the instructions into 4 steps.
* Makes the wording more explicit.
This commit:
* Removes the unnecessary screenshot. The user should be able to
easily see the fields in question in this case.
* Wraps the text at 80 chars.
* Combines the instructions into 4 steps.
The docs for this can easily be combined into 4 steps. For
uncomplicated setups, 4 seems to be like a good number.
Again, I have no way of verifying the correctness of the instructions
here because Airbrake doesn't let you do anything till you specify
your credit card information, which I didn't want to.
This commit modifies the text to:
* Use number 1 for all steps and let Markdown take care of the rest.
* Removes the line that says this webhook is "experimental". It isn't
anymore.
This commit modifies the doc.md to:
* Use consistent language and style.
* Use the number 1 for all numbered steps and let Markdown take
care of the rest.
* Have detailed steps on how to get to the Integrations settings
instead of just linking to the page.
* Remove unnecessary screenshots.
This commit:
* Adds a missing step to the documentation.
* Replaces wording such as "Go to X" with "Click on X".
* Removes the unnecessary screenshots.
* Rearranges the doc to contain only 4 steps. For uncomplicated
setups, 4 seems to be the right number.
This commit:
* Removes the unnecessary screenshot.
* Reorders the instructions and combines them in to 4 steps.
* Improves the contents of the webhook-url-with-bot-email-indented.md
macro and makes it more consistent with create-bot-construct-url.md.
* Sets the recommended stream name to "commits", since that's what
the webhook function for Beanstalk expects in
zerver/webhooks/beanstalk/view.py. This allows us to use the
create-stream.md macro.
This got broken at some point when we moved around the context
processing logic for integrations/webhooks. Thankfully, the
context value for external_uri_scheme was only used in a couple of
our less popular integration docs. It should render perfectly now.
* Remove unnecessary screenshot. It doesn't help very much in this
case.
* Update text to instruct users to not leave the `Title` field
empty (it cannot be blank).
* Replace wording such as `Go to Settings` with `Click on Settings`.
* Combine the "fill out the form" and "click 'Save'" steps.
* Replace "Choose X on the left-hand side" with "Choose X".
* Replace "Remember to check the X" with "Check the X".
Creating the very first organization administrator user and
subscribing them to streams before any messages were sent resulted in
RealmAuditLog entries being created with a `event_last_message_id` of
None, because that's the maximum ID in the empty set.
We correct this by fixing the incorrectly created RealmAuditLog
entries, both for new servers and also fixing old broken entries on
existing servers.
This fixes an issue where if a user setup a Zulip server with just the
organization administrator, and then forgot about it (so that the
initial user became soft-deactivated), trying to sign in 3 weeks later
would throw an exception.
This fixes the issue reported here:
https://chat.zulip.org/#narrow/stream/9-issues/subject/500.20error.20on.20login/near/511981
The other emojiset types are things like "Google", and "Twitter",
which don't need to be translated. And the string is already tagged
for translation as a variable where it is used here:
static/templates/settings/display-settings.handlebars#L87
Fixes#7970.
psycopg2 package contains both binary and source
currently.The binary would soon be only available
in psycopg2-binary. So currently installing psycopg2
will show a warning about the future migration of binary.
To avoid the warning we should only install the source.
https://www.postgresql.org/message-id/CA%2Bmi_8bd6kJHL
TGkuyHSnqcgDrJ1uHgQWvXCKQFD3tPQBUa2Bw%40mail.gmail.com
Hotkey reminders in user popovers( i.e. `@` and `R`) should be present
only for sender info popovers, hence for user-mention and user-presence
popovers it will show no such reminders.
Fixes: #8313.
We were using underscore as a delimiter in `emoji_id` in emoji picker
but since `emoji_type` also contains underscores, it seems wrong to
use underscore as a delimiter. This commit switches `emoji_id` to use
comma as a delimiter instead of underscore.
In active user section, while editing user's full name,
if we press enter it reloads the whole page, rather than
submitting the form.
Change button type of user-name-form to submit.
In active user settings, when user click on user-name-form to edit
user's full name, we don't set any default value for full-name
input field, which results in garbage or falsy value in input field.
Set true value of selected user's full name in input field by default
in user-name-form.
Currently, an error message on active user setting is showing at
the row of edited user profile in user-profiles-list-table.
Instead, show error message on top of setting page.
Tweaked by tabbott to also remove the HTML element.
In active users setting, an errors occurred on editing user
profiles, are not reported correctly because there is missing
argument in call of ui_report.error() function.
Fix the issue by passing proper arguments in ui_report.error()
function call.
The total number of stream objects are allocated to
total_users. They should be allocated to the total_channels.
This passed the tests as the total number of users in the test
where greater than the total number of channels.
Previously, we used to raise an exception if the direct dev login code
path was attempted when:
* we were running under production environment.
* dev. login was not enabled.
Now we redirect to an error page and give an explanatory message to the
user.
Fixes#8249.
The Markdown extension that lives inside
zerver/lib/bugdown/api_code_example.py previously used ujson.
ujson's `dumps` function doesn't accept a `separators` argument,
which means we have no control over how the JSON is pretty-printed.
This resulted in JSON fixtures with no spaces after the colon, which
looks unnecessarily convoluted.
So now, we use the built-in `json` module to get around this.
For further reading, this issue
<https://github.com/esnme/ultrajson/issues/82> opened on ujson's
repo explains why they are reluctant to support such formatting
due to performance considerations.
This commit adds a test for the sample fixture for when an invalid
stream name is passed to a query that expects a valid stream name
as an argument. This is the case with almost all of our queries
documented under the sidebar heading "Streams".
EDIT: Actually, I was wrong. This payload is highly specific to
get-stream-id, so it shouldn't be a part of common-error-payloads
at all.
In templates/zerver/api/delete-queue.md, we have a sample fixture
for when the queue_id passed to client.deregister_queue is not
valid or the event queue in question has already been deleted.
This commit tests that fixture.
Note that this error payload is specific to client.deregister_queue.
In templates/zerver/api/create-user.md, we have a sample fixture
for when a client attempts to create a user with the same email
as an existing user. This commit adds a test for that fixture.
Note that this error payload is specific to client.create_user
and this error payload isn't generated anywhere else.
In templates/zerver/api/add-subscriptions, we have a sample fixture
for when the user being subscribed is already subscribed to a
stream. This commit tests that fixture against a running server.
This commit adds tests for the fixture for when a user is not
authorized (perhaps because the query requires the use of admin
privileges) for a particular query.
In templates/zerver/api/update-message.md, we have a sample fixture
for when a zulip.Client does not have the permission to update/edit
a particular message. This commit adds a test for that fixture.
Also, tools/test-api now also uses a non-admin client for this test,
which might come in handy in the future.
This commit adds tests for the sample fixture for when a required
request argument is missing. Also, it moves the sample fixture
to common-error-payloads.md, since this is an error payload that
is common to most requests (except the ones that don't take any
arguments).
In templates/zerver/api/private-message.md, we have a sample fixture
for when the email address of the PM's recipient is invalid. This
commit makes sure that fixture is tested against a running server.
In templates/zerver/api/stream-message.md, we have a sample fixture
for when the target stream does not exist. This commit adds a test
for that sample fixture.
This allows for pasting in comma separated lists that will validate
pills on paste, along with creating an API for inserting many pills.
This now allows in the `input_pill.pill.append` a comma separated list
that will then call the `insertManyPills` method that will break up the
input and then call `append`.
This fixes an issue where we allowed both the CMD+CTRL keys for our
compose markdown shortcuts. The correct behavior is to allow either
Cmd or Ctrl, based on whether it's MacOS (Cmd) or Ctrl
(Linux/Windows), to match how those platforms work.
Fixes#8430.
We now isolate the code to transmit messages into transmit.js.
It is stable code that most folks doing UI work in compose.js don't
care about the details of, so it's just clutter there. Also, we may
soon have other widgets than the compose box that send messages.
This change mostly preserves test coverage, although in some cases
we stub at a higher level for the compose path (this is a good thing).
Extracting out transmit.js allows us to lock down 100% coverage on that
file.
I think it makes more sense to first tell the user that
the character you are entering is invalid than telling
minimum length requirement is not satisfied.
Fixes#3058.
We make request to check availibity of subdomain which also checks the
subdomain for illegal characters. Duplicating the backend logic in
frontend in this case is thus no longer necessary.
This uses an actual query to the backend to check if the subdomain is
available, using the same logic we would use to check when the
subdomain is in fact created.
This changes the missed-message logic to use the new encoding
scheme for stream url fragments that prefixes them with
"{stream_id}-".
For simplicity sake, we just get a Stream object to pass to
the helper functions, and we only call get_display_recipient()
now for the HUDDLE case. (We were calling it wastefully before
for the PM case.)
This commit prefixes stream names in urls with stream ids,
so that the urls don't break when we rename streams.
strean name: foo bar.com%
before: #narrow/stream/foo.20bar.2Ecom.25
after: #narrow/stream/20-foo-bar.2Ecom.25
For new realms, everything is simple under the new scheme, since
we just parse out the stream id every time to figure out where
to narrow.
For old realms, any old URLs will still work under the new scheme,
assuming the stream hasn't been renamed (and of course old urls
wouldn't have survived stream renaming in the first place). The one
exception is the hopefully rare case of a stream name starting with
something like "99-" and colliding with another stream whose id is 99.
The way that we enocde the stream name portion of the URL is kind
of unimportant now, since we really only look at the stream id, but
we still want a safe encoding of the name that is mostly human
readable, so we now convert spaces to dashes in the stream name. Also,
we try to ensure more code on both sides (frontend and backend) calls
common functions to do the encoding.
Fixes#4713
We use the command
'select nextval('sequence') from generate_series(1, increment_number)'
which returns a list of allocated values for the ids.
This list is used to assign ids to the to be converted objects.
This adds button under "Organization profile" settings, which
deactivates the organization and sends an "event" to all the
active user and log out them.
Fixes: #8212.
This removes some options when a user clicks on their own profile, and
replaces them with an option to edit their own profile settings.
Front end test adjusted because it was testing an option that does not
appear if a user clicks on their own profile.
Fixes#5075.
This really only touches a couple of key points, but it does most of
what's needed to collapse stuff down into columns or a single column
when stuff is narrow.
Fixes#7925.
This adds a click handler to `.user-group-mention` which works in a
fashion that is quite similar to `.user-mention`. It generates and
displays a popover.
The popover has a list of members, their online status (if they are
not bots) or their bot status if they are bots (it's not clear whether
ultimately bots should be able to be members of usergroups, but I'm
able to add one, so I thought it would be worth supporting).
The popover's `UL` element has max-height and overflow-y atttributes
so large groups will grow a scrollbar.
Fixes#8300.
This change resets the list_of_popovers array to a new empty array after
hide_all has been called. Prior behavior kept it around, and it kept
pointers to orphaned DOM nodes.
Fixes#8416.
We have a good "Help Center" section in the footer, where a user
may already expect to look for help/support related requests, so
we can replace the navbar spot there with the "Why Zulip" page link.
This reverts commit 8e2d9b8f68.
This adds the arrows back to the recipient bars because even though
it's not our end state, it looks better than the boxy design that we
had in between.
This enforces `**` around all the mentions including "at-all" and
"at-everyone" mentions. Hence this makes `@all` and `@everyone`
invalid mentions, resulting into proper syntax for these mentions as
`@**all**` and `@**everyone**` respectively.
Note from tabbott: This removes an old feature/syntax, which made
sense back when @Tim was also a way to mention a user with Tim as
their first name. Given how nice typeahead is now, the user part of
the feature was removed a while ago; this should have gone at the same
time.
Fixes: #8143.
This avoids hitting Zulip's rate limits when there are more than 100
streams involved.
Manual testing:
- Changed notification settings for all streams from notifications tab of user settings
- Monitored the network tab to make sure 1 query is being sent
- Checked notification settings of individual streams
Fixes#5898.
Previously, a user with "ham" anywhere in their email address would be
sorted before a group whose name starts with "ham", which resulted in
a lot of frustrating when trying to mention groups.
Fixes: #8301.
This commit modifies the Markdown extension in bugdown/api_code_examples.py
to support rendering code examples in multiple languages by specifying
the language like so:
{generate_code_example(python)|doc.md|example}
This makes us one step closer towards adding support for testable
JavaScript code examples.
This commit simply adds a comment in zerver/lib/api_test_helpers.py
explaining why it isn't worth the effort to explicitly test the
code example in api/get-events-from-queue.md.
This may be helpful for some API clients, since it avoids them needed
to do somewhat messy post-processing on the results (the data was
always available via scanning for the first unread message in the result).
Fixes#6244.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.
NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.
Fixes: #320 and #291 partially.
In this commit we add support for some tags which are also called
void-elements according to
http://w3c.github.io/html/syntax.html#void-elements to be parsed by
our template parser and get tagged as singleton_html_tags.
Fixes: #8387.
Because the Transifex API is hilariously bad at namespacing, this will
work for uploading to any project with its "slug" set to `zulip-test`
that you have access to; for me, this is a fresh test project I
created in my own personal Transifex organization.
To use, just follow the command in `tools/push-translations`, but
s/zulip/zulip-test/g.
Now executable! Just run `tools/tagmessages`.
Also, get the username and password from a `.transifexrc` file.
And hardcode the project slug to `zulip-test` rather than to `zulip`;
the Transifex API is bad at namespacing, so this makes it possible to
run this script on a test project (the only way we're currently using
it) even for people like me who can also upload to the real Zulip
project on Transifex.
This dynamically resizes the box to fit the variable text size present
with different languages.
At the same time, it adds padding to the English version to make it
look similar to previous versions.
Fixes#8275.
With minor fixes by eeshangarg!
Eeshan: I decided to remove the screenshot. It looks very old and
was blurry and the instructions were very screenshot-agnostic
anyway!
I couldn't update the screenshot because Airbrake doesn't even let
you use the free trial till you give them your credit card info,
which I didn't want to do!
External bots may call bot_handler.quit() when
they wish to terminate, e.g. due to a misconfiguration.
Currently, embedded bots ignore calls to quit(), even
though they signal a problem. This commit does the first
step in handling quit() calls by logging a warning.
Now, all the various DOM elements are named by a variable, keyed off
the configuration of the upload_options object.
This is most of the work required to support file upload in the
message edit area.
This is a nonfunctional refactor that is key preparation for allowing
uploading files in message editing.
Note that this makes no actual changes to the code; it just changes
the function structure.
This is based on usage in bulk_change_user_names.py, and that
the RealmAuditLog acting_user field is Optional[UserProfile].
This could be more meaningfully changed in future, perhaps to
indicate that the command was run by a specific zulip user.
This comment didn't really explain things unless you were looking at
the code, and/or had a strong enough association for what "cn" means
that it was obvious what must be meant. Maybe this will be clearer.
There is one other meaningful key, which is optional: "short_name",
for which I guess a typical value if supplied would be "uid" or
"userid". I'm not sure we really do much with a user's `short_name`,
though, so didn't add a comment for it. When this key is omitted,
we set the user's `short_name` to the same thing as `full_name`.
It catches the `UserProfile.DoesNotExist` exception and
hence prevent internal server error.
Also remove option to select empty bot owner.
Fixes: #8334.
Update version of Vagrant from 1.8.6 to 2.0.2.
Update version of VirtualBox from 5.1.8 to 5.2.6.
We needed to update Vagrant because older versions have networking
problems on modern Macs, resulting in weird `apt` errors when
provisioning.
Commit message and some text tweaked by tabbott.
When in the stream-searchbar, a user can now use the arrow keys to iterate
through the suggestions. Therefore the currently selected list element is
assigned a CSS class 'highlighted_user'.
The main functional testing is done with casper but node test are still
included to keep the high coverage.
Line-wrapping issues are resolved. Night-mode CSS handling is included.
We do the following here:
* Remove libjasper-dev from THUMBOR_VENV_DEPENDENCIES.
Reason: This dependancy wasn't really needed by us for using
thumbor. It was a dependancy for using open-cv as Imaging Engine
in thumbor but we use PIL (Pillow now) as Imaging Engine.
* Add zlib1g-dev, libfreetype6-dev to THUMBOR_VENV_DEPENDENCIES.
Reason: These are dependancies of Pillow which are required for it
Pillow to function. Since we use Pillow in thumbor as Imaging Engine
we need these. Stuff before this didn't break because we also use
Pillow in development Environment and have these dependancies
installed from VENV_DEPENDENCIES as well.
When the answer is False, this will allow the mobile app to show a
warning that push notifications will not work and the server admin
should set them up.
Based partly on Kunal's PR #7810. Provides the necessary backend API
for zulip/zulip-mobile#1507.
We keep having to change the same thing in three places here; and also
the duplicates have accumulated unnecessary variation that makes it
hard to see what's actually supposed to be different and not different
in the three cases.
* Put imports in order.
* `import stripe`; that's the style upstream docs recommend, and it avoids
confusion e.g. between our StripeError and the library's StripeError.
* Simplify loading JSON.
* Keep lines largely to 100 columns.
This helper function will allow us to cycle through PM narrows
that are unread, once we map it to a hotkey and/or other UI.
(We intend to make something like the "n" key for topics, but
that works on PM narrows instead.)
This is a pretty pure code move, where we moved stuff from
message_store to pm_conversations:
insert_recent_private_message() -> recent.insert()
recent_private_messages -> recent.get()
The object message_store.recent_private_messages was not
encapsulated in a function before this change. Now it is
hidden in the scope of pm_conversations.recent.
Both of the modules touched here maintain 100% line coverage.
The error message, in relevant part:
```
zerver/templatetags/app_filters.py: error: INTERNAL ERROR --
please report a bug at https://github.com/python/mypy/issues version: 0.560
File ".../site-packages/mypy/nodes.py", line 497, in accept
return visitor.visit_func_def(self)
File ".../site-packages/mypy/report.py", line 317, in visit_func_def
assert start_indent is not None and start_indent > old_indent
AssertionError:
zerver/templatetags/app_filters.py: : note: use --pdb to drop into pdb
```
Seems mypy gets confused by the comment following the decorator.
I got distracted, came back later to a successful test run in my
terminal, and thought I remembered finishing the change and just
kicking off a final test run to check.
In fact, there was an `assert False` right in the normal case for
production, and I just hadn't finished a test for that path. (m.-)
Definitely the most grateful I've been for our coverage checks,
which highlighted this for me.
Remove the `assert False`, and also finish writing the test it was
there to help me write. Those lines are covered now.
This is a wrapper over lru_cache function. It adds following features on
top of lru_cache:
* It will not cache result of functions with unhashable arguments.
* It will clear cache whenever zerver.lib.cache.KEY_PREFIX changes.
This will essentially run the code paths to go from whatever you were
at before to /all and back in the case of /new, which will call the
render function three times (!!), so remove this call because it isn’t
really necessary anyways.
Currently the new streams user list will populate twice when you click
the new stream button (or “+”), because it is triggered once directly
by the button click and then once by the hash change to /new, so we
want to ignore the changes by the hash change.
This should make it possible to use the zulip_ops base rules
successfully on chat.zulip.org. Many of the changes in this commit
are hacks and probably can be cleaned up later, but given that we plan
to drop trusty support soon, it's likely that most of them will simply
be deleted then.
All the event handler did was resetting some entries in the edit
bot form. This is unnecessary, because the whole form gets
destroyed anyway when closed.
This is done by rewriting JS manipulations of the DOM tree
in the bot-settings.handlebars template. Dead code involving
the affected JS variables is removed.
This is the first step in cleaning up the bot edit code.
Since the bot edit form appears dynamically, we remove
it from the static HTML scaffold, of which settings_sidebar
is a part of.
The fresh imported data shows that the users emails are not included
in the data. However, the data received from the older method of slack
(which is using legacy tokens) contains the email data of the users.
When removing the description from a stream (i.e. setting it to ""),
the UI was not correctly updating the description. This is because we
were checking incorrectly for a falsey value, rather than the specific
value undefined (which means the description wasn't changed).
It's too easy to go over the rate limits when using the webapp.
The correct fix for this probably involves some changes to which
routes get covered by what sort of rate limit, but for now, just
increase the limits.
If some bug in Bugdown results in a rendered message content that is
bigger than twice the message size, we now just throw an exception
from Bugdown. This is considerably better than the old behavior,
which might result in an enormous message being placed in the database
(potentially, bigger than the 1MB limit to store in memcached), which
would in turn result in tragic consequences.
This fixes#8322, in that it prevents the super bad outcome seen there
(where basically Zulip became unusable for everyone on the stream
where the message is posted). Now, the failure mode is just the
message failing to send. Still not ideal (and requires further work
on the URL embed feature), but not a minor problem, not a major one.
For now, we still need the Travis badge, since Travis is where we test the
production installation process. But ideally, we'll end up removing that too.
This commit adds tests (and thus, an extra code example) for
unsubscribing another user from a particular stream by passing in
the `principals` argument to client.remove_subscriptions. The
ability to pass in `principals` was added in the latest release
of the zulip API PyPI package.
We now have a separate page for common error payloads, for example,
the payload for when the client's API key is invalid. All error
payloads that are presented on this page will be tested similarly
to our other non-error sample fixtures.
The zulip user has no need to see this file; it's used by nginx.
And when we set up the cert early in install, there's no zulip user
yet anyway, so this fails.
Otherwise prepare-base is likely to fail when first run (but then
succeed when rerun, because the container is left running), because
the container isn't up yet when we try to operate in it.
Also clean up the placement of `set -e` vs `set -x`.
We've been running this change on zulipchat.com for a couple of months
now. Before then, we used to regularly get exceptions like this:
File "./zerver/views/messages.py", line 749, in get_messages_backend
setter=stringify_message_dict)
File "./zerver/lib/cache.py", line 275, in generic_bulk_cached_fetch
cache_set_many(items_for_remote_cache)
File "./zerver/lib/cache.py", line 215, in cache_set_many
get_cache_backend(cache_name).set_many(items, timeout=timeout)
File "/home/zulip/deployments/2017-09-28-21-04-12/zulip-py3-venv/lib/python3.5/site-packages/django/core/cache/backends/memcached.py", line 150, in set_many
self._cache.set_multi(safe_data, self.get_backend_timeout(timeout))
pylibmc.Error: error 48 from memcached_set_multi
This error means memcached was unable to find space for the new value.
You might think that because memcached provides an LRU cache, this
shouldn't happen because it would just evict something... but in fact
* memcached splits its data into "slabs" by object size, and
* until recently, once a 1MiB "chunk" is allocated to a given "slab"
i.e. size class, it wouldn't be reclaimed to allocate to another.
So once the cache has been filled up with objects of some distribution
of sizes, if some objects come in that would go in a different size
class, we have no chunks for that size class / slab, and can't get one.
And that's exactly what was happening on zulipchat.com.
Useful background can be found in
https://github.com/memcached/memcached/wiki/ServerMaint#slab-imbalancehttps://github.com/memcached/memcached/wiki/ReleaseNotes1411https://github.com/memcached/memcached/wiki/ReleaseNotes1425https://github.com/memcached/memcached/wiki/ReleaseNotes150
We're already running v1.4.25, which provides an "automover" that should
be well equipped to fix this; v1.5.0 turns it on by default.
With this commit, adopt the "modern start line" recommended in the
release notes for our v1.4.25, including turning on the automover.
This workaround is only for new users. Users who are/were using old
versions of Vagrant has to manually change the value of box
metadata_url. See https://github.com/hashicorp/vagrant/issues/9442
Apparently, we've now had the first time one of our contributors had
their account deleted (at least, the author page for the contributor
who has 21 commits in python-zulip-api now 404s).
This fixes a bug where the open graph preview bottom fade is dark
rather than white when not in dark mode, which results in a heavy
dark faded line at the bottom of the description.
This commit adds support for passing in an argument to the macro
"call" to explicitly specify a fixture to render, like so:
{generate_code_example|doc_name|fixture(stream_message_with_args)}
api/get-all-streams - make use of the api_code_example Markdown extension's
feature of recursively extracting multiple code examples from a single
test method.
To generate a code exammple,
{generate_code_example|<api_doc_md>|example} sounds better and
more intuitive than,
{generate_code_example|<api_doc_md>|method}
The count of integrations is automatically computed now, so this will
change every time we add 10 more. Just stop asserting on the number.
Thanks to @hackerkid for spotting the issue.
This code duplicated the code in setup_realm_internal_bots, with some
added logic to avoid trying to create the same bot twice. That logic
was buggy so that it would never work at all -- it subtracted a set of
UserProfile objects from a set of email strings -- so it looked like
the command might blow up when run after the users already existed.
In fact, the buggy logic wasn't necessary, because the work the
command does after it is idempotent -- in particular `create_users`,
within its subroutine `bulk_create_users`, already filters out users
that already exist. So just cut the buggy stuff out, deduplicate the
rest with `setup_realm_internal_bots`, and document that invariant on
the latter.
While we're here, in the common case bail early without doing any
per-realm work in Python, since we're running this on every upgrade.
These were removed in 953ee778f, along with the stream names and
right-sidebar user names mentioned in the commit message. But unlike
those, these don't get any background change on hover, so we need some
way to show the hover.
We can probably still do something that looks nicer than this, but
it's better than not having it.
This makes them roughly the same visual weight as the (non-muted)
stream icons below -- though that's imprecise because the latter
varies with color. Tested in both normal and dark-mode.
This reverts commit dd406e87e. Whatever awkward error message might
have motivated that change, we can fix in a more targeted way when we
see it again. This version would delete the potentially-informative
context the caller provided in `response` -- even just "Failed" --
based on whether an error message happened to contain a colon
somewhere in it, which is sure to end up feeling unpredictable.
See 625939 for more information. In short, the purpose of this delay is
to give autoreload code enough time to touch every watched file at least
once before the change is made.
The last bit of #8059 (merged as b875fe07) didn't get merged, so just adding
it as a followup.
The "Other notification settings" section will also eventually house the
"pick your notification sound" setting, which is why it isn't called "Other
email settings" or similar.
Users having only account in one realm will not be distracted by realm
name in subject lines of every email. Users who have multiple
accounts in realms can turn this setting on and receive a
corresponding realm name in email's subject.
Tweaked by tabbott to rebase and address a few small issues.
Fixes#5489.
For messages where the entire rendered body is a message_inline_image
object, we actually don't display any text and just display the
image. These messages may have links to images which might or might
not be internal to Zulip but in both cases there is a chance of this
links being broken when accessed by an email server like Gmail that
doesn't possess the recipient user's cookies.
We don't want to have ugly looking broken images displayed in email
notifications. So we patch this by inserting a replacement for the
`message_inline_image` block in which we essentially replace the
content with the textual link.
Edited for clarity by tabbott.
This is easy to do, and prevents this feature from getting a server
admin stuck in potentially a pretty uncomfortable way -- unable to
roll back a deploy.
This reverts commit acebd3a5e, as well as a subsequent fixup commit
0975bebac "quick fix: Fix migrations to be linear."
These changes need more work and thought before they're ready to
deploy on any large established Zulip server, such as zulipchat.com.
See discussion on #6534.
In place of the removed migration, leave behind a placeholder so
`manage.py migrate` doesn't get confused on installs where it was
already applied.
This adds some styling to make the open graph previews look a bit nicer,
including:
1. Adding a bottom fading gradient to slowly fade out text that is out
of bounds rather than chopping it off.
2. Using font anti-aliasing to make the characters appear smoother.
3. Increasing the font size of the title to give it prominence.
4. Changing the height to 80px from 70px.
Also refactor the "panels"/banner code to be a bit clearer about how
it's supposed to generically work, using [data-process] as a uniquely
identifying marker.
Fixes: #8166.
[greg: rebased and squashed a series of fixup commits.]
The original code made a 3/4-hearted effort to generically accommodate
more banners/"panels" later, but named itself after the first one made.
[greg: expanded commit message.]
This is to break the bigger functions into smaller
ones and hence, makes unit testing easier.
Also includes renaming the stream's topic name to 'from slack'.
The issue is that the server wants to display an error when you make a
request and fail, but the jQuery form validator also wants to display
errors at the same time. This removes the server errors to display the
jQuery.
Fixes: #8239.
This properly vertically centers the deactivation modal in the
user settings section by setting the top to 50% and the transform
to -50% (50% of the height of the actual modal).
This is an alternate solution to #7888, which just removes the
animation, breaking the normal modal behavior.
The header had a padding of "20px 0 15px", which meant that there
was more padding on the top than the bottom, so it has been changed
to "15px 0" instead.
This is a partial revert of c5cdede891.
The added 4px padding made the dropdown pill not vertically center
within the portico header. This removes the padding as it was a
cosmetic change that was not necessary.
This is a partial revert of a37e993907.
We'll make this the normal behavior soon, once we're satisfied with
our arrangements for sending the admin straight to realm creation and
using the app without configuring email. The instructions in the docs
will also have to change accordingly, of course.
This will let us defer configuring outbound email to the end of the
install procedure, so we can greatly simplify it by consolidating
several scripted steps.
The new flow could be simplified further by giving the user the full
form in the first place, rather than first a form for just their
email address and then a form with the other details. We'll leave
that improvement for a separate change.
Now, there's just one spot at the beginning of the function where we
inspect the string key the user gave us; and after that point, we not
only have validated that string but in fact are working from our own
record that it pointed to, not the string itself.
This simplifies the code a bit, e.g. by not repeatedly searching the
database for the key (and hoping everything agrees so that we keep
getting the same row), and it will simplify adding logic to inspect
row attributes like `presume_email_valid`.
There's no use case for presenting a key that's invalid; if we haven't
given the user a valid key, we needn't send them to a URL that
presents an invalid one. And the code is simpler to think about if
the only keys that can exist (after the validation at the top of the
function) are valid ones.
Apart from the case where creation_key is None, but invalid, and
settings.OPEN_REALM_CREATION is True so that we'd previously let the
invalid key slide, this is a pure refactor.
This fixes an issue where the user's own avatar was being sent down
the wire as None. We could have fixed it, as in #8265, by adding code
in the webapp and mobile apps to compute medium-size gravatar URLs as
well, but that would be messy, and there's little benefit to that
complexity (saving at most 2 URLs from the payload).
Fixes#8253.
Different formats for configuration files have a wide variety of ways
of representing lists; so if you're not accustomed to Python syntax,
or aren't thinking of this file as Python code, the syntax for several
ALLOWED_HOSTS entries may not be obvious. And this setting is one
that an admin is likely to want to touch quite early in using Zulip.
So, demonstrate a multi-element list.
For similar reasons, demonstrate an IP address. This one is in a
range reserved for documentation (by RFC 5737), like `example.com`.
Fixes improper katex rendering in markdown help template. The
rendering was originally correct, but was broken when we upgraded to a
new version of KaTeX with a different HTML/CSS interface.
"Sign up" -- pretty important for people to be able to read that!
There's also the "or" that gets rendered as a big "OR" in a divider --
but that styling is too brittle, so that if "or" becomes e.g. "oder",
the lines stay the length they should be for "or" and the lot of them
wrap. Yuck. Better to leave it as "or" until that's fixed.
In password modal, submit button's data-dismiss property should be
removed, as on data-dismiss we flush all data of form fields,
which results in passing empty string to json request.
We already flush all data of fields after compeletion of request.
This restores the property that changing one's name in on browser's
"account settings" also changes the user's name in other browser windows'
"account settings" pages.
This makes this UI widget more consistent with its neighbors.
tabbott: This introduces a bug with how the `full_name_field` HTML is
managed; it should be done via the `server_events.js` handler.
- When password fields are cleared, update password strength bar.
- On data dismiss, clear password fields.
- Exclude forgot-password div from focus.
- On enter key, modal should be submitted not dismissed.
- Change password btn text from "Change password" to "*****"
This makes many diffs that touch our JS dependencies in package.json
much, much more readable by suppressing multi-thousand-line changes.
With that kind of change routine, it's clear this isn't a file people
are closely reviewing the text diffs in anyway.
This script and our CI scripts tools/travis/{backend,frontend} have
stayed pretty well in sync in the 6 months since 360c27ded made that
relationship explicit and easy to check!
Just one small exception; so fix that.
This may or may not be temporary, but either way, the other code is
there in source control, and the "why" of disabling gitlint is the
helpful bit for a comment.
If there was a realm on the base URL, its logo and name were being
displayed when registering a new realm (i.e. the page where realm details
are entered, after confirming email). This commit prevents the realm
details from being displayed.
Fixes#8186
Circle has been much more robust than Travis in the few weeks we've
had them both enabled, as well as somewhat faster. By disabling
Travis, we're no longer exposed to its spurious failures.
One suite remains on Travis: the "production" suite. This work is
being tracked in #7748.
We haven't actually been using this zulipbot feature for a long time
(haven't applied the `travis updates` label which it looks for.)
Rather than port the config to CircleCI, just cut it for now; we can
always add something like it back if we start using it.
Injecting the generated-file warning into the settings dict felt a
little unnecessarily magical. A warning like this is always going
to be at the top; the way it might differ between files is mainly
if the syntax for a comment varies, and in that case a simple
substitution like we're doing in this template wouldn't be enough
to express the difference anyway. So, embrace the hardcoding.
Now, the template and the images.yml entry have a very simple
relationship: the keys in one are exactly the keys in the other.
That's good for people quickly and confidently understanding it.
A comment like this was removed in
fa44d2ea6 "settings: Remove autoscroll_forever setting."
The comment went on to say something about autoscroll, but this
part still seems relevant. While here, adjust grammar and caps.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
Some of our code examples can only be run with administrator
credentials (such as create-user). Thus, the Markdown extension
for generating code examples should have an option to include
the lines that recommend using an admin zuliprc instead of a
non-admin one.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
Now that we have a Markdown extension-based test framework for
generating and testing code examples on our /api pages, we don't
need this macro anymore!
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested in
tools/lib/api_tests.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_example to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
This commit adds a Markdown extension that allows the following
syntax,
{generate_code_example|<md_file_name>|<fixture or method>}
to generate code examples and fixtures found in tools/lib/api_tests.py
and templates/zerver/api/fixtures.json, respectively.
We don't disable code by commenting it out -- that leaves a mess.
We delete it. Remembering what the code was is what source control
is for.
This fixes cd849bc3f "test-run-dev: Disable Nagios check."
from a few weeks ago.
This completes the separation of our logic for managing Stripe
customers from the view code for the billing page.
As we add more features to our Customer model and to our Stripe
integration, we might further separate those two things; but for now
they're nearly synonymous and there's no problem in them being mixed
together.
Pull the code that talks to Stripe out into its own functions.
In a followup commit we'll move these to a separate file, as well
as the error-handling logic that remains in the view function
for now.
Also fix the translation markings: the translated string must be a
constant (e.g. a format string), or else translation is impossible.
Viewing with `-b` shows the few changes that happen in the logic
as it moves out of the view function; viewing without shows the
few changes in the rest of the view function.
If an exception was thrown inside `send_email` resulting in a retry,
we would include the `failed_tries` data in the event, which turned
out to thrown an exception itself.
This fixes that flow, including deepening the test so that it would
fail if we didn't have the new logic.
The Travis config does this by a `pip install codecov` in setup and
running the resulting `codecov` command at the end. That risks
confusion because it makes the library look like something we might
be using *during* the tests to collect coverage information.
Instead, install and run codecov together at the end.
(The docs also suggest using their bash-based uploader instead;
but following those directions seems to result in coverage.xml
not getting included (maybe not even built), and no coverage data
getting uploaded for Python source files, only JavaScript. Not
sure why that is, but using the pip package is fine.)
Also, because this step is inherently about talking to a
third-party service whose reliability we can't control, and because
it's purely a reporting step that happens after our actual test
suite has succeeded, don't fail if it fails; just print a message.
(I'd prefer a somewhat louder message, like turning that step
yellow or something in the Circle web UI, but Circle doesn't offer
an option like that. If the coverage reports start consistently
failing for some reason, then if we're actually reading them we'll
notice anyway.)
[greg: Edited step name and error message; wrote this commit message.]
The first argument to the error callback is *usually* a string code
from a list in the filedrop source; but sometimes it was the text
the server sent in the HTTP status line, instead. The latter isn't
predictable, and so it's not possible to write app code that uses it
to handle error consistently.
Instead, use that parameter for the numeric HTTP status code. This
still isn't totally clean in that sometimes it's internal filedrop
errors, as strings, and sometimes it's HTTP status codes, as numbers;
but at least both of those are things we can sanely handle with a
`switch` statement.
Also pass through `serverResponse`, which for a nice JSON error from
the server will contain meaningful information about the error which
the calling code can use for nice error handling. And just drop the
HTTP status text, which at best is redundant with the numeric code.
In passing, fix one case where for no obvious reason filedrop was
passing the file object but not the index.
This should be a pure refactor.
We'll replace this primarily with per-realm quotas (plus the simple
per-file limit of settings.MAX_FILE_UPLOAD_SIZE, 25 MiB by default).
We do want per-user quotas too, but they'll need some more management
apparatus around them so an admin has a practical way to set them
differently for different users. And the error handling in this
existing code is rather confused. Just clear this feature out
entirely for now; then we'll build the per-realm version more cleanly,
and then we can later add back per-realm quotas modelled after that.
The migration to actually remove the field is in a subsequent commit.
Based in part on work by Vishnu Ks (hackerkid).
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
This commit also adds a tool to push translation sources to Transifex.
This tool makes sure that we don't push mobile source file. Mobile
source file is supposed to be handled from Zulip-Mobile repo.
This is a little cleaner in that the try/except blocks for
SMTPException are a lot narrower; and it'll facilitate an upcoming
change to sometimes skip sending mail.
(This is a small fixup to the main change, which was accidentally
included in a previous commit:
08bbd7e61 "settings: Slightly simplify EMAIL_BACKEND logic."
Oops. See there for most of the changes described here.)
The installer works out of a release-tarball tree. We typically want
to share this tree between successive test-install runs (with an rsync
or similar command to update source files of interest) because
rebuilding a release tree from scratch is slow. But the installer
will munge the tree; so instead of directly bind-mounting the tree
into the container, we need to give it an overlay over the tree, as a
sandbox to play in.
Previously we used lxc-copy's `-m overlay=...` feature to do this,
mounting an overlay in the container. But then sometimes in
development we want to reach in and edit some code in the tree,
e.g. before rerunning the installer after something failed. Reaching
inside the container for this is a pain (`ssh` would add latency, and
I haven't installed sshd in the containers; and getting rsync to work
with `lxc-attach` was beyond what I could figure out in a few minutes
of fiddling); and editing the base tree often doesn't work.
So, create the overlay with our own `mount -t overlay`, and have
`lxc-copy` just bind-mount that in. Now the host has direct access to
the same overlay which the guest is working from.
Also this makes it past time to help the user out in finding the fresh
names we've created: first the container, now this shared tree. Print
those at the end, rather than make the user scroll to the top and find
the right `set -x` line to copy-paste from.
This brings the right sidebar UI to match the similar widget in the
left sidebar. Since there's no other plausible effect for a click in
this whitespace, this small tweak should make using Zulip a bit more
convenient.
Fixes#8161.
There is a bug on MacOS on Zulip where the categories are all
duplicated due to faulty looping logic, while also changing the Map to
an Array since maps have less cross-browser support.
This is the last commit in the series of commits for completing the
project of cleaning up our html templates to have 4 space and
valid indentation.
Fixes: #1236.
In this commit we also fix a test which would fail as a result of
doing this cleanup since the test wasn't designed to take into
account the space chars which might occur in the beginning of a
html line.
This script iterates over all the mobile.json resources and creates a
single file at static/locale/mobile_info.json which contains total and
not-translated strings information against each language. After doing
this, it deletes all the mobile i18n resources downloaded by
tools/sync-translations because we neither want to check them in our
repository nor we want to make our repository dirty.
This test randomly fails far too often in Travis -- I think more than
all our other tests combined. It needs to be fixed before we can ask
everyone to look at build failures it causes.
In the email integration, previously, EMAIL_GATEWAY_EXAMPLE wasn't
rendered at all, which was recently fixed. So, now, we should make
sure that it gets rendered!
In order to get test coverage on topic name checks, we
do them in Addressee, so that we don't hit an assertion
first. The assertion in question is in Addressee.topic(),
and it was added partly to appease mypy.
Gmail is a bad example for outbound email; use a generic example.
Also leave the `= None` default out of the config file, as it's
redundant with DEFAULT_SETTINGS in our internal settings.py ; and
explain in the latter why we don't mention the other SMTP settings.
This quiets the last of the warnings that Sphinx was giving us about
documents not being in any toctree, now that we've explicitly told it
with `:orphan:` about the documents we intentionally don't link there.
Now that we have `eval_rst` and can explicitly exclude pages from the
toctree completely, we no longer need to set `includehidden`, and we
can return to using upstream's template.
(Meanwhile, our feature request upstream was successful! See
rtfd/sphinx_rtd_theme#485, which upstream implemented just a week
after we requested it. So that would have been another option.)
This reverts commit 11b8b8f48 "docs: Add rtd layout template."
This is what the Sphinx docs recommend when you actually don't want
the page to be included in navigation:
http://www.sphinx-doc.org/en/stable/markup/toctree.html
And now that we have `eval_rst`, we're able to take advantage of it!
One difference between doing this and the old way of making "hidden"
toctree entries is that with the latter, the "previous" and "next"
links at the bottom of each page would thread through the hidden
entries; which gets kind of confusing when they don't appear in the nav.
The careful testing with dozens of realms was a discrete thing
that happened, not a general state that the tool was in;
certainly not the last state it was known to be in, as the rest
of the sentence explains how it hasn't been so carefully tested
since then. So "as of" doesn't fit.
This causes us to give an error if you pass the installer any
positional arguments, e.g. with `--`. There's no reason you'd want
to do this, but I accidentally did it by passing an extra `--` to
the `test-install/install` wrapper and spent a few minutes on
confused debugging.
This doesn't yet pass all Nagios checks correctly, and still has a few
flaws:
* The ideal setup code for the `nagios` user in the database isn't included.
* Some of the other details are a bit off; we need to split some host roles.
But it's better than nothing, and we can iterate from here.
Thanks to the magic of `set -x`, I noticed this:
```
+ cat
++ ssl-cert
/tmp/src/zulip-server/scripts/setup/generate-self-signed-cert: line 49: ssl-cert: command not found
+ apt-get install -y openssl
[...]
```
In other words, we were trying to run `ssl-cert` -- the name of a
Debian package I meant to refer to in a comment inside the templated
temporary config file for `openssl req` -- as if it were a command.
It wasn't, hence the error.
Because `set -e` has loopholes like a sieve, this didn't cause the
script to exit, just produced this funny output and presumably caused
the config file's comment to be missing a word. In principle, it
could do something surprising if for some reason there were a command
named `ssl-cert` on PATH.
Fix it.
This is easy now, so make it known to admins who are looking for a
fast path for a test install.
Also totally cut the painfully complicated steps for generating a
self-signed cert by hand. Anyone who actually wants that can find a
hundred explanations on the Web, or can look at our script if they
want to specifically mirror how we do it (which is mercifully much
simpler than this.)
This gives us just one way of adopting a self-signed cert, rather than
one script which would generate a new one and an option to another
which would symlink to the system's snakeoil cert. Now those two
codepaths converge, and do the same thing.
The small advantage of generating our own over the alternative is that
it lets us set the name in the cert to EXTERNAL_HOST, rather than the
system's hostname as embedded in the system snakeoil certs. Not a big
deal, but might make things go slightly smoother if some browsers are
lenient (in a way that they probably shouldn't be.)
Take the core of the logic from how Debian generates the system's
/etc/ssl/certs/ssl-cert-snakeoil.pem ; that gives me more confidence
in the various config choices, and it also demonstrates a much cleaner
way to use the `openssl` tool. Also replace the outer shell logic for
CLI and logging with a cleaner version.
It runs in kind of a peculiar environment -- in particular with the
`tags` identifier injected into the namespace -- and it contains
very little code more complex than `foo = "bar"`, so there's not
much to check anyway.
Since we need KaTeX to be available for zerver/lib/tex.py and
static/third/katex/cli.js to be able to shell out to it. However, for
some reason, the KaTeX we bundle using Webpack doesn't seem to be
importable by Node (and it's also kinda a pain to find its filename
from `cli.js`).
So, we work around this by just using the legacy system for KaTeX.
Something similar is needed for zxcvbn.js, in order to support the
settings_account.js use case (basically deferred loading of this
file); that requires JS code to have access to the correct path for
zxcvbn.
This implementation never worked, for two reasons:
(1) The logic for getting the path to the katex package had invalid
syntax in the version of Node we use.
(2) The KaTeX package bundled by webpack doesn't seem to be importable
from Node at all.
This allows us to selectively use some of the powerful features of
ReST which Python projects with high-quality documentation (like
Python core, and Django) rely on.
It's now January 2018, so we can delete this caveat, right?
Not quite yet -- the original post we link to now has an
update saying 2018-02-27. Let's make it less specific,
in case the date changes again.
Bulleted information instead of prose, huzzah! Also I think we need
to explain the options a bit right here, or at least link to where
they're documented somewhere. (If the list gets much longer, we'll
want to shift toward the latter.)
Also reorganize existing information a bit, and clean up a couple
of nits.
What I really want is to give these sections nice stable slugs
to put on the anchors and use as the URL fragment, independent of
any wording tweaks on the text headings. But I don't think we
have that feature with Markdown and our current docs infrastructure.
At least for Certbot, the brevity helps make this heading clearer
than the previous one.
Previously we had a problem of id clashes while importing converted
slack data into an existing zulip instance with realms which are actively
populating the database.
This counts the total objects to be imported and does a db transaction
to increase the SEQUENCE number for that table by that number,
and hence allocates a range of ids for the to be converted slack data
objects.
Adds a check for newline that was present on backend, but missing in the
frontend markdown implementation. Updating messages uses is_me_message flag
received from server instead of its own partial test. Similarly, rendering
previews uses markdown code.
Fixes#6493.
This is done by using a bot's ID instead of email in
the handler methods for bot_data.bots and bot_data.services,
and updating all code paths involved.
This is the first step for allowing users
to edit a bot's service entries, name the
outgoing webhook configuration entries. The
chosen data structures allow for a future
with multiple services per bot; right now,
only one service per bot is supported.
Before this fix, the installer has an extremely annoying bug where
when run inside a container with `lxc-attach`, when the installer
finishes, the `lxc-attach` just hangs and doesn't respond even to
C-c or C-z. The only way to get the terminal back is to root around
from some other terminal to find the PID and kill it; then run
something like `stty sane` to fix the messed-up terminal settings
left behind.
After bisecting pieces of the install script to locate which step
was causing the issue, it comes down to the `service camo restart`.
The comment here indicates that we knew about an annoying bug here
years ago, and just swept it under the rug by skipping this step
when in Travis. >_<
The issue can be reproduced by running simply `service camo restart`
under `lxc-attach` instead of the installer; or `service camo start`,
following a `service camo stop`. If `lxc-attach` is used to get an
interactive shell, these commands appear to work fine; but then when
that shell exits, the same hang appears. So, when we start camo
we're evidently leaving some kind of mess that entangles the daemon
with our shell.
Looking at the camo initscript where it starts the daemon, there's
not much code, and one flag jumps out as suspicious:
start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
--exec $DAEMON --no-close -c nobody --test > /dev/null 2>&1 \
|| return 1
start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
--no-close -c nobody --exec $DAEMON -- \
$DAEMON_ARGS >> /var/log/camo/camo.log 2>&1 \
|| return 2
What does `--no-close` do?
-C, --no-close
Do not close any file descriptor when forcing the daemon
into the background (since version 1.16.5). Used for
debugging purposes to see the process output, or to
redirect file descriptors to log the process output.
And in fact, looking in /proc/PID/fd while a hang is happening finds
that fd 0 on the camo daemon process, aka stdin, is connected to our
terminal.
So, stop that by denying the initscript our stdin in the first place.
This fixes the problem.
The Debian maintainer turns out to be "Zulip Debian Packaging Team",
at debian@zulip.com; so this package and its bugs are basically ours.
This is a tool that throws away `fsync` calls and other requests for
the system to sync files to disk. It may make the install faster; for
example, if it has to install a number of system packages, `dpkg` is
known to make a lot of `fsync` calls which slow things down
significantly. Conversely, if there's a power failure in the middle
of running a test install, we really don't mind if the test install's
data becomes corrupt.
When the install script is successful, one of the final things it
wants to do is to move the tree that Zulip was installed from into the
deployments directory. It can't do that, at least not in a naive way
with `mv`, if the tree is actually a mount point. So, stick the tree
inside some other directory that we create just for the purpose of
being the mount point and containing the install tree.
This saves several minutes off the install time. Sadly pip still
clones Git repos for dependencies that point to them, but for many
others (not all? not sure) it just gets a wheel from the cache.
This provides a major simplification for non-production installs,
including our own testing (it's already in both the test-install
harness script and the "production" test suite) as well as potential
admins evaluating Zulip.
Ultimately this should probably be the default behavior, with perhaps
something shown to admins on the web as a reminder and link to help on
installing a better certificate. For now, pending working through
that, just get the behavior in and leave it opt-in.
It's not appropriate for our script to pass the `--agree-tos` flag
without any evidence of the user actually having any knowledge of,
let alone intent to agree to, any such ToS. Stop doing that.
Fortunately this script hasn't been part of any release, so it's
likely that no users have gone down this path.
The third-party `install-yarn.sh` script uses `curl`, and we invoke it
in `install-node`. So we need to install it as a dependency.
We've mostly gotten away with this because it's common for `curl` to
already be installed; but it isn't always.
This greatly simplifies iterating on changes to the installer and
associated code: just edit in the shared directory (or edit in your
worktree and rsync to the directory), and rerun.
With this change, the form with a directory is now really the main
way to run the script; the form accepting a tarball is really just
a convenience feature, unpacking the tarball and then proceeding with
that directory.
This will facilitate testing interesting installer features
using its own CLI.
On my laptop, with a recent base image (updated a few days ago with
`prepare-base`), it takes just 7 or 8 seconds to get to the installer
running, as timed by passing `--help` so that the installer promptly
exits.
Previoulsy, we display "0 subscribers" if user can't access stream's
subscribers. Replace subscriber count with lock icon in case of
unsubscribed private stream, in "All stream" list.
Display warning, saying "You can not access private stream subscribers,
in which you aren't subscribed", if user can not access subscribers;
instead of showing zero subscriber to stream.
It makes sense to make our Python and JS API examples more visible
than our curl examples, since Python is what most people will tend
to use.
Also, from a design perspective, an API documentation page that
starts off with a shiny Python example with syntax highlighting
looked much better than having a bland curl example be the first
thing readers see.
In order to do development on the installer itself in a sane way,
we need a reasonably fast and automatic way to get a fresh environment
to try to run it in.
This calls for some form of virtualization. Choices include
* A public cloud, like EC2 or Digital Ocean. These could work, if we
wrote some suitable scripts against their APIs, to manage
appropriate base images (as AMIs or snapshots respectively) and to
start fresh instances/droplets from a base image. There'd be some
latency on starting a new VM, and this would also require the user
to have an account on the relevant cloud with API access to create
images and VMs.
* A local whole-machine VM system (hypervisor) like VirtualBox or
VMware, perhaps managing the configuration through Vagrant. These
hypervisors can be unstable and painfully slow. They're often the
only way to get development work done on a Mac or Windows machine,
which is why we use them there for the normal Zulip development
environment; but I don't really want to find out how their
instability scales when constantly spawning fresh VMs from an image.
* Containers. The new hotness, the name on everyone's lips, is Docker.
But Docker is not designed for virtualizing a traditional Unix server,
complete with its own init system and a fleet of processes with a
shared filesystem -- in other words, the platform Zulip's installer
and deployment system are for. Docker brings its own quite
different model of deployment, and someday we may port Zulip from
the traditional Unix server to the Docker-style deployment model,
but for testing our traditional-Unix-server deployment we need a
(virtualized) traditional Unix server.
* Containers, with LXC. LXC provides containers that function as
traditional Unix servers; because of the magic of containers, the
overhead is quite low, and LXC offers handy snapshotting features
so that we can quickly start up a fresh environment from a base
image. Running LXC does require a Linux base system. For
contributors whose local development machine isn't already Linux,
the same solutions are available as for our normal development
environment: the base system for running LXC could be e.g. a
Vagrant-managed VirtualBox VM, or a machine in a public cloud.
This commit adds a first version of such a thing, using LXC to manage
a base image plus a fresh container for each test run. The test
containers function as VMs: once installed, all the Zulip services run
normally in them and can be managed in the normal production ways.
This initial version has a shortage of usage messages or docs, and
likely has some sharp edges. It also requires familiarity with the
basics of LXC commands in order to make good use of the resulting
containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`,
in particular.
In this we change the way 'Sending...' is displayed. Instead of
hardcoding it into the template we make change the paradigm so
that we can have a flexible message about what's happening
rather than just always saying 'Sending...'. For eg. this will
help in the upcoming feature of Scheduled Messages by having this
message say 'Scheduling...'.
This is responsible for:
1.) Handling all the incoming requests at the
messages endpoint which have defer param set. This is similar to
send_message_backend apart from the fact that instead of really
sending a message it schedules one to be sent later on.
2.) Does some preliminary checks such as validating timestamp for
scheduling a message, prevent scheduling a message in past, ensure
correct format of message to be scheduled.
3.) Extracts time of scheduled delivery from message.
4.) Add tests for the newly introduced function.
5.) timezone: Add get_timezone() to obtain tz object from string.
This helps in obtaining a timezone (tz) object from a timezone
specified as a string. This string needs to be a pytz lib defined
timezone string which we use to specify local timezones of the
users.
This is a server setting so I created the section "Server settings" in the help sidebar for this to go under, and rewrote the copy and retook the images that were originally done by @Privisus due to some issues.
Hopefully this version makes it somewhat clearer how the different
methods relate to each other, how to choose between them, what
`ZulipRemoteUserBackend` is for, and how the latter works.
The main cleanup here is to move the examples out of the bulleted
list, which was getting rendered in an ugly/confusing way.
I also removed some redundant text, fixed some typos, and changed
the wording a bit.
This code takes care of the environment running Python 3.4 when
test label is passed directly to the test-backend command:
./tools/test-backend test_alert_words
Several changes:
* De-duplicate code for different error types.
* No need to list lots of error subtypes where we aren't treating
them differently; StripeError is the base class of them all.
* Unexpected, non-Stripe-related, exceptions we can handle in the normal
way. Just make them show up in the billing-specific log too.
* The Stripe client library already logs type, code, param, and message
before raising an error, so we don't need to repeat those; just add the
HTTP status code (because it's not there already and sure why not),
and the Python exception type the client library chose to raise
in case that makes things a bit easier to interpret.
Normal server admins will never run this code, and zulipchat.com will
have this information configured before users see it, so this message
is really just for development.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.
There are more features we should add to this page and changes we
should make, but this gives us an MVP.
[greg: expanded commit message; fixed import ordering and some types.]
This also amends a commit from Brock Whittaker <brock@zulipchat.com>
that merges two separate functions for YouTube videos and Vimeo videos
into a generic video recall function.
Fixes#7550.
On the "Organization settings" page, we had two Save buttons
for admins that had identical markup. This was confusing for
people reading the code. Now the two buttons have different
markup and individual, targeted click handlers (albeit still
calling the same function to do most of the work).
The context of this fix is that I was debugging a
Casper flake where our Casper tests were essentially clicking
on the same button twice. Depending on the timing,
the second button click could cause a "No changes saved!"
behavior that confused the Casper test. It is unclear whether
Casper was clicking both buttons here (in which case this fix
is necessary) or the same button twice (in which case this fix
just removes a nasty red herring for debugging).
The code still has the flaw that both buttons basically submit
the same data to the server, despite the appearance on the page
that there are two forms. The best fix for that is probably
just to move the Language/Notifications stuff to another
panel. I wanted to avoid touching this code altogether, but
the minor modifications here were necessary to improve the
Capser testing situation.
Showing the server output transparently in casper tests
will save developers headaches chasing down flakes. It's
particularly important to see server output intermingled
with the Casper output.
We now always show JS console output when running Casper tests.
The app is not spammy for the "happy path", so there's no real
reason to quiet it down, and it's never been well documented
how to turn on the option, so we've subjected developers to
needless head scratching in the past.
The alert word highlighting is too light to work with the font
when it is white, so this changes it to have a translucent background
so that the color adjusts to be more visible on a darker background.
This is originally taken from #7844 and is a modification of the
solution in #7847.
The reason this approach is better for the codebase is now there is
only one color to change for alert words which reduces the likelihood
that someone in the future will change the color of one theme's
alert words and not the other.
Fixes: #7844.
We add two functions:
1.) check_schedule_message(): This function is responsible for
doing the essential initial checkes to verify the validity of
the message. These checkes include things like if user is
allowed to send messages to some stream or not or if the user is
a super_user. All this is basically done by further calling
check_message() with appropriate parameters. This is on the same
lines as is check_send_message().
2.) do_schedule_messages(): This function is responsible for
creating ScheduleMessage table rows for a list of messages that
are to be scheduled. This basically accumulates the ScheduleMessage
objects in a list and then bulk creates the rows.
An image link such as [example](http://www.example.com/example.png)
is supposed to be inlined by the backend. The regex
backend_only_markdown_re in markdown.js does not recognize this link
as an image link, causing issues when adding fixtures to
zerver/fixtures/markdown_test_cases.json .
This is fairly often -- though not always! -- failing, with a nasty
failure mode where it takes like 6 minutes to time out. See
discussion on #7748 (search for "bad link").
Actually, after seeing it happen just now when running
test-documentation on my laptop, on some other link, it occurs to me
that I've seen this before -- it's fairly common in Travis, too. It's
just that it doesn't actually cause the build to fail :-/, and on
Travis we haven't been paying as close attention to slow builds as we
are on Circle right now.
Generally stderr is the conventional place for this sort of running
commentary, and it's better set up for it: by default stdout may have
a buffer inside the process so that things written to it don't reach
the outside until later, while stderr is always by default unbuffered,
so messages are printed immediately.
Here, until the previous commit, because our color-reset sequence was
being printed without a following newline (with `echo -n`), it was
getting buffered; and then error messages from `scrapy` to stderr were
being erroneously painted with the color intended for the message
"Testing links in documentation...".
This adds UI fields in the bot settings for specifying
configuration values like API keys for a bot. The names
and placeholder values for each bot's config fields are
fetched from the bot's <bot>.conf template file in the
zulip_bots package. This also adds giphy and followup
as embedded bots.
To toggle email change display, replace display = None
to disabled = true.
Email field shouldn't be removed from settings, it should only
disabled if email changes are disabled in realm.
This endpoint is about to become an API-style route and have the legacy
decorator removed from its view. Other endpoints will be used in tests
instead of it.
* For strikethrough formatting: Slack's '~strike~' to Zulip's '~~strike~~'.
* For bold formatting: Slack's '*bold*' to Zulip's '**bold**'.
* For italic formatting: Slack's '_italic_' to Zulip's '*italic*'.
* For mentioning formatting: Slack's <@slack_id|short_name> to Zulip's @**full_name**.
* Checking links.
load_bot_config_template(bot) parses the <bot>.conf
template file, which can be found in the zulip_bots
package for each bot. It then returns the INI content
of that file as a dict.
Currently the zip file is extracted to the root of
zulip directory no matter where the the zip file.
The extracted data is not useful after running the command
which pollutes the zulip directory. It make more sense to
extract it to the same directory of zip file especially
when the zip file gets downloaded to /temp like in the tests.
The autoreload code of Django works by looping over the files associated
with all the loaded modules. This loop is run after every 1 second. If
the file is found for the first time by the loop, it is assumed that the
file is new and is not modified between the time it is loaded and is
checked by the loop. This assumption is the source of a race condition.
We can either implement a more sensitive version of the loop or we can
just allow enough time to the Django loop to touch every file at least
once.
For the time being, we are going with the second option.
Previously, there were following problems with the implmentation:
* Same file handle was being used to read and write. We used to do
`seek(0)` and then `read()`. This had a chance to overwrite
file data. Now we use different file handles to read and write data.
* We were using text streams. Text streams cannot be used with
`bufferring=0`. Now we use binary streams without buffering so that
data is available for reading without any delay.
This commit also updates the key(s) that we search in the logfile.
Previously, launch of all queues was announced in the log, now we only
anounce the number of threads that were launched.
This commit also makes sure that we always exit after gracefull shutting
down the development server.
Before this change, we were pulling iago's credentials from
the wrong database, which usually was a non-issue (dev and test
are populated the same way), but which would break things if
you modified iago's credentials in dev.
Now the credentials properly come from the test database.
For public stream, always show subscription option.
For private stream, if user is subscribed display unsubscribe option.
If user is not subscribe, do not display subscription option.
Only show edit option for stream name and description if user is admin and
either stream is public or stream is private and admin is subscribed to
private stream.
As per backend restrictions for editing stream name/description.
This commit adds a setting to limit creation of generic bots
to admins for realms that want that restriction. (Generic
bots, apart from being considered spammy on some realms,
have less locked down permissions than webhook bots).
Fixes#7066.
This mostly moves code from server_events_dispatch, but
it also de-duplicates the logic in emoji_success(), and
it does early-return for the "text" case to keep the
function a bit flatter.
We no longer have a special UI setting and model
field ("emoji_alt_code") for saying users want text-only
emojis. We now instead make "text" be a fifth choice
for "emojiset".
Fixes#7406
Previously, there were following problems with the implmentation:
* Same file handle was being used to read and write. We used to do
`seek(0)` and then `read()`. This had a chance to overwrite file
data. Now we use different file handles to read and write data.
* We were using text streams. Text streams cannot be used with
`bufferring=0`. Now we use binary streams without buffering so that
data is available for reading without any delay.
This commit documents the get-streams endpoint, which can be
by making a GET request to /api/v1/streams.
Note that in the code examples, JavaScript is missing an example
for how to pass in the `include_` query parameters. That is
because zulip-js's client.streams.retrieve function doesn't take
any arguments and zulip-js does not export any function equivalent
to client.call_endpoint in python-zulip-api/zulip.
The JS module we ask our users to install in the installation
instructions is zulip-js, not zulip. These examples would fail
with the latter. This commit updates the examples to use the
name zulip-js.
This commit does the following:
* Move the Arguments table data from stream-message.md and
private-message.md to a JSON file.
* Add a Markdown extension that allows one to include and render
a table from a JSON file like so:
{generate_arguments_table|arguments.json|private-stream.md}
* Use Bootstrap's .table class to format the table instead of
relying on custom CSS.
The JSON response for when an invalid API key is used to initiate
an API call seems to be identical in every case, so this macro
can be reused all the time.
The JSON response for when an API call to send a message is
successful is the same for both private and stream messages, so
these macros may be used again.
This commit adds the following:
* A table specifying the arguments that are required for this API
call.
* Examples of JSON responses.
This will help us in obsoleting api_endpoints.html.
This commit adds the following:
* A table specifying the arguments that are required for this API
call.
* Examples of JSON responses.
This will help us in obsoleting api_endpoints.html.
This commit splits usage.md into two separate docs,
stream-message.md and private-message.md. The arguments and return
values for sending a stream message are somewhat different from
those of sending a private message, so it made sense to split the
two up for clarity.
There might be case that NOTIFICATION_BOT is none, so before sending stream
announce notification, check first if settings.NOTIFICATION_BOT is not none.
I am increasing the threshold even more for when
codecov complains. Since the tool reports so many
false negatives, we should probably change our philosophy
on code coverage so that this tool only reports
catastrophic commits (like .e.g. a commit that prevents
running a whole test suite) and let our more specific
tools continue to catch things like regressions of files
from 100% coverage.
This uses the correct regex for strikethrough. Also, added
a test to make sure that strikethrough works when it contains
link with whitespace.
Fixes#7596.
We join extra text message for notification and xhr response text by ':'.
Before joining, check if xhr response text already include ':', to
prevent notification with more than one ':'.
A `None` value is not properly handled in this function, which
indicates some lack of testing or a recent regression we don't
understand. We were getting lots of tracebacks from this line
of code on our test server:
mentioned = 'mentioned' in flags and 'read' not in flags
This change has a few cleanups:
* We early-return on last_msg_container === undefined
to make the function flatter.
* We avoid comparing two boolean values for equality,
which can be a landmine if one of the values is
`undefined`, which is falsy but not actually `false`.
* We extract some local variables for readability.
* We make the conditions for subscribe/unsubscribe
more explicit.
This fixes a bug where this used to happen:
* Alice has not read a message
* Bob edits the message
* Alice immediately reads the message
* Bob's edit arrives to Alice and sets her
message status back to unread
Essentially, the root cause of the bug is that we update
message.unread for edits, possibly from stale data, even
though Alice has more current info about reading the message.
This is the final fix to that scenario. There were some
aggravating factors that widened the race window which were
fixed in earlier commits.
Fixes#6248
We were queuing up individual messages to be flagged as
read on the server before this change, and we used debounce
to avoid sending individual POSTs, but this created delays
that were ripe for race conditions.
Now we batch them in the caller and use throttle instead.
This now prevents us from slamming the server with lots of
individual requests, without as many opportunities for races.
(Note that we still have some possibility of race conditions,
but they should be rare now, and other commits will address
some of the other contributors to read/unread glitches.)
Once we convert message.flags to more specific boolean attributes
like message.mentioned and message.alerted, we should get rid of
the `flags` attribute, as it will only confuse debugging.
We no longer set message.flags in the local echo path.
In the markdown parsing step, we just set message.mentioned
directly.
And then we change `insert_new_messages` to no longer
convert flags to booleans, and move that code to only
happen for incoming server message events.
We want to call `set_message_booleans` as soon as we
get data from the server, to avoid confusion about whether
`flags` is the authoritative field.
This commit has callers to `add_message_metadata` call
`set_message_booleans`.
This also sets us up to **not** call `set_message_booleans`
in the local echo codepath, where we can just have the
markdown processor set booleans natively.
In all cases the value of `flags` we were passing in was
actually `message.flags` (although it was slightly obscured in
one place), so now we just pass in `message`.
(We also move a tiny bit of defensive code to set `flags`
into `set_message_booleans`.)
In the JS code, we now use `message.unread` universally as
the indicator of whether a message is unread, rather than
the `message.flags` array that gets passed down to us
from the server.
In particular, we use the unread flag for filtering when
you search.
A lot of this commit is just removing logic to add/remove
"read" from `message.flags` and updating tests.
We also explicitly set `message.unread` to `false` inside of
`unread.mark_as_read()` and no longer have `unread.set_flag()`.
(Some of the callers to `unread.set_flag` were also calling
`unread.mark_as_read`, which was updating the `message`
object, so now we just have `unread.mark_as_read` update
the `message` object. And then unread_ops.mark_all_as_read()
was already calling unread.declare_bankruptcy().)
This adds two similar functions to simplify
our batch processing of unread messages.
unread.get_unread_messages
unread.get_unread_message_ids
They are used to simplify two functions that loop
over messages. Before this change, the functions
would short circuit the loop to ignore messages
that were already read; now they just use the
helpers before the loop.
We now make direct calls to `send_flags_update` for
seting collapsed/uncollapsed, rather than going through
the `batched_updaters` mechanism.
Also the methods in `message_flags` to set
collapsed/uncollapsed now take a single message.
We now do all of the main logic for starring/unstarring
a message in `message_flags.toggle_starred`:
* mark the message as read (just in case)
* update the UI (i.e. the green star in the message)
* update the server
The calling code in both the click handler and the hotkey
handler remains simple--they just handle minor details like
finding the message and clearing popovers.
For updating the server, we now call the new
`send_flag_update` helper.
And we continue to delegate some of the logic to
`ui.update_starred`, but we remove some code there that's
now pushed up to `message_flags.toggle_starred`.
This change should be mostly transparent to users, but it
does remove some inconsistent behaviors between the click
handler and the hotkey handler. Before this change, the
click handler was more aggressive about updating the UI
and marking the message as read. For people using the "*"
key to star/unstar, they probably would only have noticed
different behavior on a slow connection or in an edge
case scenario where only half of the message was onscreen.
More importantly, by simplifying how we talk to the server,
this eliminated up to a one-second lag due to the debounce
logic in the batch_updater code. The complicated debounce
logic is only really needed for batch-updating "read"
messages, and it was overkill and sluggish for starring
messages.
Last but not least, we add defensive code for the local
echo case. (Users have to wait till the message gets acked
to star it.)
Wait until the server acks a message before we enable
the message popover menu. This prevents a whole class
of bugs related to re-drawing the message and changing
the message id, and it also makes room for a little
spinner in the future.
Users with decent internet connections will generally
get server responses before they can click on the
chevron or hit esc/i, anyway.
Trying to collapse a locally echoed message is a rare
thing to do, and it was buggy before this due to races
between the server acking the original message and the
user flipping the collapsed flag.
We now calculate `should_display_collapse` and
`should_display_uncollapse` in the JS code and simplify
the template by eliminating all the inline if/else
logic.
(Note that we are about to disable the message menu
altogether for locally echoed messages, so this change
is partly future-proofing for when we put the menu
back for more innocuous commands.)
Adds type "embedded bot" to bot creation menu. Lets
users select a bot to run from a list of bots.
Currently, this list is hard-coded into the backend.
This diff is nothing but dedentation -- it's empty under
`git diff -b`. These with-statements are only needed for
a pretty narrow scope of code, so make that clear in the
source.
There are two different things you need to patch in order to get error
emails (at `/emails`) in dev. Flip one of them in dev all the time,
and make the comment on the other a bit more explicit.
When I added this "Deployed code" feature to the error reporting,
I apparently hadn't worked out enough of how this code works to
realize that `notify_server_error` may be in a different process,
at a different time and potentially even on a different machine
from the actual error being reported.
Given that architecture, all the data about the error must be computed
in `AdminNotifyHandler`, before sending the report through the queue,
or else it risks being wrong. The job of `notify_server_error` and
friends is only to format the data and send it off. So, move the
implementation of this feature in order to do that.
(@showell added some "nocoverage" directives here for code that
is hard to test (exceptions being thrown, deployment files not
existing) and that was originally part of a file that didn't
require 100% coverage)
This helps prevent them from diverging and getting different sets of
features and fixes. As a bonus, the email path gets a nice tweak that
the Zulip path has had for years, since f7f2ec0ac, which makes the
emails clearer and less broken-looking when logging a message with no
stack trace.
This deduplicates a little bit of logic, and also has us always put
things into `report` the same way.
Empirically an exception in this codepath is very rare, so we won't
complicate the code by trying to salvage a lot of partial information
if it happens -- just log the traceback, and try to get a minimal
notification sent of the bare fact this happened.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email. I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
This is just a basic Dropbox webhook integration. It just
notifies a user when something has changed, it does not
specify what changed. Doing so would require storing data,
as Dropbox API was created mainly for file managers, not
integrations like this.
Closes#5672
We have shifted to a generic queue to send all the emails. This queue
can retry in case of network issues; this makes sure that the emails are
always sent.
This commit just copies all the code from MissedMessageSendingWorker
class to a new EmailSendingWorker class. All the logic to send an email
through a queue was already there. This commit only makes the logic
generic. It does so by creating a special purpose queue called
'email_senders' to send any type of email. To make
MissedMessageSendingWorker still work we derive it from
EmailSendingWorker. All the tests that were testing
MissedMessageSendingWorker now run against EmailSendingWorker.
Previous versions of these links were removed earlier today because
Docker upstream had broken them.
To see what the links had been intended to point to, I used the
Wayback Machine (web.archive.org/web), the super-handy project of the
Internet Archive. These are the current equivalents.
The function should execute initially and return a function, not
be a function that when executed returns another function. This
fixes an existing bug.
Currently, users are warned when mentioning @all and @everyone, but not
when posting on the #announce stream. Confirm with users that they want
to send their message on #announce if over 60 people are going to be
notified.
Fixes#6928.
We should move to a design that doesn't require so many success messages,
since each one carries translation risk (e.g. what if the translation is too
long/overflows, or is not a clear translation, etc.). But doing a quick fix
until then.
We get the following error (edited slightly):
Dec 19 06:13:27 commit_messages| An error occurred while executing
'/usr/bin/git rev-list --max-count=-1 upstream/master..HEAD':
fatal: ambiguous argument 'upstream/master..HEAD':
unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
We'll need to adjust the remotes to make `upstream` mean what we expect.
This isn't really the right way to do this -- commit
dcd80e665 "travis/setup-backend: Remove the '--travis' flag"
took us in the wrong direction by introducing more magic, deeper in
the stack.
But it's the same way we do it for Travis. For now, just copy that.
[Thanks to hackerkid for cleaning up my original crude hack.]
At this point if we were accidentally using `/srv/zulip-venv` for
anything, we'd have run into it by now. So just drop the bit of
historical logic that we had to ensure that.
We introduced this a couple of commits ago (in #7782) for use in the
tests only. Putting it here avoids pulling in a new dependency in
production which we don't use there.
Such payloads are generated when a GitLab repository has merge
request approvals enabled and a project member approves a merge
request. Approving is not the same as merging.
Previosly, a pointer cursor would display when hovering over some of the
headings in the API docs, however nothing would happen when clicked.
Fixes#7702.
This commit ensures that size of each graph is the same before and
after the data is loaded. It also adds a loading indicator to each
graph until the data is loaded.
Fixes https://github.com/zulip/zulip/issues/6490
This applies the same overflow changes to “.message-info-popover” by
imposing a max width on the popover (so it doesn’t increase in width
when it gets larger) and adding the tooltip on hover.
This reverts commit 66261f1cc. See parent commit for reason; here,
provision worked but `tools/run-dev.py` would give errors.
We need to figure out a test that reproduces these issues, then make a
version of these changes that keeps that test working, before we
re-merge them.
This reverts commit 620b2cd6e.
Contributors setting up a new development environment were getting
errors like this:
```
++ dirname tools/do-destroy-rebuild-database
[...]
+ ./manage.py purge_queue --all
Traceback (most recent call last):
[...]
File "/home/zulipdev/zulip/zproject/legacy_urls.py", line 3, in <module>
import zerver.views.streams
File "/home/zulipdev/zulip/zerver/views/streams.py", line 187, in <module>
method_kwarg_pairs: List[FuncKwargPair]) -> HttpResponse:
File "/usr/lib/python3.5/typing.py", line 1025, in __getitem__
tvars = _type_vars(params)
[...]
File "/usr/lib/python3.5/typing.py", line 277, in _get_type_vars
for t in types:
TypeError: 'ellipsis' object is not iterable
```
The issue appears to be that we're using the `typing` module from the
3.5 stdlib, rather than the `typing=3.6.2` in our requirements files,
and that doesn't understand the `Callable[..., HttpResponse]` that
appears in the definition of `FuncKwargPair`.
Revert for now to get provision working again; at least one person
reports that reverting this sufficed. We'll need to do more testing
before putting this change back in.
This reverts commit 5a1869901. I reviewed this change and decided not
to merge it yet, but then I did anyway <_< >_>, along with some others
in the same PR.
This line was added in e8ab7cd1a as a desperate measure to get the
`set_loglevel` helper to successfully suppress the `zulip.send_email`
logs.
In fact, the reason that just setting `level` there wasn't doing the
job had nothing to do with `propagate`, which doesn't interact with
`level`. (See the long block comment in `zproject/settings.py`,
searching for "Python logging module", for my attempt at concisely
explaining these semantics.) Rather, our setting for `level` was
getting clobbered by the use of `create_logger`; and setting
`propagate` to False worked by completely suppressing all logs to
`zulip.send_email` and descendants from reaching the normal handlers,
regardless of the log messages' levels.
Now that the `create_logger` issue is fixed (see a few commits before
this one), drop that.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before. Its one real function is to
send logs to a specific file.
So, pull out that logic to an appropriately-named function just for
it. We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
From the docs:
> This function does nothing if the root logger already has handlers
> configured for it.
Which we do if we've started up Django and configured settings, and in
particular allowed Django to process `settings.LOGGING`.
So, cut it out -- all it can do is confuse people about how logging
works.
If we ever actually used the `log_format` parameter, this would be
doubly confused, because only the first call would have any effect.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`. Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels. Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.
The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
Adds a markdown preprocessor that finds ordered lists where all items
use the same number and change them to be in normal increasing order,
starting with that number.
Fixes#5159.
I've wanted this when looking at a tab from the day before.
Also provides the date and time in UTC, which is handy for
interpreting some of the data.
Pretty sure this is not the world's cleanest way to do this in the
front-end code. It'll do for now.
Not only does this look better, but in the parallel case of ≤
(coming soon) it prevents a naive parser, like in our linter,
from getting confused by seeing a "<" character.
These aren't of much use without the corresponding secrets (despite us
calling one of them a "key", both are public IDs), which are in my
personal dev-secrets.conf. So, remove them.
I had this commit applied in order to test what became 90944983f...
and then I accidentally included it in what I pushed, as 0fa9a489d.
Oops.
When a user with no account tries to login using GitHub/Google auth
they are given an option to register. If the user chooses to
register, a mail is sent for verification and the user has to continue
registration as if they signed up using email. This is confusing and
a bit annoying, so instead of sending email I have changed the
procedure to just redirect to /register where they can start
registration over and proceed normally.
An ideal implementation for this would be to just allow users to
continue registration without email or going back to register.
[greg: tweaked to keep existing user-facing text]
Eventually this check for the realm will be done in get_object_from_key
itself. Rewriting this to fit the pattern in get_object_from_key.
No change to behavior.
Commit d4ee3023 and its parent have the history behind this code.
Since d4ee3023^, all new PreregistrationUser objects, except those for
realm creation, have a non-None `realm`. Since d4ee3023, any legacy
PreregistrationUsers, with a `realm` of None despite not being for
realm creation, are treated as expired. Now, we ignore them
completely, and remove any that exist from the database.
The user-visible effect is to change the error message for
registration (or invitation) links created before d4ee3023^ to be
"link does not exist", rather than "link expired".
This change will at most affect users upgrading straight from 1.7 or
earlier to 1.8 (rather than from 1.7.1), but I think that's not much
of a concern (such installations are probably long-running
installations, without many live registration or invitation links).
[greg: tweaked commit message]
This reverts commit eb2bdb706 "sidebar: Narrow to latest topic if
not in stream." On a trial deploy, many users were surprised and
preferred the old behavior.
This reverts commit aebf63c28 "sidebars: Change to more muted
hover/active colors."
In addition to the effects described in its commit message, this
commit gave the unread counts in the left sidebar a much lighter
background, making them harder to see. I'm not quite sure which part
of this change caused that effect how; reverting for now and we
can sort out a new version of the change.
Audible notifications is a confusing term, especially since it could
plausibly mean audible notifications on your phone.
Mobile push notifications is also not great, since most people don't know
what push notifications are.
I removed "receive" from stream settings strings since I think it isn't
necessary, and so that the strings (and translations) would be the same as
what's on the settings pages.
This fixes compose.test_video_link_compose_clicked to just
use a stub for compose_ui.insert_syntax_and_focus.
It also adds direct tests for compose_ui.insert_syntax_and_focus.
Fixes#6362
We should omit these for mypy. For most class definitions,
mypy doesn't need `Any`, and it provides no real useful info.
For clever monkeypatches, you should provide a more specific
type than `Any`.
We now narrow to the latest topic in stream if we are narrowing from
outside the stream, and show all topics grouped together (previous
default) if we are already narrowed to the stream.
Fixes#7555.
This removes the underlines on the stream names and user names when
hovered over for a cleaner look. This is acceptable due to the fact
that hover is signified already by a light grey background.
The original logic is buggy now that emails can belong to (and be
invited to) multiple realms.
The new logic in the `invites` queue worker also avoids the bug where
when the PreregistrationUser was gone by the time the queue worker got
to the invite (e.g., because it'd been revoked), we threw an exception.
[greg: fix upgrade-compatibility logic; add test; explain
revoked-invite race above]
This code changes frequently enough that errors are bound to creep in. The
main change is that this sends the original invitation email instead of the
reminder email, but I think that's fine.
This was accidentally changed due to an inheritance problem of
`.typeahead.dropdown-menu a` overriding the default bootstrap
selector `.dropdown-menu .active > a` with its color styling
preferences.
This removes the width attribute that actually makes the notification
about a pixel too long, because removing this with a block element will
automatically make it 100% auto fit.
This makes all the alerts in the compose box compatible with dark mode
by choosing different colors and fixing borders to be properly
pronounced, along with removing text shadows that make text unreadable.
On the warning that occurs when you are cross posting to a different
stream, move the “x” to be vertically centered and horizontally aligned
with the”x” on other warnings in the compose box.
This makes the bankruptcy modal compatible with dark mode by adding the
`.modal-bg` class to switch it to dark mode, and by setting a darker
background and border color to the modal header.
This fixes and adds to the logic in commit `525e8e3`. That commit
would only have the stream be the correct color if it was active,
but really it should be dark text by default regardless of whether
it is the active tab bar list item.
These commands are super boring standard Docker commands,
so this probably isn't helpful for anyone who is familiar
with building Docker images... but I had to consult docs
to work out the right commands again today, so they'd help me.
This new base image just comes from me doing a fresh `docker build`
and uploading the result. That gives it fresher versions of our
APT dependencies than the image I built a couple of months ago,
speeding up provision by giving it less to install.
Provision was failing at tools/setup/postgres-init-dev-db
with this in the log:
Sep 20 02:27:01 + sudo -i -u postgres psql ''
[sudo] password for circleci:
The issue is that the old version of this line (from Circle upstream)
only lets the `circleci` user sudo to root -- not to other users, or
not directly anyway -- because sudoers syntax is complicated. Fix it,
after studying `man sudoers`.
This is only an optimization -- if this list is missing anything,
we'll get to it in `provision` in the actual build. That's important,
because we want an existing image to work fine for testing new
versions of our codebase, including changes that may install more
packages in `provision`.
What this does accomplish is keeping provision's `apt-get install`
fast, by leaving it very little work to do.
The list comes from looking at the APT output during provision in an
actual run without this step, and leaving out two packages which
aren't available at this stage, because we get them from PPAs:
postgresql-9.3-pgroonga and postgresql-9.3-tsearch-extras.
Install `jq` with APT -- that's a lot simpler to read than this
explicit download.
And coalesce several commands, following Docker upstream's
recommendation and avoiding unnecessary overhead.
From just my experience attempting this change, I've been impressed
with the very fast turnaround in CircleCI for a change -- from `git
push` to the build starting is usually just a couple of seconds.
It also looks like we get much clearer control over the caching
system, and the setup of the environment via a Docker image of our
choice, which can help avoid most of the several minutes Travis takes
to set up our environment before it actually starts running the tests
themselves.
Plus it supports fine-grained parallelism, e.g. by test file:
https://circleci.com/docs/2.0/parallelism-faster-jobs/
So we can probably get much faster builds by making this switch.
Because the environment is much more transparent and even under
our control, I'm also hopeful for more reliable builds.
This is nearly the same as Circle's version, linked in the comment.
I've
* changed the FROM line to get Ubuntu,
* added a couple of distro packages to compensate, and
* revised the comments.
Substantively, this makes the table more readable by grouping users
into expanding sets by level of activity: active in last day, active
in last week, have an account at all. The class "active in last week",
as opposed to "active in last week but not in last day", makes more
natural comparisons both between realms and for one realm through time,
and it's less sensitive to the details of our definitions.
This also makes the terminology more standard. We already made that
change in the display, in the previous commit; as we go through the
logic here, we adjust the terminology in the code too.
This seems to have been causing the travis production suite to fail.
It's a direct consequence of removing travis' giant library of apt
sources.list files; now that those are gone, there aren't copies of
all these extra packages available anyway.
This makes the typeaheads dark-mode compatible by changing to the
background to be dark and the text to inherit from the body text
(rather than bootstrap’s default of #333).
The streams can be light and if it inherits the white text color they
will not be readable. It should default to dark text with the exception
of when the tab is is `.dark-background`, in which case then it should
revert to inheriting the white color.
This makes the edit history overlay dark mode compatible by changing
the background to the dark blue along with changing the highlight
colors to work with white text and dark backgrounds.
This fixes most of the current Travis CI failures, and should help
prevent future ones from cropping up, since we're just deleting Travis
CI's giant library of apt sources up front.
The production build is still failing, since apparently some of the
packages we were doing `apt-mark hold` on no longer exist.
This adds custom CSS through JavaScript for things that do not
scope well and will override other inherited styles.
This should ONLY be used for problematic CSS that has no obvious
or easy CSS-only solution.
(Specifically, we need this for the "default link" styling, which is
hard to override because we don't want to start winning ties due to
specificity that we would not have won in the light theme).
We'll need the expanded test coverage when we move
check_prereg_key_and_redirect to zerver/views/registration.py to avoid
test failures, and these are also tests we should really have anyway.
The previous code gave the user an extra day past
REALM_CREATION_LINK_VALIDITY_DAYS. Also rewrote it to match the parallel
logic in get_object_from_key.
Empirically, the retry in `_on_connection_closed` didn't actually work
-- if a reconnect failed, that was it, and the exception handler
didn't get run. A traceback would get logged, but all its frames were
in Tornado or Pika, not our own code; presumably something magic and
async was happening to the exception.
Moreover, though we would make one attempt to reconnect if we had a
connection that got closed, we didn't have any form of retry if the
original attempt at connecting failed in the first place.
Happily, upstream offers a perfectly reasonable bit of API that avoids
both of these problems: the on-open-error callback. So use that.
This method was new in Tornado 4.0. It saves us from having to get
the time ourselves and do the arithmetic -- which not only makes the
code a bit shorter, but also easier to get right. Tornado docs (see
http://www.tornadoweb.org/en/stable/ioloop.html) say we should have
been getting the time from `ioloop.time()` rather than hardcoding
`time.time()`, because the loop could e.g. be running on the
`time.monotonic()` clock.
Adding it afterward is inherently racy, and upstream's API is quite
reasonable for avoiding that -- just like we can pass an on-open
callback up front, we can do the same with the on-close callback.
This is a more thorough version of 4adf2d5c2 from back in 2013-04.
The default value of this parameter is already False upstream.
(It was already False in pika version 0.9.6, which we were
supposedly using when we introduced this in 4baeaaa52; not sure
what the story was there.)
This makes the gear-menu icon translucent rather than medium grey and
black (which isn't even our base text color) to be half opaque (approx)
and base text color.
This makes the compose icons translucent rather than medium grey
and black (which isn't even our base text color) to be half opaque
(approx) and base text color.
This changes it to be compatible with the dark-mode which involves one
change to light mode of changing a grey to a translucent white in
reactions.css.
There isn’t any reason for either of these mention code paths to be
using specified background types, so convert them all to background so
inheritance is more simple.
For setting the night mode, it checks to see if you’re in development
by checking `page_params.development`, but the actual attribute is
`page_params.development_environment`.
Previously, we weren't doing a proper left join in
user_groups_in_realm_serialized, resulting in empty user groups being
excluded from the query. We want to leave decisions about excluding
empty user groups to the UI layer, so we include these here.
This solves the issue with typeahead appearing in the middle of an
already-completed typeahead word.
Example: Earlier, '@ran|dom' would also trigger the typeahead and show
'random', but selecting it would turn it into '@**random** dom'.
We still have a problem to solve of preventing typeahead from
appearing on a space in the middle of an already-completed typeahead
word, but that is its own independent bug.
These are the exceptions to the rule that our queues correspond to
queue-processor workers.
Purging `notify_tornado` in particular is a useful workaround right
now for some error spew in the dev environment.
Because we use access_stream_by_id here, and that checks for an active
subscription to interact with a private stream, this didn't work.
The correct fix to add an option to active_stream_by_id to accept an
argument indicating whether we need an active subscription; for this
use case, we definitely do not.
The top and bottom sections of the lightbox overlay do not close the
lightbox when clicked. Now, this triggers the close_overlay path when a
valid non-actionable background element is clicked.
Fixes: #7500.
Using user_group_name_dict.get() will return `undefined`.
`blueslip.error` statement caused an exception notification to the
admins.
Tweaked by tabbott to add a test for a nonexistent group.
tools/setup/generate_zulip_bots_static_files now starts off by
deleting static/generated/bots/ (if it hasn't been removed already)
so that outdated static files from older versions of the zulip_bots
package don't supress errors in the main repo that would otherwise
break.
For more info, see #7542Fixes: #7542.
For example, this means that if a user already has an account on one
realm and they try to make an account on another by hitting "Sign in
with Google" (rather than following the little "Register" link to a
"Sign up with Google" button instead), they'll get to make an account
instead of getting an error.
Until very recently, if the user existed on another realm, any attempt
to register with that email address had to fail in the end, so this
logic gave the user a useful error message early. We introduced it in
c23aaa178 "GitHub: Show error on login page for wrong subdomain"
back in 2016-10 for that purpose. No longer! We now support reusing
an email on multiple realms, so we let the user proceed instead.
This function's interface is kind of confusing, but I believe when its
callers use it properly, `invalid_subdomain` should only ever be true
when `user_profile` is None -- in which case the revised
`invalid_subdomain` condition in this commit can never actually fire,
and the `invalid_subdomain` parameter no longer has any effect. (At
least some unit tests call this function improperly in that respect.)
I've kept this commit to a minimal change, but it would be a good
followup to go through the call sites, verify that, eliminate the use
of `invalid_subdomain`, then remove it from the function entirely.
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
As we migrate to allow reuse of the same email with multiple realms,
we need to replace the old "no email reuse" validators. Because
stealing the email for a system bot would be problematic, we still ban
doing so.
This commit only affects the realm creation logic, not registering an
account in an existing realm.
Just now this is largely redundant with `test_signup_already_active`;
but very soon when we allow reusing an email across realms, the logic
will diverge.
The one thing this bit of logic is used for is to decide whether
there's an existing user which is a mirror dummy that we should
activate. This change causes us to ignore such an existing user if
it's on some other realm, and go straight into `do_create_user`.
This code appears to exist to cover a few extra lines in
zerver/lib/digest.py. But it's rather brittle, tucked as it is into
the middle of a different test's loop, and with the upcoming
introduction of the `lear` realm in testing, this test code itself
loses coverage.
For now, rather than fix this test code up just delete it; we don't
have 100% coverage on `zerver/lib/digest.py`, while we do on this test
file, so that avoids breaking coverage in CI. As a followup, we
should add back some logic like this but in a more robust way,
probably as its own separate test method.
This test would produce a bunch of log messages with tracebacks,
complaining that `welcome-bot@zulip.com` tried to send cross-realm PMs
and can't. The issue is that the test overrides
`settings.CROSS_REALM_BOT_EMAILS`, and hasn't kept up with additions
to the normal value for that setting. Update it so that welcome-bot
is permitted as usual.
The `re.match` function in the Python stdlib is a trap for the unwary,
with surprising and asymmetrical semantics; we should probably add a
lint rule to ban it entirely. The docstring says:
> Try to apply the pattern at the start of the string, [...]
In other words, it effectively adds a `^` at the start (or `\A`, where
the distinction matters.) It's bad enough that this differs from what
grep, sed, perl, less, and every other tool I can think of do when
looking for matches to a regex; on top of that, it treats the
beginning of the string differently from the end, for no obvious
reason. The function that does what the rest of the world understands
by "match against this regex" is `re.search`.
In this case, it's unlikely that anyone intended for comments with
URLs, or `api_url` references, to miss out on their respective
exceptions to the long-line rule if they happen to start after the
first column. So fix those rules by just switching to `re.search`
with the same pattern.
I think Markdown URL references may have to start at the beginning of
the line, so I've left a `^` there to preserve -- but now make
explicit -- the `re.match` behavior.
Originally this used signals, namely SIGRTMIN. But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)
So, we listen on a socket. Bit more code, and a bit trickier to
invoke, but it works.
This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".
For usage instructions, see docstring.
We made this change because users often unnecessarily click "Home"
first in their use of Zulip, because it seems appealing. While "All
messages" isn't quite precise (it doesn't include muted streams), it
does describe relatively simply the interleaved view that this
represents.
This commit leaves everything as "home" in the code, and only changes
user-visible strings and docs. Changing the code will be a big project;
there are hundreds of relevant occurrences in variable names, etc.
Further, we'll probably want to convert those various variable names
in different ways.
Tweaked by tabbott to extend the commit message and update a few comments.
In two factor authentication every step adds a unique prefix to the fields,
due to this the name of the form fields differs from the HTML fields. If
we do not do this we will have to change the name in the HTML, which
will cause the change in tests.
We're not yet ready to add 2FA to Zulip yet, but we've determined we'd
like to work from these libraries.
I'm not bothering to bump PROVISION_VERSION for this, since we're
likely to do something else that bumps it soon, and we're not merging
anything that uses these new libraries.
Tweaked by tabbott to move changes from the next commit that are
required for this to pass tests into this commit.
Note that this exports a few items that were not previously exported.
There's one migration required by this release:
* queue_processors: Stop passing state_handler to handle_message.
state_handler is now a property of bot_handler and thus, does
not need to be passed to bot_handler.handle_message().
The commit responsible is:
2a74ad11c5
This completes the last commit's work to fix CVE-2017-0910, applying
to any invite links already created before the fix was deployed. With
this change, all new-user registrations must match an explicit realm
in the PreregistrationUser row, except when creating a new realm.
[greg: rewrote commit message]
We would allow a user with a valid invitation for one realm to use it
on a different realm instead. On a server with multiple realms, an
authorized user of one realm could use this (by sending invites to
other email addresses they control) to create accounts on other
realms. (CVE-2017-0910)
With this commit, when sending an invitation, we record the inviting
user's realm on the PreregistrationUser row; and when registering a
user, we check that the PregistrationUser realm matches the realm the
user is trying to register on. This resolves CVE-2017-0910 for
newly-sent invitations; the next commit completes the fix.
[greg: rewrote commit message]
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
This should help protect us from future issues with the way that
`bulk_get_users` does caching.
It's likely that we'll want to further restructure `bulk_get_users` to
not have this base_query code path altogether (since it's kinda
buggy), but I'm going to defer that for a time when we have another
user.
The previous implementation had a subtle caching bug: because it was
sharing its cache with the `get_user_profile_by_email` cache, if a
user happened to have an email in that cache, we'd return it, even
though that user didn't match `base_query`.
This causes `get_cross_realm_users` to no longer have a problematic
caching bug.
Hides URL if the message content == image url so that sending gifs or
images feels less cluttered. Uses the url_to_a() function to generate
the expected url string for matching.
Fixes#7324.
Appends "Test: " text to some tests to make changes to the image preview
rendering. In the future, if the message is only a link to an image,
the link will be hidden.
This change does a few things:
* I use "early return" to make the code a bit flatter
and easier to comment.
* I added more comments.
* I removed some unneeded passing of `invite_only` into
the template.
We include ERROR_BOT in this set, even though it's not technically
cross-realm (it just lives in the admin realm).
This code path does not correctly handle emails that correspond to
multiple accounts (because `get_system_bot` does not). Since it's
intended to only be used by system bots, we add an appropriate
assertion to ensure it is only used for system bots.
This was causing problems, because internal_send_message assumes that
there is a unique user (across all realms) with the given email
address (which is sorta required to support cross-realm bot messages
the way it does).
With this change, it now, in practice, only sends cross-realm bot
messages.
We now ignore payloads where payload['push']['changes'] is empty,
because an empty push doesn't really convey any useful information.
I couldn't find a way to replicate the action that would generate
such a payload, so I took one of our existing payloads and editted
out payload['push']['changes'] myself, so this payload is not
authentic.
Previously, this was a ValidationError, but that doesn't really make
sense, since this condition reflects an actual bug in the code.
Because this happened to be our only test coverage the ValidationError
catch on line 84 of registration.py, we add nocoverage there for now.
I probably should have just done this in the original implementation;
there's only a small downside in the form of an extra database query
when trying to authenticate a user who doesn't exist.
Long URLs in Markdown macros overflow their containers without this
property. This reverts 0dc91a9148. We
will likely need to fix the original Windows bug that
0dc91a9148 has resolved another way.
This is part of our efforts to change our integrations/webhooks
docs to follow the same sort of numbered-list format as our /help
docs. In order to indicate that paragraphs separated by newlines
are part of the same numbered-list point, every paragraph must be
indented 4 spaces.
This resulted in problems with the desktop app, where clicking these
links would leave the user in a situation where they needed the "back"
button in order to get back to the webapp.
Fixeszulip/zulip-electron#342.
This is intended to fix serious issues with frequent guest crashes on
OSX High Sierra with Virtualbox and the default file synchronization.
Note from Tim: We've tried this idea before, and it worked for Mac,
but caused problems for non-Mac users of our Vagrant setup that
resulted in it being reverted in
1432e9afb0.
Hopefully the new conditional will handle things correctly in both Mac
and Linux.
Apparently, this was checking the wrong path in Travis CI, and thus
never actually running (meaning we'd accumulate every `node_modules`
directory ever in the Travis caches, which in turn resulted in very
slow builds).
This both improves the comment to be more readable, and also uses the
new and improved exclude feature to limit the exclusion to just the
webhook fixtures (where it's needed).
Also fixes a mypy error.
The previous exclude rules only allowed excluding a directory (and
things in subdirectories would silently still be linted). Anyone
using this would expect it to exclude a directory tree, so we make it
do that.
This updates commit 11ab545f3 "install: Set the locale ..."
to be somewhat cleaner, and to explain more in the commit message.
In some environments, either pip itself fails or some packages fail to
install, and setting the locale to en_US.UTF-8 resolves the issue.
We heard reports of this kind of behavior with at least two different
sets of symptoms, with 1.7.0 or its release candidates:
https://chat.zulip.org/#narrow/stream/general/subject/Trusty.201.2E7.20Upgrade/near/302214https://chat.zulip.org/#narrow/stream/production.20help/subject/1.2E6.20to.201.2E7/near/306250
In all reported cases, commit 11ab545f3 or equivalent fixed the issue.
Setting LC_CTYPE is redundant when also setting LC_ALL, because LC_ALL
overrides all `LC_*` environment variables; so skip that. Also move
the line in `install` to a more appropriate spot, and adjust the
comments.
This PR removes unnecessary white background from realm-icon.
Also, I have given padding and border-radius to make it look good.
This also fixes a bug where realm-name didn't show up because of the
white font color.
Sphinx was displaying "WARNING: document isn't included in any toctree"
for files we just don't want in the TOC. We can hide them from the index,
but the rtd theme defaults to display hidden index entries in the nav bar.
This commit excludes these files from such warnings, and patches layout.html
so that hidden index entries stay hidden from the navigation sidebar.
This commit also moves password-strength.md under docs/production and
adds it as a hidden entry in production/index.rst.
Fixes#7417.
The readthedocs theme overrides a few settings in their layout template.
We might want to change some settings back to their default values.
This commit copies the original readthedocs layout file from
https://github.com/rtfd/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html
to _templates/layout.html, and excludes it from lint and template checks.
Addresses #7417.
This buggy logic from e1686f427c had
broken do-destroy-rebuild-test-database.
Now that we're not just trying to add the Recipient objects for every
user on the system here to profiles_by_id, we also shouldn't be
processing every Recipeint object on the system. The fix is simple:
because of the patch we got merged into Django upstream,
recipients_to_create actually has the object IDs added to the
Recipient objects passed into Recipient.objects.bulk_create.
This was missed in manual testing, since it only broke `populate_db
--test-suite`.
An Integration object doesn't need access to the context dict used
to render its doc.md, since the context dict is just passed directly to
render_markdown_path.
Previously, when rendering a single integration, we tacked on the
following information to the context dict that was redundant:
* An OrderedDict containing all of the Integration objects for
all integrations.
* An OrderedDict containing all of the integration categories.
The context dict for rendering a particular integration doc would
contain 4 OrderedDicts, 2 for categories, 2 for Integration objects
because of how many times add_integrations_context had been called.
This was very wasteful, since an Integration object doesn't need
to access any other Integration object (or itself for that matter)
to render its documentation. This commit adds a function that
allows us to only pass in the context values that are necessary.
This is checked for in the caller of OurAuthenticationForm, which
meant this code was never run. But it is worth having an assertion
here to catch any possible regressions.
Structurally, the main change here is replacing the `clean_username`
function, which would get called when one accessed
self.cleaned_data['username'] with code in the main `clean` function.
This is important because only in `clean` do we have access to the
`realm` object.
Since I recently added full test coverage on this form, we know each
of the major cases have a test; the error messages are unchanged.
This deletes the old mock-covered test for this, which was mostly
useless. We have a much less messy test, which we extend to provide
the same test coverage the old one did.
While the result was the same before, this makes it more obvious.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak. While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.
Fixes#7095.
This uses the to-markdown.js library to do all the hard work of
parsing HTML and turning it into markdown and not e.g. uploaded files.
Tweaked by tabbott to better scope when it activates to just include
pastes of HTML content.
Fixes#5853.
We created this redundant pair of conditionals in a preceding commit,
in order to match the indentation of an `except` block so as to slice
the diffs extra finely as we're refactoring auth code.
This lets us delete some duplicate code, since common_get_active_user
handles an account in the wrong subdomain for us.
Also lets us delete the now-unused common_get_active_user_by_email.
This way, we don't attempt to evaluate whether the user's account is
active (etc.) until after we've checked the backend is enabled. This
won't change the result of actual auth, but feels more readable.
This is a behavior change, though we don't check the value in the
caller regardless. It just seems more logical for us to correctly
report to the caller whether the Google auth itself was valid
unconditionally.
This code path was only required because we had remote_user set as a
positional argument here, and thus we'd be running this auth backend's
code when actually using another auth backend (due to how Django auth
backends are selected based on argument signature).
This require some care to ensure we still provide the same nice error
messages for the case of a user who has an account, just not with this
organization.
Also, we fix the fact that the docstring was (and I think always has
been) at best confusing and perhaps even inaccurate.
We'd rather this work be just executed immediately, rather than
queued, since queued events can confuse the queue workers if the
database is dropped and recreated repeatedly.
Fixes#7374. Since we use `table-layout: fixed`, the width of the first
row are applied to all the remaining rows, thus wrapping all the columns
to 20% width. This is fixed by making the selector specific to only
user_rows and not the editable data row.
The first part of this change is to have the "Your bots"
tooltip not lie about creating `.zuliprc`, because it
doesn't put a dot in front of the file.
And then the more significant change here is to make
the "Running a bot" documentation use realistic filepaths,
both in terms of where the download typically puts the file,
and where you want to move it to.
This doesn't touch the main path through the install docs; that will
see a broader rewrite soon as we make outbound email optional for a
nascent server, make the hostname and admin email into mandatory
installer flags, and then radically simplify the instructions by
removing mandatory editing of `settings.py` and folding most of what's
left into the installer.
Pointing these at the latest release, rather than the latest version
in master, allows us to make changes to the installer and document
them properly in master, without making the instructions confusingly
wrong for people who just go to the website or the GitHub repo page
and follow instructions to install.
This commit modifies `test-locked-requirements` to use some caching
so that we don't need to use the `update-locked-requirements` tool
everytime for checking the validity of locked requirements as it is
slow.
Fixes: #6969.
Inorder to provide more explicit error messages I have merged the
`emoji_code_is_valid()` and `emoji_name_is_valid()` functions into
`check_emoji_code_consistency()` and `check_emoji_name_consistency()`
respectively.
The installation admin is not the right person to get support requests from
deactivated users, regardless of the situation.
Also updates the wording to be a bit more concise.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
This often can cause minor caching problems.
Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
This was basically rewritten by tabbott, because the code is a lot
cleaner after just rewriting the ZulipPasswordResetForm code to no
longer copy the model of the original Django version.
Fixes#4733.
This adds a few missing entries to the TOC, which hadn't made sense
back when Zulip's ReadTheDocs didn't have the new collapsing feature.
Tweaked by tabbott to also give the SSL certificates doc an
appropriate title for its new role.
Payloads that don't have a payload['object_attributes']['action']
attribute are generated when GitLab sends a test payload to verify
if the webhook was set up successfully. In this case, we should
send a message notifying that the webhook was configured
successfully.
Instead of populating the context dict with integration-specific
information in render_markdown_path, we now do that in
zerver.views.integrations.integration_doc instead.
Fixes#7401.
Tweaked by tabbott to use cast to handle the typing issues here.
This prevents the caches in /srv from growing to fill up the disk --
e.g., on my laptop after 6 months of regular development the venv cache
was 12G and the NPM cache 5G, making them by far the largest disk hogs
on the machine.
It costs about 0.4s, apart from any time spent actually removing
things. This is a little annoyingly slow to be adding to every
provision, and seems like it could be optimized, but I think already
worth it as is.
This adds tests for a new more cases. Some were already covered
elsewhere in the codebase, but it feels best for LoginTest to fully
cover OurAuthenticationForm.
Now we have moved the `do_auth` function to `SocialAuthMixin`. Instead
of overriding `do_auth`, derived class is now expected to override
`get_authenticated_user`.
`do_auth` now contains code which is expected by all backends.
The character ">" now only starts a blockquote if the resulting
blockquote would be non-empty. Thus, by itself, ">" is now
interpreted literally by bugdown, fixing #687. The message
with contents consisting of ">>>" is now parsed as a doubly
(not triply) nested blockquote with contents ">". Properly
formed blockquotes have identical behavior as before, but now
bugdown can no longer produce empty blockquotes as output.
Fixes#2886, #687.
Ideally, empty messages are never created. However, sometimes even
non empty messages can act as if empty, such as one consisting only
of an opening and closing blockquote tag. This sets the
.message_content to have a minimum height equal to the line height.
This is overridden with zero (the default) in the case of collapse.
Expand bullets "[+]" show up on locals instance, but they
do not show up online on readthedocs. The expand icon is valuable
to indicate that each section has more content.
The problem is readthedocs appends code to conf.py that overrides the
html_theme_options prior to a build. This commit prevents this from
happening so that collapse_navigation remains set to False.
In response to merged PR 7326.
We already do this by default in tools/build-docs, but since we
migrated test-documentation to not run that directly (to disable
collapsing), we need to add the recent parallelism fix here too.
It saves about 5-10s when running this test suite for me, which is
good, but definitely leaves me feeling like there could be more
improvement.
It appears as though the ordering of the overlays in the DOM is
overriding their z-index in Safari on mobile. This moves them up to
the top of the template ahead of the header so that the header will no
longer display above the overlays in positioning.
Fixes: #7248.
This changes the icons to be translucent rather than grey and also
makes them get darker on hover rather than lighter (which didn’t really
make sense) on the normal theme.
This removes the background color declaration on the search bar so it
can inherit the standard input background color.
navbar: Let search bar inherit from base color.
Storage limititations are only set on the value of
a config entry, since this is the only user-accessible
part of the schema. Keys are statically set by each
embedded bot.
In this commit we add new dependencies needed for running thumbor.
Also we add the script for creating the virtual environment ready
for thumbor.
Note: Thumbor will use python2 and thus have different virtualenv
dedicated to it.
Credits to @TigorC and @joshland as well for there work on this.
In this commit we add a new option which could be used to specify
python version. When 'py2' is specified, future/futures are not
removed from the requirements lock file generated.
We don't have our linter checking test files due to ultra-long strings
that are often present in test output that we verify. But it's worth
at least cleaning out all the ultra-long def lines.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.
This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
When the RabbitMQ server disappears, we log errors like these:
```
Traceback (most recent call last):
File "./zerver/lib/queue.py", line 114, in json_publish
self.publish(queue_name, ujson.dumps(body))
File "./zerver/lib/queue.py", line 108, in publish
self.ensure_queue(queue_name, do_publish)
File "./zerver/lib/queue.py", line 88, in ensure_queue
if not self.connection.is_open:
AttributeError: 'NoneType' object has no attribute 'is_open'
During handling of the above exception, another exception occurred:
[... traceback of connection failure inside the retried self.publish()]
```
That's a type error -- a programming error, not an exceptional
condition from outside the program. Fix the programming error.
Also move the retry out of the `except:` block, so that if it also
fails we don't get the exceptions stacked on each other. This is a
new feature of Python 3 which is sometimes indispensable for
debugging, and which surfaced this nit in the logs (on Python 2 we'd
never see the AttributeError part), but in some cases it can cause a
lot of spew if care isn't taken.
Even where this is actually used for a temporary checkout, it obscures
the relationship between this and $TMPDIR -- and some of our logic
depends on that. In other places, it isn't actually even a checkout.
In all cases, the expanded version is clearer.
This script, and tools/update-prod-static which it relies on,
have kept getting more complex since this conditional was added
in 2013, and the places we rely on GNU features have probably
multiplied beyond `mktemp -d`. It's unlikely this works on
macOS with BSD tools now, and it'd be hard to maintain that way
if it did; drop the pretense.
There's no need to remove this file here -- the whole tree will be
removed a few commands later, and the `tar` command we do first, to
supplement our tarball with various generated files, is quite
selective and wouldn't look at this file anyway.
This updates the developer documentation to recommend `git clone -c`
to set the git pull mode to be rebase by default in the repository
configuration.
The hope is that this will help minimize how often new contributors
end up accidentally messing up their PRs by using `git pull`.
Tweaked by tabbott to fix broken links and optimize the language.
Fixes: #7022.
We got asked about this from a potential user, and they seemed fairly
excited and confused by it. In particular, it wasn't obvious that
deactivating a user was the feature they were looking for.
With the new portico work we've done, the help documentation does
sorta depend on the database if you're logged in. So it's best to
just require it for these tests.
This should get test-documentation and test-help-documentation passing
again after we moved everything around in the last commit.
There are a lot more absolute links (generally in code comments) to
update, but they are lower-priority and can be done in a follow-up
pass.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.
This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.
Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).
Fixes#5265.
While fixing an issue related to email gateway messages not getting
rendered properly, I unknowingly introduced a bug in the markdown
engine updation code. This commit fixes it. The issue was that for
a realm having email gateway setup, updation of realm filters would
lead to the updation of only one of the markdown engines not both.
The 'simple' realm was super broken and confusing for new users. We
should replace this with having an easy way to make a new realm in
development, done properly.
Fixes#6116.
In remove_members_from_group_backend, we are passing user group to
remove_members_from_user_group. In remove_members_from_user_group,
expect user_group_id.
The script already won't work without them; so if the user gets the
invocation wrong, give a halfway-reasonable error rather than just
crash into the ground.
This allows the installer to continue using this script for the
`standalone` method, while the no-argument form now uses the same
`webroot` method as the renewal cron job, suitable for running
by hand to adopt Certbot after initial install.
Certbot replaces the cert files under /etc/letsencrypt/live/,
which our nginx config refers to symlinks to; but it doesn't
tell nginx there's been an update, so nginx keeps serving the
old cert.
This is fine as long as nginx is restarted, or just told to
reload its config, at some point before the cert actually
expires about 30 days later. Which is probably the common
case, but of course we should make it just work. So, if we
actually renew a cert, tell nginx to reload its config now.
This causes the cron job to run only when a Zulip-managed certbot
install is actually set up.
Inside `install`, zulip.conf doesn't yet exist when we run
setup-certbot, so we write the setting later. But we also give
setup-certbot the ability to write the setting itself, so that we
can recommend it in instructions for adopting certbot in an
existing Zulip installation.
This helps make this script suitable to run on existing installations,
by mitigating any worry about clobbering existing certs with links to
the new ones, in case the admin changes their mind or was using the
certs for something else too.
If we were making an old-fashioned webroot where hand-written static
HTML files went, somewhere under `/srv` would be most appropriate.
Here, this webroot is really more of an implementation detail of the
certbot set up by the Zulip installer/packaging, containing transient
state. So someplace under `/var` is appropriate, and specifically
under `/var/lib/zulip` in order to properly namespace it.
For background on `/var/www` and friends, see the top couple of answers
on
https://unix.stackexchange.com/questions/47436/why-web-server-var-www
This fixes a regression in ae5ba7f4fd,
where Zulip would 500 if the newly added system bots didn't exist on
the server.
This also fixes a moderate size performance problem where we'd fetch 5
users from memcached or the database in a loop.
Note from tabbott: While this initial version is experimental and
definitely incomplete, we expect to have a solid version done over the
next few weeks (after more refactoring). We're merging this now to
make it easy to test both versions when refactoring our CSS.
Fixes#267.
This fixes a regression in 25c669df52.
We were draining the queue in both the superclass and the subclass,
so by the time the subclass started processing events, there were
no events to process. Now the subclass properly uses the events
passed in from the superclass.
These are new:
new-user-bot
emailgateway
Our cross-realm bots are hard coded to have email addresses
in the `zulip.com` domain, and they're not part of ordinary
realms.
These have always been cross-realm, but new enforcement in the
frontend code of all messages having been sent by a known user means
that it's important to add these properly.
The warning here means that the admin can't really act on this yet if
they want to disable email auth, which is likely among admins that
want to make any changes here. And for admins who don't, this is an
extra thing to read and make a decision about before they can get a
server running. See #6985.
Conversely, we already discuss auth backends right at the top of the
`prod-customize` doc, which is linked under "Next steps" at the end of
these instructions.
The warning about EmailAuthBackend is important; but we can move it to
the config file right next to the setting, and then it's available
right when it's actionable, which is if the admin is actually thinking
about changing the setting.
LXC on Debian does not ship with a default network config for
containers, so we now recommend setting up the lxc network config
manually.
Tweaked by tabbott to make these details much harder to miss.
The CSS linter was pretty hard to reason about. It was
pretty flexible about certain things, but then it would
prevent seemingly innocuous code from getting checked in.
This commit overhauls the pretty-printer to be more composable,
where every object in the AST knows how to render itself. It
also cleans up a little bit of the pre_fluff/post_fluff logic
in the parser itself, so comments are more likely to be "attached"
to the AST node that make sense.
The linter is actually a bit more finicky about newlines, but
this is mostly a good thing, as most of the variations before
this commit were pretty arbitrary.
This restyles and rewords some of the emoji style section to look
better and fit it more with the current style guide.
Tweaked by tabbott to modify the historical migration rather than
adding a new one. This is OK because the emojiset choices text change
doesn't touch the database; it's just a Django Python code thing.
Also removed translation tags, since we don't need them for a set of
brand names.
The intended use of $$ is for inline expressions, not for multiline
ones; ```math is an acceptable alternative for the latter. Hence,
the $$-syntax for inline TeX no longer permits newlines within it.
This was also necessary for the next change to be sensible; namely
allowing for spaces around both $$ when crafting inline TeX instead of
forcing everything to be crammed together, e.g. $$x=7$$. In order to
avoid uninentionally creating inline expressions, the opening and
closing $$'s of an inline expression must now both exactly consist of
two dollar signs, no more and no less.
Fixes: #6488.
Some of the calls in our codebase will call for the `.keys()` method a
lot, so this caches the results and returns them rather than mapping
the array if it knows the state to be the same as before.
If a user clicks on the pill container and not directly on a pill
or on an "x" inside of a pill, it should be presumed they are
trying to input new text. This effectively makes the whole non-pill
section an input bar.
The os.mkdir call is straightforward and doesn't testing.
Workers relying on LoopQueueProcessingWorker are tested
with its consume method that exists solely for this purpose.
Previously, these push notification events were being generated, but
then ignored in handle_push_notification because there was no
user_message object.
This should help make it easier to add tests coverage for these queue
processors, since they now use a system more similar to the other
queue processors.
In python3 base64.b64decode() can take an ASCII string, and any
legit data will be ASCII. If you pass in non-ASCII data, the
function will properly throw a ValueError (verified in python3 shell).
>>> s = '안녕하세요'
>>> import base64
>>> base64.b64decode(s)
Traceback (most recent call last):
File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 37, in _bytes_from_decode_data
return s.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 83, in b64decode
s = _bytes_from_decode_data(s)
File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 39, in _bytes_from_decode_data
raise ValueError('string argument should contain only ASCII characters')
ValueError: string argument should contain only ASCII characters
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.
Fixes: #7045.
This now allows all tab switcher components to be used with left
and right arrows, given that a tab is already in focus, which is
the default behavior unless overridden (like on the streams
overlay).
Do not attempt to autoscroll down to view new messages if popovers are
open. This prevents the issue where someone can be viewing a profile or
reacting to a message and not be able to due to a new message coming in.
Fixes: #7319.
This adds a slide class that specifies that the JS actions for sliding
up and down sections is the desired behavior, along with a bit of CSS
to help display correctly in the case of not being a sliding section.
This adds the "--disallow-any=generics" option to run-mypy, which no
longer permits:
- inheriting from "list"; use "List[sometype]" (or a TypeVar)
- generic types with no following square brackets specifying the type
(even if initially 'Any')
Any (and '...' for Callable) is a lot easier to search for than an
absence of square brackets, and should improve overall typing quality.
This removes the leading whitespace that was approximately the width of
a space character that would get underlined when hovering over any one
of the global filters.
In templates/zerver/api/main.html, since the current context isn't
passed to render_markdown_path when rendering an article,
render_markdown_path doesn't have the context to render values such
as api_url. This commit makes sure that it does by passing a dict
called api_uri_context to render_markdown_path when rendering an
article.
This commit puts the guts of parse_usermessage_flags into
UserMessage.flags_list_for_flags, since it was slightly faster
than the old implementation and produced the same results.
(Both algorithms were super fast, actually.)
And then all callers use the model method now.
The logic to set search_fields was essentially the same for both
sides of the include_history conditional.
Now we have just one code block that sets search_fields, and we
can quickly short-circuit the loop when is_search is False.
For some historical reason we'd had the Postgres documentation on
valid SSL modes copied into the Zulip settings.py template file. This
fixes that historical artifact.
The left navigation bar defaults to "sticking" to the screen as you
scroll. This commit adds the sticky_navigation setting to conf.py to
disable the default behavior.
This addresses part of #5265.
This change affects realm_users and realm_non_active_users.
Note that we still send full avatar urls in realm_user/add
events, so apply_events has to do something mildly hacky to
turn the avatar_url to None in that case.
Fixing the event is probably not worth the trouble, as single
urls are not bandwidth hogs; we only need this optimization
for bulk data.
This change affects these values:
* page_params.avatar_url
* page_params.avatar_url_medium
It requires passing the client_gravatar flag through this
codepath:
* home_real
* do_events_register
* fetch_initial_state_data
* avatar_url
Seems like the more logical check. Also, the previous code makes it feel
like there is a potential vulnerability where one could get an email change
object in a realm where email changes are disabled, and then open that link
while logged in to a different realm.
While we're at it, remove the unnecessary check that the user is
logged in when clicking the confirmation link; that creates
unnecessary trouble for users who use multiple browsers.
Removes an assert, which at this point is there just for readability, since
the second argument to
get_object_from_key(confirmation_key, Confirmation.EMAIL_CHANGE)
ensures that the returned object is of the correct type.
Add a spinner for when a stream is being created to show that
an operation is being performed, while also disallowing users to
modify the form in the meanwhile.
Commit modified by Brock Whittaker <brock@zulipchat.com>.
Fixes: #5268.
This commit allows clients to register client_gravatar=True, and
then we recognize that flag for message events. If the flag is
True, we will not calculate gravatar URLs and let the clients do
it themselves. (Clients can calculate gravatar URLs based on
emails with just a little bit of code.)
This change prepares us to have the server send avatar_url
of None when somebody wants a gravatar avatar (as opposed
to a user-uploaded one).
Subsequent commits will change behavior on both the server
and client to have this happen. So this commit has no-op
code for now, but it will soon use the fallback-to-gravatar
logic.
This refactoring doesn't change behavior, but it sets us up
to more easily handle a register setting for `client_gravatar`,
which will allow clients to tell us they're going to compute
their own gravatar URLs.
The `client_gravatar` flag already exists in our code, but it
is only used for Django views (users/messages) but not for
Zulip events.
The main change is to move the call to `set_sender_avatar` into
`finalize_payload`, which adds the boolean `client_gravatar`
parameter to that function. And then we update various callers
to supply that flag.
One small performance benefit of this change is that we now
lazily compute the client message payloads in
`event_queue.process_message_event` now, so this will improve
performance if all interested clients have the same value of
`apply_markdown`. But the change here is really preparing us
for the additional boolean parameter, which will cause us to
have four variations of the payload.
The i18n.t function already escapes HTML, so we should avoid
calling jQuery's text() method, which double escapes the HTML.
The symptom reported here was that if you changed your
timezone to something like like America/Mexico_City, you'd
see `/` instead of `/`.
Most callers to the `ui_report` functions clearly call `i18n.t`
on the messages with code like this:
ui_report.success(i18n.t("No changes to save!"), status);
There are some exceptions:
blueslip.js: has really long hard coded messages
reload.js: mostly says "Reloading...", which perhaps we should
translate
settings_account.js: uses helper functions
settings_lab.js: uses local variable
settings_org.js: i18n happens in property_types
ui.js: uses generic_embed_error (unaffected by this change)
Fixes#7280
This commit is easy to revert if we want to tone down errors
to warnings for the short term, while our codepath still does
proper handling for adding users when they come in messages.
This logic used to be in extract_people_from_message(), but
we are deprecating extract_people_from_message(), whereas
the maybe_incr_recipient_count() function has logic that we
want to keep.
This change is the first step in making it so that we load
non-active users at page load time in the webapp.
Before this change, we would reactively handle deactivated
users when we saw them as senders in messages. Subsequent
changes will make it a warning if we see unknown senders
in messages.
This is mostly a pure extraction, but we needed to duplicate
a bit of setup from people.js, and I added an assertion
for warning about "No user_id provided".
There's no reason to include deactivated users in compose
typeahead, since we won't let the compose happen.
This also removes cross-realm bots from the typeahead, which
is mostly driven by not having a convenient function to include
them, but also it's rare that it makes sense to talk to a cross-realm
bot unless replying to it.
This gets used when we call `process_client`, which we generally do at
some kind of login; and in particular, we do in the shared auth
codepath `login_or_register_remote_user`. Add a decorator to make it
easy, and use it on the various views that wind up there.
In particular, this ensures that the `query` is some reasonable
constant corresponding to the view, as intended. When not set, we
fall back in `update_user_activity` on the URL path, but in particular
for `log_into_subdomain` that can now contain a bunch of
request-specific data, which makes it (a) not aggregate properly, and
(b) not even fit in the `CHARACTER VARYING(50)` database field we've
allotted it.
The only place this attribute is used is in `update_user_activity`,
called only in `process_client`, which won't happen if we end up
returning a redirect just below. If we don't, we go and call
`add_logging_data` just after, which takes care of this already.
This won't work for all call paths without deeper refactoring,
but for at least some paths we can make this more direct -- function
arguments, rather than mutating a request attribute -- so it's easier
to see how the data is flowing.
I remember being really confused by this function in the past, and I finally
figured it out. It should be removed, and the dev_url added by
00-realm-creation should call a function that just gets the confirmation_key
from outbox like all of the backend tests, but until then this comment
should help.
This change:
* Prevents weird potential attacks like taking a valid confirmation link
(say an unsubscribe link), and putting it into the URL of a multiuse
invite link. I don't know of any such attacks one could do right now, but
reasoning about it is complicated.
* Makes the code easier to read, and in the case of confirmation/views.py,
exposes something that needed refactoring anyway (USER_REGISTRATION and
INVITATION should have different endpoints, and both of those endpoints
should be in zerver/views/registration, not this file).
This test helper method duplicated a bunch of logic in
`zerver/worker/queue_processors.py` in a specialized fashion for the
tests. Now that we're using `call_consume_in_tests` in this code
path, we don't need it.
Before this commit, ResponseMock() was initialized
with a data attribute, which isn't used in the tests
and does not occur in the outgoing webhook code.
The control panel on the Google side doesn't seem to match the
instructions we have; it looks pretty 2017 to me, so I imagine
it's had a redesign since the instructions were written.
Also, in dev, EXTERNAL_HOST is now a port on zulipdev.com, not on
localhost.
Update these instructions for those developments, and edit lightly.
In dev, recommend setting in `dev_settings` instead of in
`prod_settings_template`; that feels to me a little more reflective of
the actual intent, and the effect should be equivalent.
In some minutes of searching yesterday afternoon on the docs site,
I couldn't find docs on how to set up our OAuth integrations for
the dev environment -- even though I was pretty sure they should
be there somewhere, because I'd just been told that. Wasn't until
I considered the problem fresh today, and grepped the docs source
instead, that I found them.
So, move them to a separate page so they're in the nav.
The main limitation of this version is that it's controlled entirely
from settings, with nothing in the database and no web UI or even
management command to control it. That makes it a bit more of a
burden for the server admins than it'd ideally be, but that's fine
for now.
Relatedly, the web flow for realm creation still requires choosing a
subdomain even if the realm is destined to live at an alias domain.
Specific to the dev environment, there is an annoying quirk: the
special dev login flow doesn't work on a REALM_HOSTS realm. Also,
in this version the `add_new_realm` and `add_new_user` management
commands, which are intended for use in development environments only,
don't support this feature.
In manual testing, I've confirmed that a REALM_HOSTS realm works for
signup and login, with email/password, Google SSO, or GitHub SSO.
Most of that was in dev; I used zulipstaging.com to also test
* logging in with email and password;
* logging in with Google SSO... far enough to correctly determine
that my email address is associated with some other realm.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
It's kind of awkward that this takes the scheme-relative URL -- either
the full URL, or maybe just the host, would make more sense. But on a
bit of study of the plugin's source, I can't convince myself that it
won't break if given the full URL with the scheme. (See a code
comment thread in #7116.) So leave the substance as is, pending the
plugin itself getting some cleanup, but do fix the sentence's English
grammar since we're looking at it.
The original PR to allow generic bots to be mentioned had
some merge issues that we detected about a week after the
fact. This commit restores the logic from the original PR.
The reason we didn't detect this bug earlier is that the
merge issues didn't break any existing behavior. Instead,
they made it so that only UserMessage rows got written for
bots, but no events were being set. The part of the commit
that got lost is restored here, so now events get sent as
well.
Thanks to @derAnfaenger for reporting this and being patient
as we tracked it down.
Fixes#7140
On stream names that don’t quite fit, we’ll add overflow ellipsis to
them. It appears we already did this on mobile in a sub-optimal way so
we can remove that media query with preference to the new, better
styling.
This allows a user to scroll all the way down on the sidebar and the
main panel in the settings page on narrow height windows by removing
the min-height specification and making the “Log out” line in the
sidebar become statically positioned at shorter heights.
Fixes: #7251.
This adds the data model and bugdown support for the new UserGroup
mention feature.
Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
tsearch_extras returns search offsets in bytes but our highlight
function treated them as character offsets. Added a check to subtract
extra bytes if the tsearch search backend is being used.
Fixes#4084.
Fixes#7021.
This was apparently completely broken for some time, and got a bit
worse with my recent changes (enough to fail a test).
Since the plan for this page likely involves dissecting and
eliminating it, I think it makes sense to just fix it hackishly and
move on.
In addition to decreasing the excessive number of bundles we had, this
will set us up to fix rendering of code blocks when clicking the
sidebar links in the /api-new site.
This commit allows for the /api-new/ page to rendered similarly to our
/help pages. It's based on the old content for /api, but we're not
replacing the old content yet, to give a bit of time to restructure
things reasonably.
Tweaked by eeshangarg and tabbott.
Because this is for tests, a heuristic like this that's right in most
situations is actually fine; we can override it in the few cases where
a test might set up a situation where it fails.
So just make it clear for the next reader that that's what's going on,
and also adjust the helper's interface slightly so that its callers
do have that flexibility.
The "subdomain" label is redundant, to the extent it's even
accurate -- this is really just the URL we want to display,
which may or may not involve a subdomain. Similarly "external".
The former `external_api_path_subdomain` was never a path -- it's a
host, followed by a path, which together form a scheme-relative URL.
I'm not quite convinced that value is actually the right thing in
2 of the 3 places we use it, but fixing that can start by giving an
accurate name to the thing we have.
This was part of the logic to handle EXTERNAL_API_PATH varying.
But also it was already no longer used -- it was only ever passed
into template contexts, as `external_api_uri`, and it'd been
overtaken there by `external_api_uri_subdomain`.
So, update our dev docs to reflect that, and eliminate the variable.
This setting isn't documented at all, and I believe nobody has used it
since the end of api.zulip.com in 2016. So we get to complete the
cleanup of this logic.
We extract get_bulk_stream_subscriber_info() from this
function to remove some of the complexity. Also, in that
new function we avoid a hop to the database by querying
on stream ids instead of recipient ids. The query that
gets changed here does require a join to the recipient
table (to get the stream id), so it's a little bit of a
tradeoff.
There's an implicit assumption in bulk_remove_subscriptions
that all users belong to the same realm. We use the realm
for things like comparing occupied streams before and
after our main operation of deactivating streams.
Before this change, we just used the user_profile variable
that leaked from some prior loop to look up the realm, which
was super brittle.
Now we're a bit more explicit.
We were using Google's diff-match-patch library to diff HTML. The
problem with that approach is that it is a text differ, not an HTML
differ and so it ends up messing up the HTML tags. `lxml` is a safer
option.
Fixes: #7219.
For some reason, we have the USING_PGROONGA setting on in development
right now. I'm going to disable that in another commit to match what
we're doing in production, but we'll still want that setting to work
in development.
The problem here was that process_fts_updates only attempted to read
the USING_PGROONGA setting from a /etc/zulip/zulip.conf source, and
thus would just not be updating the index in development.
Note that this code leads to a slightly different query, because
we join to one row in the small Recipient table to match
stream_id to recipient.type_id.
The first method we extract to this library is
get_active_subscriptions_for_stream_id().
We also move num_subscribers_for_stream_id() to here, which
is slightly annoying (having the method on Stream was nice)
but avoids some circular dependency issues.
The HelpView class will render a directory as markdown with an index HTML
page. This however can also be used for other generics and applied to
the API pages as well, so change the class to a generic class and
specify the path templates and names.
Tweaked by tabbott and Eeshan Garg.
FuncT was unused in decorator.py, and only imported into profile.py.
The @profiled decorator is now more strongly typed on return-type.
Annotations were converted to python3 format.
This extraction moves all the huddle logic into models.py, which
hopefully can reduce friction for things like re-organizing our
caches (there are two cache entries for every huddle) and/or
just putting huddle_id on Message directly.
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)? I could never
remember, and it was not very type safe, since both parameters
are integers.
Almost all callers to do_create_user were trying to
create active users, except for one test. The
active=False codepath was kind of broken (things
like sending welcome messages had sort of undefined
behavior there), so instead of trying to maintain it,
we just update the one test (`test_people`) to flip the
`is_active` flag manually.
Fixes#7197
We mostly introduce these functions (as part of a big
code sweep):
send_stream_message
send_personal_message
send_huddle_message
In two cases, where we want to specifically manipulate
queue ids, we now call check_send_message directly. (The
above three functions deliberately don't support kwargs
to ensure simple code and better type safety.)
Along with fixing some minor bugs, this requires extracting out the
default functions so that we can do type: ignores on them properly.
While we're at it, we switch to the Python 3 syntax.
We correct a bug on Firefox where using the ESC key to close an edit
box that was opened by the left arrow key caused the message feed to
lose focus, making it difficult to navigate the message feed by
keyboard afterwards.
We fix this bug by changing the function that handles the ESC key
during an edit to pass the correct object to the message_edit.end
function.
Fixes#7072.
Our set_loglevel tool didn't set propagate to False, so just using it
directly wouldn't work unless the logger is explcitly declared in
zproject/settings.py, which this one isn't.
This creates a template for the /team/ page that is currently just
embedded inside the /about/ page. This includes the titles for core
team members with their pictures.
If a Zulip install at example.org got a request at an HTTP `Host`
like foo.example.org.evil.com (or even foo.example.orgevil.com),
we would accept it as subdomain foo. This isn't likely to happen
in practice because it shouldn't pass ALLOWED_HOSTS, and it's not
obvious to me that anything untoward could be done with it even
if ALLOWED_HOSTS were set wide open, but if nothing else it
multiplies the cases in analyzing this logic.
The reason we had a loose match like this, I assume, is to allow
the user to come from arbitrary ports -- especially in development.
So tighten the pattern to allow just that, and add some tests for
that behavior and a comment explaining why this complication is
needed.
The cookie mechanism only works when passing the login token to a
subdomain. URLs work across domains, which is why they're the
standard transport for SSO on the web. Switch to URLs.
Tweaked by tabbott to add a test for an expired token.
This makes the tests a little cleaner in itself, and also prepares
them to adjust with less churn when we change how
redirect_and_log_into_subdomain passes the signed token.
Most of these have more to do with authentication in general than with
registering a new account. `create_preregistration_user` could go
either way; we move it to `auth` so we can make the imports go only in
one direction.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.
There are a few changes that we can expect for the future:
* It is currently possible to invite an email that you have already
invited, it might make sense to change this behavior.
* Resend currently sends an invite reminder instead of resending the
original invite, this is because 'custom_body' was not stored when
the first invite was sent.
Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.
Fixes: #1180.
Tweaked by tabbott to have the field before the invitation is
completed be called invite_as_admins, not invited_as_admins, for
readability.
Fixes#6834.
The tighter interface prevents the need to specify
Recipient.PERSONAL (which can often be inaccurate in the
huddle case, anyway), and it prevents tests from confusingly
specifying a "subject" field for PMs.
Having send_stream_message() avoids the need to supply
Recipient.STREAM as a parameter, and it also uses the more
modern name of `topic_name` for topics. Under the hood, it
avoids some annoying steps for re-formatting the recipients,
since we just have a single stream name.
When possible, we want to use direct APIs for sending
stream messages.
This changes the codepath slightly, by not using
forwarded_user_profile, but it doesn't impact the number
of queries, and it's a simple check.
We also remove a couple "subject" references here.
This change allows normal bots to get UserMessage rows when
they are mentioned on a stream, even if they are not actually
subscribed to the stream.
Fixes#7140.
We now find all (possibly) relevant service bots for a message
in the call to get_recipient_info. This allows us to eliminate
some code that would patch them after we rendered.
The get_service_bot_events() function will ignore any service
bots that weren't actually mentioned in the message (due to
backticks) or part of the active user ids.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.
Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.
We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process. This info
now gets passed down the call stack as an optional parameter.
Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.
This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
In do_send_messages, we only produce one dictionary for
the event queues, instead of different flavors for text
vs. html. This prevents two unnecessary queries to the
database.
It also means we only put one dictionary on the "message"
event queue instead of two, albeit a wider one that has
some values that won't be sent to the actual clients.
This wider dictionary from MessageDict.wide_dict is also
used for the `feedback_messages` queue and service bot
queues. Since the extra fields are possibly useful down
the road, and they'll just be ignored for now, we don't
bother to remove them. Also, those queue processors won't
have access to `content_type`, which they shouldn't need.
Fixes#6947
Before this change, we populated two cache entries for each
message that we sent. The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.
This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.
One legacy source on confusion here is that `content`
changes meaning when you're on the front end. Here is the
situation going forward:
database:
content = raw
rendered_contented = rendered
cache entry:
content = raw
rendered_contented = rendered
payload for the frontend:
content = raw (for apply_markdown=False)
content = rendered (for apply_markdown=True)
All links by default had an underline on hover, including when
<a> tags were wrapping <div> and <img> tags, which made for a small
underline near them on hover. This better focusses the underline
behavior to just paragraphs and lists.
This detects the meta key being pressed to open a page in a new
tab and therefore will prevent the page animation from fading out
the body content, in case the user wants to go back to that page
again.
We were incorrectly reporting active bots as non-active in
popovers, and we had no test coverage for cross-realm bots.
We also rename the function to is_active_user_for_popover,
since the old name, realm_user_is_active_human_or_bot, suggested
the wrong semantics for cross-realm bots.
Last but not least, we only do a blueslip warning if a user id
is not found. When lookups fail, we are pretty confident that
the user is not active, so an error is overkill. We can change
that as part of issue #7120.
Fixes#7153
Wherever possible, we always want to move checking for error
conditions to the views code, so that we don't need to worry about
handling failures with (in this case) a user that's half-created
because a DefaultStreamGroup doesn't exist.
This effectively implements the feature of default stream groups,
except for a UI, nice styling, etc.
Note that we're careful to not have this do anything in an
organization that doesn't have any default stream groups.
This adds two constraints to the image:
1. The `max-width` can not be more than 500px (which prevents it
from being to vertically tall.
2. The `display` is set to `none` below 1024px because the image is
too small at that point to be legible.
This has exactly the same behavior so long as self.subdomain contains
no colon character, ':'; and of course we don't allow those in
subdomains, because they aren't allowed by DNS.
These are just instances that jumped out at me while working on the
subdomains code, mostly while grepping for get_subdomain call sites.
I haven't attempted a comprehensive search, and there are likely
still others left.
I'd much rather see something like
if (thing_is_permissible(user, thing)
or (user_possesses_hammer(user)
and glass_break_requested(thing))):
than
if (thing_is_permissible(user, thing) or
(user_possesses_hammer(user) and
glass_break_requested(thing))):
because the former makes the overall logic much easier to scan.
Similarly for a formula full of arithmetic rather than Boolean
operators. And the actual PEP 8 agrees (though until 2016 it
unfortunately had the opposite advice.)
The upstream linter still applies the backward rule, so disable that.
Now that the old `check_subdomain` has no callers except in
implementing the new, improved interface `user_matches_subdomain`,
inline it into that. Also simplify the Boolean logic a bit.
Now that every call site of check_subdomain produces its second
argument in exactly the same way, push that shared bit of logic
into a new wrapper for check_subdomain.
Also give that new function a name that says more specifically what
it's checking -- which I think is easier to articulate for this
interface than for that of check_subdomain.
This should be a pure refactor: the only asymmetry in the behavior
of `check_subdomain` between its two arguments is if one of them
is None, and in this case we have a non-nullable model field on
one side and the return value from `get_subdomain` on the other.
With these swapped, this call site now matches all other
`check_subdomain` call sites in having the second argument come as
the subdomain of some user's realm.
The type of get_subdomain's parameter is non-Optional, and
in fact if passed an argument of None it would promptly
blow up. So this `getattr` can't be serving any purpose.
It's easier to unit test logic inside of people.js than compose.js.
We allow users to compose emails to any of our cross-realm bots.
Someday we may tighten up which cross-realm bots are valid targets,
since it's not necessarily the case that those bots do anything
useful when you send them messages.
This dictionary includes bots, so the reference to
"people" in the name `realm_people_dict` was misleading.
We omit `realm` for brevity sake--it's usually the case
that folks implementing new features can safely ignore
cross-realm bots, and it's on our roadmap to move those
bots into the realm.
The function name `get_realm_human_user_ids` was a lie--it
includes active bots as well.
The only user of this function is `activity.js`, which wasn't
impacted by the misleading name, because we eventually filter
out bots in the `info_for` function.
It's possible that we actually want to include bots in the right
sidebar, since they can be difficult to discover in other parts
of the UI. Or, if we want to keep the right sidebar as all
human users, we may eventually want to make the logic to exclude
bots happen higher in the stack (but for real, this time).
This changes some text that would display gray when on a blue body
text page; we considered changing the opacity instead, but probably we
should just delete this..
This changes the <ul> styling so that when not nested in a <p>
tag it'll have the standard font-weight (400) and be the same
color as the body text (blue/gray).
Adds support to add "Embedded bot" Service objects. This service
handles every embedded bot.
Extracted from "Embedded bots: Add support to add embedded bots from
UI" by Robert Honig.
Tweaked by tabbott to be disabled by default.
This fixes an exception occurring when engaging an embedded
bot in a PM, makes it respond as itself instead of the sender,
and makes it respond to the PM conversation it is engaded in.
This removes the old blue styled outline around the PM recipient
box that was part of the older bootstrap styling in favor of the
dark outline on :focus that had been implemented for the rest of
the recipient boxes recently.
This creates a dropdown in place of the normal register/login links
you get when logged out, with an option to go to the app or log out if
that appears you click on the avatar.
A bit more work is needed to make this look really good, but it's a
great start.
Every time we updated a UserProfile object, we were calling
delete_display_recipient_cache(), which churns the cache and
does an extra database hop to find subscriptions. This was
due to saying `updated_fields` instead of `update_fields`.
This made us prone to cache churn for fields like UserProfile.pointer
that are fairly volatile.
Now we use the helper function changed(). To prevent the
opposite problem, we use all the fields that could invalidate
the cache.
This test had a little bug, where we weren't actually
verifying `realm_bots` before, because we weren't using
`field` to look it up.
This commit fixes that bug and adds additional checks,
particularly for the recently added `realm_non_active_users'.
We now add `realm_non_active_users` to the result of
`do_events_register` (and thus `page_params`). It has
the same structure as `realm_users`, but it's for
non-active users. Clients need data on non-active users
when they process old messages that were sent by those
users when they were active. Clients can currently get
most of the data they need in the message events, but it
makes for ugly client code.
Fixes#4322
This is a prepatory commit that adds non-active users to
the realm user cache. It mostly involves name changes and
removing an `is_active` filter from the relevant DB query.
The only consumer of this cache is `get_raw_user_data`, which
now filters on `is_active` in a dictionary comprehension (but
this will get moved around a bit in a subsequent commit).
We make a few things cleaner for populating `realm_users`
in `do_event_register` and `apply_events`:
* We have a `raw_users` intermediate dictionary that
makes event updates O(1) and cleaner to read.
* We extract an `is_me` section for all updates that
apply to the current user.
* For `update` events, we do a more surgical copying
of fields from the event into our dict. This
prevents us from mutating fields in the event,
which was sketchy (at least in test mode). In
particular, this allowed us to remove some ugly
`del` code related to avatars.
* We introduce local vars `was_admin` and `now_admin`.
The cleanup had two test implications:
* We no longer need to normalize `realm_users`, since
`apply_events` now sees `raw_users` instead. Since
`raw_users` is a dict, there is no need to normalize
it, unlike lists with possibly random order.
* We updated the schema for avatar updates to include
the two fields that we used to hackily delete from
an event.
@@ -180,7 +179,7 @@ processes that process event queues. We use event queues for the kinds
of tasks that are best run in the background because they are
expensive (in terms of performance) and don't have to be synchronous
--- e.g., sending emails or updating analytics. Also see [the queuing
guide](queuing.html).
guide](../subsystems/queuing.html).
### memcached
@@ -227,7 +226,7 @@ processes started by Supervisor are queue processors that continually
pull things out of a RabbitMQ queue and handle them; they are defined
in `zerver/worker/queue_processors.py`.
Also see [the queuing guide](queuing.html).
Also see [the queuing guide](../subsystems/queuing.html).
### PostgreSQL
@@ -295,8 +294,8 @@ are welcome!
* **star**: Zulip allows a user to mark any message they can see,
public or private, as "starred". A user can easily access messages
they've starred through the "Starred messages" link in the menu
near "Home", or use "is:starred" as a narrow or a search
they've starred through the "Starred messages" link in the
left sidebar, or use "is:starred" as a narrow or a search
constraint. Whether a user has or has not starred a particular
message is private; other users and realm admins don't know
whether a message has been starred, or by whom.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
